Municipal road pavement crack multi-modal fusion detection method

Abstract

The invention discloses a municipal road pavement crack multi-modal fusion detection method, which belongs to the technical field of pavement crack detection, and comprises the following steps: S1, obtaining visible light images, infrared thermal imaging and three-dimensional laser scanning data in multi-modal data, generating a time-unified and space-aligned multi-modal data set based on the three-dimensional laser scanning data; s2, extracting a visible light image and an infrared thermal image from the multi-modal data set, and obtaining features corresponding to the visible light image and the infrared thermal image to obtain a multi-modal feature set; and S3, inputting the multi-modal feature set into a deep neural network, and carrying out weighted integration on feature vectors through multi-layer convolution operation to obtain a crack detection result with three-dimensional coordinates. The municipal road pavement crack multi-modal fusion detection method solves the problem that an existing pavement crack detection method is low in accuracy and reliability.

Description

Technical Field

[0001] The present invention relates to the technical field of pavement crack detection, and particularly relates to a method for multi-modal fusion detection of pavement cracks on municipal roads. Background Art

[0002] The detection of pavement cracks on municipal roads is a core research direction in the field of infrastructure maintenance, which is directly related to road safety and service life. With the acceleration of the urbanization process, the growing demand for road maintenance has promoted the continuous innovation of crack detection technologies. Currently, single-modal detection methods have shown obvious deficiencies in practical applications. Visible light images are easily affected by illumination and occlusion interference, the resolution of infrared thermal imaging is insufficient, and the efficiency of three-dimensional laser scanning is low. These limitations make it difficult for a single modality to adapt to the diversity and complexity of the road environment, and the detection results are often not comprehensive or stable enough to meet the high standards of modern road maintenance requirements.

[0003] Although multi-modal fusion detection is highly anticipated, the physical characteristics of visible light, infrared, and laser data are significantly different, making it difficult to achieve precise data registration and synchronization in software. Secondly, the manifestations of pavement cracks in each modality are quite different, and it is difficult to define unified discriminant features. In summary, the existing multi-modal fusion detection methods for pavement cracks on municipal roads are characterized by low accuracy and reliability. Summary of the Invention

[0004] In order to overcome the defects existing in the prior art, the present invention provides a method for multi-modal fusion detection of pavement cracks on municipal roads to solve the above problems.

[0005] The technical solution adopted by the present invention to solve its technical problems is: a method for multi-modal fusion detection of pavement cracks on municipal roads, including the following steps: S1: Obtain visible light images, infrared thermal images, and three-dimensional laser scanning data in multi-modal data, and generate a multi-modal data set with unified time and aligned space based on the three-dimensional laser scanning data; S2: Extract visible light images and infrared thermal images from the multi-modal data set, and obtain the corresponding features of the visible light images and infrared thermal images to obtain a multi-modal feature set; S3: Input the multi-modal feature set into a deep neural network, and perform weighted integration on the feature vectors through multi-layer convolution operations to obtain a crack detection result with three-dimensional coordinates.

[0006] It should be noted that in step S1, the visible light images, infrared thermal images, and three-dimensional laser scanning data in the multi-modal data are subjected to time synchronization processing according to the timestamp alignment algorithm; a reference coordinate system is determined in the three-dimensional laser scanning data, and the visible light images and infrared thermal images are mapped to the reference coordinate system to obtain a multi-modal data set with unified time and aligned space.

[0007] Preferably, in the step S1, timestamps at the time of collecting each data are extracted from the original multi-modal data; by comparing the timestamps, it is determined whether there is a time deviation among the various data of the multi-modal data during collection; for the data with a time deviation, the timestamp corresponding to the 3D laser scanning data is obtained as the synchronization time, and the content of each data between its corresponding timestamp and the synchronization time is adjusted by using the linear interpolation method to obtain the adjusted data points, and a multi-modal data set with consistent time is generated.

[0008] Optionally, in the step S1, before mapping the infrared thermal imaging to the reference coordinate system, the infrared thermal imaging is first resampled to make its resolution consistent with that of the visible light image.

[0009] Specifically, in the step S2, an edge detection algorithm is used to obtain the crack morphology features from the visible light image.

[0010] Specifically, in the step S2, the gray value distribution data is extracted from the visible light image, and the Canny edge detection algorithm is used to perform edge detection on the gray value distribution data to obtain the crack boundary; According to the crack boundary, a region segmentation mechanism is used to determine the crack morphology distribution; According to the crack boundary and the crack morphology distribution, the crack morphology features are determined.

[0011] It should be noted that, in the step S2, if there is an overlapping area in the crack morphology features, the overlapping area is adjusted by pixel clustering to obtain an independent feature area to distinguish different crack branches.

[0012] Preferably, in the step S2, thermal gradient analysis is used to obtain the thermal radiation features from the infrared thermal imaging.

[0013] Optionally, in the step S3, a multi-modal feature set is obtained, and the multi-modal feature set is hierarchically processed by ResNet50 to obtain a feature vector group, and the dimension of the feature vector group is 2048 dimensions; A 3x3 convolutional kernel is used to perform channel weighting on the feature vector group, and the significance scores of each channel are calculated through the Softmax function; if the significance score exceeds the preset threshold of 0.7, the weight of the corresponding channel is multiplied by a preset coefficient to obtain a weighted feature vector group; The weighted feature vector group is input into the fully connected layer to obtain a discriminant feature vector, and the dimension of the discriminant feature vector is 128 dimensions; the discriminant feature vectors are fused by element-wise addition to obtain the crack detection result.

[0014] The beneficial effects of the present invention are as follows: In the method for multi-modal fusion detection of pavement cracks on municipal roads, first, time synchronization and spatial registration are performed on visible light images, infrared thermal images, and three-dimensional laser scanning data to establish a unified multi-modal data set. Then, specific features such as crack morphology and thermal radiation characteristics are extracted from each modal data and input into a deep neural network for weighted fusion. Finally, combined with the reference coordinate system of the three-dimensional laser scanning data, a crack detection result with three-dimensional coordinates is obtained. By synergistically analyzing multi-modal data, the present invention unifies various multi-modal data in terms of time and space, effectively improving the accuracy and reliability of crack risk detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a flowchart of the method for multi-modal fusion detection of pavement cracks on municipal roads in an embodiment of the present invention; Figure 2 It is a flowchart of obtaining crack morphology features from visible light images using an edge detection algorithm in an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0016] The following further describes the specific embodiments of the present invention with reference to the drawings. It should be noted here that the description of these embodiments is for helping to understand the present invention, but does not constitute a limitation to the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0017] As Figure 1 and 2 shown, a method for multi-modal fusion detection of pavement cracks on municipal roads includes the following steps: S1: Obtain visible light images, infrared thermal images, and three-dimensional laser scanning data in multi-modal data, and generate a multi-modal data set with unified time and aligned space based on the three-dimensional laser scanning data; S2: Extract visible light images and infrared thermal images from the multi-modal data set, and obtain the corresponding features of the visible light images and infrared thermal images to obtain a multi-modal feature set; S3: Input the multi-modal feature set into a deep neural network, and perform weighted integration on the feature vectors through multi-layer convolution operations to obtain a crack detection result with three-dimensional coordinates.

[0018] In the method for multi-modal fusion detection of pavement cracks on the municipal road, first, time synchronization and spatial registration are performed on visible light images, infrared thermal images, and 3D laser scanning data to establish a unified multi-modal data set. Then, specific features, such as crack morphology and thermal radiation features, are extracted from each modal data and input into a deep neural network for weighted fusion. Finally, combined with the reference coordinate system of the 3D laser scanning data, a crack detection result with 3D coordinates is obtained. Through the collaborative analysis of multi-modal data, the present invention unifies various multi-modal data in terms of time and space, effectively improving the accuracy and reliability of crack risk detection.

[0019] It should be noted that in step S1, time synchronization processing is performed on the visible light image, infrared thermal image, and 3D laser scanning data in the multi-modal data according to the timestamp alignment algorithm; a reference coordinate system is determined in the 3D laser scanning data, and the visible light image and infrared thermal image are mapped to the reference coordinate system to obtain a multi-modal data set with unified time and aligned space.

[0020] It can be understood that visible light images and infrared thermal images are usually two-dimensional pixel data, while the reference coordinate system is a three-dimensional space. In a possible implementation manner, based on the internal and external parameters of the camera or thermal imager, the perspective projection matrix is calculated for projection transformation to associate the plane of the visible light image or infrared thermal image with the space of the 3D laser scanning data, obtaining the transformed 3D coordinates, so as to project the visible light image or infrared thermal image onto the reference coordinate system; assuming that a crack on the roadside is located at pixel coordinates (200, 300) in the visible light image, through perspective projection transformation, it is mapped to the 3D point (15, 25, 35) in the reference coordinate system. In addition, the temperature value of a certain area in the infrared image is 40°C, corresponding to the point (15, 25, 35) in the reference coordinate system, thus aligning with the visible light image. Then, the 3D laser scanning data, visible light image, and infrared thermal image are spatially superimposed to form a spatially aligned multi-modal data set, ensuring the spatial consistency of the multi-modal data.

[0021] In this embodiment, the internal parameters of the camera are parameters describing the internal attributes of the camera, including focal length, principal point (optical center) coordinates, distortion coefficients, etc. The internal parameters are usually determined during camera calibration because they are usually fixed for a specific camera model and do not change with time. Once the camera internal parameters are determined, they usually remain unchanged during the use of the camera; the external parameters of the camera are parameters describing the position and orientation of the camera in the world coordinate system, usually including a rotation matrix and a translation vector. The external parameters may change at different camera positions or shooting moments. For example, in stereo vision, the relative position and direction of the camera and the target object being photographed will change each time the camera is moved, resulting in changes in the external parameters.

[0022] Preferably, in the step S1, timestamps when each data is collected are extracted from the original multi-modal data; the timestamps are compared to determine whether there is a time deviation among the data of the multi-modal data during collection; for the data with a time deviation, the timestamp corresponding to the 3D laser scanning data is obtained as the synchronization time, and the linear interpolation method is used to adjust the content of each data between its corresponding timestamp and the synchronization time to obtain the adjusted data points, so as to synchronize the visible light image, the infrared thermal imaging and the 3D laser scanning data, and generate a multi-modal data set with consistent time.

[0023] For example, the timestamp when the visible light image is collected is "2024-04-02-10:00:00.789", the timestamp when the infrared thermal imaging is collected is "2024-04-02-10:00:00.456", and the timestamp when the 3D laser scanning data is collected is "2024-04-02-10:00:00.123". The extraction of the timestamp can be based on the file metadata or the built-in clock information of the data packet header.

[0024] In this embodiment, the dynamic time warping algorithm is used to compare the timestamps to determine whether there is a time deviation among the data of the multi-modal data during collection. By calculating the similarity between time series, the dynamic time warping finds that the intervals between the above three timestamps are 0.333 seconds and 0.333 seconds respectively, indicating that there is a deviation but the regularity is relatively strong. The advantage of the dynamic time warping algorithm is that it can adapt to non-linear time offsets and is suitable for the situation where the sampling frequencies of sensors are inconsistent.

[0025] If there is a time deviation, for visible light images, infrared thermal imaging, and three-dimensional laser scanning data, the linear interpolation method is used to obtain the adjusted data points. For example, the timestamp when the visible light image is collected is "2024-04-02-10:00:00.789", the timestamp when the infrared thermal imaging is collected is "2024-04-02-10:00:00.456", the timestamp when the three-dimensional laser scanning data is collected is "2024-04-02-10:00:00.123", and the target synchronization time is "2024-04-02-10:00:00.123". In this way, the temperature value of the infrared thermal imaging between "2024-04-02-10:00:00.123" and "2024-04-02-10:00:00.456" can be estimated by linear interpolation, and the pixel value of the visible light image between "2024-04-02-10:00:00.123" and "2024-04-02-10:00:00.789" can be estimated by linear interpolation, so as to obtain the adjusted visible light image, infrared thermal imaging, and three-dimensional laser scanning data. This method is simple and efficient and can effectively fill the time gap. The interpolated visible light image pixels, infrared thermal imaging temperature values, and three-dimensional laser scanning data can be uniformly aligned to "2024-04-02-10:00:00.123" to form a time-consistent multi-modal dataset. This synchronous set provides a time-consistent basis for subsequent analysis.

[0026] Optionally, in the step S1, before mapping the infrared thermal imaging to the reference coordinate system, the infrared thermal imaging is first resampled to make its resolution consistent with that of the visible light image. Resampling the infrared thermal imaging is to match the resolution of the visible light image; for example, the original resolution of the infrared thermal imaging is 320x240, while the visible light image is 640x480; during resampling, the temperature value of each pixel of the infrared image can be expanded to 4 pixels to maintain the continuity of the temperature distribution.

[0027] Specifically, in the step S2, an edge detection algorithm is used to obtain the crack morphology features from the visible light image.

[0028] Specifically, in the step S2, the gray value distribution data is extracted from the visible light image, and the Canny edge detection algorithm is used to perform edge detection on the gray value distribution data to obtain the crack boundary. When using the Canny edge detection algorithm to perform edge detection on the gray value distribution data, the core lies in identifying the clear contour of the crack. Specifically, the Canny edge detection algorithm first performs Gaussian smoothing to reduce noise, then calculates the gradient of the gray value distribution data to highlight the edge, and finally uses double thresholds to screen out the crack boundary. For example, the crack edge may appear as a line with a sharp change in gray value from 120 to 40 in the visible light image, and the Canny edge detection algorithm extracts it as a continuous boundary. This method can effectively capture the starting and ending points of the crack.

[0029] According to the crack boundary, the region segmentation mechanism is used to determine the crack morphology distribution. When using the region segmentation mechanism according to the crack boundary, the purpose is to separate the crack region from its surroundings. In one possible implementation, after obtaining the crack boundary, the region segmentation can be achieved by setting a gray value threshold, such as 100, to separate the internal and external regions of the crack. For example, the internal gray value of the crack is lower than 80, and the external is higher than 120. After segmentation by the gray value threshold, the crack morphology distribution can be obtained. This distribution can clearly reflect the width and extension path of the crack. Determine the crack morphology characteristics based on the crack boundary and the crack morphology distribution.

[0030] It should be noted that in the step S2, if there is an overlapping area in the crack morphology characteristics, the overlapping area is adjusted through pixel clustering to obtain independent feature regions to distinguish different crack branches. It should be noted that the overlapping area generally occurs at the crack branch. For example, the pixels are mixed in the intersection area of two cracks. Pixel clustering re - groups according to the similarity of gray values. For example, pixels with gray values between 70 - 90 are grouped into one category, and finally independent feature regions are separated, that is, the crack branches are separated. This adjustment can ensure the uniqueness of each region.

[0031] Preferably, in the step S2, thermal gradient analysis is used to obtain thermal radiation characteristics from the infrared thermal imaging. In this embodiment, the infrared thermal imaging is processed by thermal gradient analysis to obtain a thermal radiation characteristic distribution. When processing the initial thermal imaging data by thermal gradient analysis, the purpose is to highlight the thermal radiation changes in the crack area; in a possible implementation manner, the thermal gradient analysis generates a thermal radiation characteristic distribution by comparing the temperature differences of adjacent pixels; for example, the temperature at the crack edge drops sharply from 40 degrees Celsius to 35 degrees Celsius, and the thermal gradient analysis marks it as a significant change area, and this distribution can clearly reflect the thermal profile of the crack; when extracting the crack area characteristics according to the thermal radiation characteristic distribution, the temperature anomaly area needs to be concerned. Specifically, the crack area may be manifested as a strip area with a temperature lower than 38 degrees Celsius; in another possible implementation manner, the thermal gradient analysis can also distinguish the depth of the crack. The superficial crack is manifested as a smaller thermal radiation range in the thermal radiation characteristics, while the deep crack is manifested as a wider thermal radiation range in the thermal radiation characteristics.

[0032] Specifically, in the step S3, a multi-modal feature set is obtained. The multi-modal feature set is hierarchically processed by ResNet50 to obtain a feature vector group, and the dimension of the feature vector group is 2048; exemplarily, in road surface detection, the crack morphology features in the multi-modal feature set include the edge details of the crack, while the thermal radiation features include the temperature anomaly in the crack area; after inputting these data into ResNet50, the network extracts deep features through multiple layers of convolution and finally outputs a 2048-dimensional feature vector group; it can be understood that ResNet50 is good at capturing the spatial hierarchical structure of images, such as the shape and depth changes of cracks, which provides rich basic information for subsequent processing; A 3x3 convolutional kernel is used to perform channel weighting on the feature vector group, and the significance scores of each channel are calculated through the Softmax function; if the significance score exceeds a preset threshold of 0.7, the weight of the corresponding channel is multiplied by a preset coefficient to obtain a weighted feature vector group; in a possible implementation manner, a 3x3 convolutional kernel is used to perform channel weighting on the 2048-dimensional feature vector group. The role of the convolutional kernel is to locally enhance the features of each channel, such as highlighting the texture contrast at the crack edge; then the significance scores are calculated through the Softmax function. Suppose the score of a certain channel reaches 0.8, exceeding the preset threshold of 0.7, then its weight is multiplied by a preset coefficient, such as 1.5. This weighting method can amplify the influence of key features, such as the temperature signal in the crack area, and at the same time suppress the interference of irrelevant backgrounds; Input the weighted feature vector group into the fully connected layer to obtain a discriminant feature vector with a dimension of 128; fuse the discriminant feature vectors by element-wise addition to obtain the crack detection result. It should be noted that after the weighted feature vector group enters the fully connected layer, the dimension is compressed to a discriminant feature vector of 128 dimensions; the role of the fully connected layer is to integrate the scattered features into a more compact expression, for example, fusing the width of the crack and the temperature change into a single index. Exemplarily, a certain discriminant feature vector may represent a crack with a width of 3 cm and a temperature of 40 °C. After fusing these vectors by element-wise addition, the final crack detection result is obtained, for example, determining that there are serious cracks in a certain area.

[0033] The embodiments of the present invention have been described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the described embodiments. For those skilled in the art, without departing from the principle and spirit of the present invention, various changes, modifications, substitutions, and variations made to these embodiments still fall within the protection scope of the present invention.

Claims

1. A method for multi-modal fusion detection of pavement cracks on municipal roads, characterized in that, Including the following steps: S1: Obtain visible light images, infrared thermal images, and three-dimensional laser scanning data in the multimodal data, and generate a multimodal dataset with unified time and aligned space based on the three-dimensional laser scanning data; S2: Extract visible light images and infrared thermal images from the multimodal dataset, and obtain the corresponding features of the visible light images and infrared thermal images to get a multimodal feature set; S3: Input the multimodal feature set into a deep neural network, and perform weighted integration on the feature vectors through multi-layer convolution operations to obtain a crack detection result with three-dimensional coordinates.

2. A method for multimodal fusion detection of pavement cracks on municipal roads according to claim 1, characterized in that: In the step S1, perform time synchronization processing on the visible light images, infrared thermal images, and three-dimensional laser scanning data in the multimodal data according to the timestamp alignment algorithm; Determine a reference coordinate system in the three-dimensional laser scanning data, and map the visible light images and infrared thermal images to the reference coordinate system to obtain a multimodal dataset with unified time and aligned space.

3. A method for multimodal fusion detection of pavement cracks on municipal roads according to claim 2, characterized in that: In the step S1, extract the timestamps when each data is collected from the original multimodal data; compare the timestamps to determine whether there is a time deviation between the data in the multimodal data during collection; for the data with time deviation, obtain the timestamp corresponding to the three-dimensional laser scanning data as the synchronization time, and use the linear interpolation method to adjust the content of each data between its corresponding timestamp and the synchronization time to obtain the adjusted data points, and generate a multimodal dataset with consistent time.

4. A method for multi-modal fusion detection of pavement cracks on municipal roads according to claim 3, characterized in that: In the step S1, before mapping the infrared thermal image to the reference coordinate system, first resample the infrared thermal image to make its resolution consistent with that of the visible light image.

5. A method for multi-modal fusion detection of pavement cracks on municipal roads according to claim 4, characterized in that: In the step S2, use an edge detection algorithm to obtain crack morphology features from the visible light image.

6. A method for multi-modal fusion detection of pavement cracks on municipal roads according to claim 5, characterized in that: In the step S2, extract gray value distribution data from the visible light image, and perform edge detection on the gray value distribution data using the Canny edge detection algorithm to obtain crack boundaries; Determine the crack morphology distribution according to the crack boundaries using a region segmentation mechanism; Determine the crack morphology features according to the crack boundaries and the crack morphology distribution.

7. A method for multi-modal fusion detection of pavement cracks on municipal roads according to claim 6, characterized in that: In the step S2, if there is an overlapping area in the crack morphology features, adjust the overlapping area through pixel clustering to obtain independent feature regions to distinguish different crack branches.

8. A method for multimodal fusion detection of pavement cracks on municipal roads according to claim 7, characterized in that: In the step S2, obtain thermal radiation features from the infrared thermal image using thermal gradient analysis.

9. A method for multi-modal fusion detection of pavement cracks on municipal roads according to claim 8, characterized in that: In the step S3, obtain a multimodal feature set, and perform hierarchical processing on the multimodal feature set through ResNet50 to obtain a feature vector group with a dimension of 2048; Use a 3x3 convolutional kernel to perform channel weighting on the feature vector group, and calculate the significance score of each channel through the Softmax function; if the significance score exceeds the preset threshold of 0.7, multiply the weight of the corresponding channel by a preset coefficient to obtain a weighted feature vector group; Input the weighted feature vector group into a fully connected layer to obtain a discriminant feature vector with a dimension of 128; fuse the discriminant feature vectors by element-wise addition to obtain a crack detection result.

Images