A multi-modal perception-based road disease intelligent identification method

Abstract

This invention belongs to the field of road defect identification technology, specifically disclosing an intelligent road defect identification method based on multimodal perception. It acquires multispectral images and three-dimensional laser point clouds of the same road surface location, divides the area into oil-dominated marking zones and crack candidate marking zones based on the pixel brightness ratio between multispectral bands, and then projects the oil-dominated marking zones inversely into three-dimensional space. A local dynamic elevation baseline is constructed using the point cloud of the clean road surface outside the oil-contaminated boundary. Relative elevation difference analysis is performed within the oil-dominated marking zones, and the elevation change of the point cloud is used as a physical verification method to check the spectral marking results. This enables proactive collaboration between the multispectral image and the three-dimensional point cloud, effectively identifying hidden cracks under oil contamination. It goes beyond simply distinguishing between oil contamination and defects on the surface; instead, it uses the physical indicator of structural subsidence to confirm whether cracks are hidden beneath the oil contamination, significantly reducing the risk of missed crack detection in oil-covered scenarios.

Description

Technical Field

[0001] This invention belongs to the field of road defect identification technology, and specifically discloses an intelligent road defect identification method based on multimodal perception. Background Technology

[0002] During long-term service, roads are subjected to repeated vehicle loads and environmental factors, which gradually cause various defects such as cracks to appear on their surface and structural layers. If these defects are not detected and treated in time, they will continue to expand, leading to accelerated degradation of the pavement's structural performance. Therefore, it is necessary to conduct regular road defect identification.

[0003] Currently, image acquisition and visual analysis are commonly used for road defect identification. However, in actual road scenarios, both oil pollution and real defects appear as dark areas in visible light images. This makes it easy for detection methods based on a single visible light image to misjudge oil pollution as cracks, resulting in a large number of false alarms.

[0004] To overcome the aforementioned problems, existing technologies have developed solutions that utilize multispectral imaging to distinguish between oil contamination and actual road surface defects. For example, Chinese invention patent CN120782760B proposes a method and system for pre-inspecting road surface conditions. This method classifies oil contamination, water accumulation, and normal road surfaces by acquiring multispectral and visible light images based on the separability of their spectral reflectance characteristics. The two classification results are then fused to obtain the final road surface condition.

[0005] However, the above scheme still has the following shortcomings: First, the above scheme processes multispectral images and visible light images by classifying them separately and then fusing the results. The classification processes of the two types of images are independent of each other. When the classification result of one modality image is deviated, the other modality cannot effectively verify it, which leads to a decrease in the reliability of the fusion result.

[0006] Secondly, the above-mentioned method can only qualitatively mark the oil-stained area and cannot further determine whether there are structural defects such as cracks hidden beneath the oil-covered area. In actual road scenarios, oil stains may happen to cover cracks, making the cracks invisible in both visible light and multispectral images. Spectral information alone cannot penetrate the oil layer to confirm whether there is structural damage underneath, resulting in hidden cracks being missed. Summary of the Invention

[0007] To solve the above-mentioned technical problems, or at least partially solve them, the present invention provides a road defect intelligent identification method based on multimodal perception.

[0008] The objective of this invention can be achieved through the following technical solution: a road defect intelligent identification method based on multimodal perception, comprising: acquiring multi-band two-dimensional multispectral images and three-dimensional laser point cloud data of the same road surface location.

[0009] The inter-band pixel brightness ratio of each pixel position is extracted as the response combination quantity. Based on the distribution characteristics of the response combination quantity of each pixel position in the two-dimensional response plane, the image domain is divided into the oil stain dominant marker area and the crack candidate marker area.

[0010] The oil-contaminated marked area is inversely projected into three-dimensional space to extract the corresponding local point cloud subset, and the point cloud of the clean road surface outside the oil contamination boundary is also extracted.

[0011] Based on the elevation distribution of the point cloud of the clean road surface outside the oil pollution boundary, a local elevation baseline is constructed. The relative elevation difference distribution field is obtained by subtracting the point cloud elevation inside the oil pollution-dominated marked area from the local elevation baseline. The locations where the relative elevation difference is negative and the absolute value of the elevation difference exceeds the limit are extracted as relative elevation difference anomalies.

[0012] Extract the spatial orientation of the outcrop segment of the crack adjacent to the oil-dominated marking area from the crack candidate marking area, obtain the continuous distribution length of the relatively high anomalous points along the spatial orientation, and determine whether there are cracks under the oil cover based on this.

[0013] Combining all the above technical solutions, the positive effects of this invention are as follows: 1. This invention acquires multispectral images and three-dimensional laser point clouds of the same road surface location, divides the oil-stain-dominated marking area and crack candidate marking area according to the pixel brightness ratio between multispectral bands, then projects the oil-stain-dominated marking area inversely to three-dimensional space, uses the point cloud of the clean road surface outside the oil-stain boundary to construct a local elevation baseline, performs relative elevation difference analysis on the interior of the oil-stain-dominated marking area, and uses the elevation change of the point cloud as a physical verification method to verify the spectral marking results, so that the multispectral image and the three-dimensional point cloud actively coordinate in function, effectively improving the accuracy of crack identification in oil-stain-covered scenarios.

[0014] 2. This invention uses three-dimensional point cloud elevation information to perform physical verification of the oil stain marked area at the structural level, which can effectively identify hidden cracks under oil stain coverage. Instead of focusing on distinguishing between oil stains and defects on the surface, it confirms whether there are hidden cracks under the oil stains by using the physical indicator of whether there is structural subsidence. This solves the limitation that pure spectral analysis cannot penetrate the oil stain layer and reduces the risk of missing cracks in oil stain-covered scenarios to a certain extent. Attached Figure Description

[0015] The present invention will be further described with reference to the accompanying drawings, but the embodiments in the drawings do not constitute any limitation on the present invention. For those skilled in the art, other drawings can be obtained based on the following drawings without creative effort.

[0016] Figure 1 This is a diagram illustrating the implementation steps of the method of the present invention;

[0017] Figure 2 This is a diagram illustrating the implementation steps of S2 of the present invention;

[0018] Figure 3 This is a flowchart illustrating the process of identifying relatively high dissimilarity points in this invention.

[0019] Figure 4 This is a multi-band relative brightness response curve diagram for different road surface areas in this invention;

[0020] Figure 5 This is a curve comparing the local elevation baseline and the measured elevation of the oil-dominated marked area along the crack direction in this invention.

[0021] Figure 6 This is a graph showing the relative elevation difference distribution and abnormal threshold determination in this invention;

[0022] Figure 7 This is a graph showing the cumulative extension length along the crack direction in this invention. Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] Road defects are diverse, but cracks are the most common manifestation. Currently, visual inspection is the mainstream method for road defect identification, which involves acquiring and processing images of the road surface to locate defects. However, in real-world road environments, road surfaces are often contaminated with oil due to vehicle leaks, oil drips, etc. This oil may cover cracks, obscuring them in visible light images. Surface images alone cannot penetrate the oil layer to confirm whether structural damage exists underneath, leading to a risk of missed detections.

[0025] Therefore, this invention proposes an intelligent road defect identification method based on multimodal perception, see reference. Figure 1 As shown, the process includes the following steps: S1, acquiring multi-band two-dimensional multispectral images and three-dimensional laser point cloud data of the same road surface location.

[0026] Considering that oil stains and cracks have different spectral response characteristics in the short-wave infrared, near-infrared and visible light bands, it is necessary to acquire multi-band spectral images to obtain spectral information that can distinguish between the two. However, relying solely on a single spectral image is still insufficient to penetrate the oil stain layer and confirm whether there is structural damage underneath. Therefore, this invention simultaneously adds three-dimensional laser point cloud data. By using the changes in road surface elevation reflected by the point cloud, the oil stain area can be physically verified from a structural level, thereby achieving complementary perception of multi-modal data.

[0027] Specifically, multi-band two-dimensional multispectral images and three-dimensional laser point cloud data are acquired in the following way: the multispectral imaging device and the laser scanning device are installed on the same acquisition platform and their relative spatial positions are kept fixed to ensure that there is a definite coordinate transformation relationship between the two devices when the spectral images and point cloud data are subsequently reverse-projected.

[0028] The data acquisition platform travels along the road, triggering the synchronous acquisition of multi-band two-dimensional multispectral images and three-dimensional laser point cloud data at the same road surface location at fixed spatial intervals each time it travels, ensuring that the two correspond to each other frame by frame in space.

[0029] The data of each band channel of each frame of multispectral image is associated and stored with the corresponding three-dimensional laser point data at the time of acquisition, using the acquisition time identifier and spatial location identifier as indexes.

[0030] S2. Extract the inter-band pixel brightness ratio of each pixel position as the response combination quantity. Based on the distribution characteristics of the response combination quantity of each pixel position in the two-dimensional response plane, divide the image domain into the oil stain dominant marker area and the crack candidate marker area.

[0031] After acquiring multi-band multispectral images of the same road surface location, the differences in spectral response characteristics between oil stains and cracks in different bands can be used to make a preliminary distinction, thereby identifying areas that may be covered by oil stains and providing spatial location information for further verification of hidden cracks.

[0032] See Figure 2 As shown, the specific implementation is carried out according to the following sub-steps: S21, for each pixel position of the two-dimensional multispectral image, the light signal received by the multispectral sensor in the multispectral imaging device at that position is converted from analog to digital and output as the pixel brightness value corresponding to that band, thereby obtaining its respective pixel brightness values ​​in the short-wave infrared band, near-infrared band and visible red light band.

[0033] S22. Divide the brightness value of each pixel location in the short-wave infrared band by the brightness value in the near-infrared band to obtain the first response value. This ratio reflects the relative reflection characteristics of the pixel location in the short-wave infrared and near-infrared bands. Oil stains have strong absorption characteristics in this band, and their ratio will be significantly lower. The response characteristics of cracks in this band are basically the same as those of normal road surfaces.

[0034] Dividing the brightness value of the same pixel location in the near-infrared band by the brightness value in the visible red band yields the second response value. This ratio reflects the relative reflection characteristics of the pixel location in the near-infrared and red bands. Crack areas have a lower ratio because near-infrared reflection decreases while red reflection is relatively enhanced. In contrast, the response characteristics of oil stains in this band are basically the same as those of normal road surfaces.

[0035] S23. The first response quantity and the second response quantity together constitute the spectral response combination quantity of the pixel position, which is used as a feature parameter to characterize the spectral difference between oil stains and cracks.

[0036] To further illustrate the formation process of the first and second response quantities, pixels from normal road surface areas, oil-stained areas, and cracked areas were selected from the same road surface sample. The normalized mean brightness values ​​for each band were statistically analyzed, and the combined response quantities were calculated, resulting in the data shown in Table 1. The corresponding multi-band brightness response curves are shown in Table 1. Figure 4 Table 1 and Figure 4 This illustrates the relative spectral variation patterns in different regions, but is not a limitation on specific thresholds or brightness ranges in this invention.

[0037] Table 1. Band brightness and response combination data for different road surface areas.

[0038]

[0039] From Table 1 and Figure 4 It is known that the brightness decrease of the oil-stained area is more significant in the short-wave infrared band compared to the near-infrared band, resulting in a lower first response value; similarly, the brightness decrease of the crack area is more significant in the near-infrared band compared to the red band, resulting in a lower second response value. Based on these differences, this embodiment combines the ratio of short-wave infrared brightness to near-infrared brightness and the ratio of near-infrared brightness to red light brightness into a combined response value, enabling the oil stain and crack to deviate in different directions in the two-dimensional response plane.

[0040] S24. Using the first response value of each pixel position as the horizontal axis coordinate and the second response value as the vertical axis coordinate, map all pixel positions onto a two-dimensional response plane composed of the horizontal and vertical axes to form a distribution point set of all pixels in the response plane.

[0041] S25. Divide the response plane into equally spaced grids along the horizontal and vertical axes, count the number of distribution points falling into each grid cell, and select the grid cell containing the most distribution points. Use the position of the geometric center of this grid cell in the response plane as the background reference point. This is because in a road scene, the normal road surface pixels occupy the largest proportion of the image area, so the distribution points of background pixels are most concentrated in the response plane, and the corresponding grid cell contains the most distribution points.

[0042] S26. For each pixel position, subtract the horizontal axis coordinate value of the background reference point from the horizontal axis coordinate value of its distribution point to obtain the horizontal axis deviation of the pixel position; subtract the vertical axis coordinate value of the background reference point from the vertical axis coordinate value of its distribution point to obtain the vertical axis deviation of the pixel position; where a positive deviation value indicates that the pixel is higher than the background reference point in the corresponding direction, and a negative deviation value indicates that the pixel is lower than the background reference point in the corresponding direction.

[0043] The average absolute value of the deviation of all distribution points in the horizontal direction is calculated as the horizontal deviation reference value; the average absolute value of the deviation of all distribution points in the vertical direction is calculated as the vertical deviation reference value. The horizontal deviation reference value reflects the overall average level of the deviation of all pixels in the image from the background reference point in the horizontal direction, and the vertical deviation reference value reflects the overall average level of the deviation of all pixels in the image from the background reference point in the vertical direction.

[0044] S27. Attribute labeling based on the horizontal and vertical deviations of each pixel position: When the horizontal deviation of a pixel position is negative (i.e., the pixel is lower than the background reference point in the horizontal direction) and the absolute value of the vertical deviation is less than the vertical deviation reference value (i.e., the degree of deviation of the pixel from the background in the vertical direction is lower than the average level of the whole image), it indicates that the first response value of the pixel is low and the second response value is close to the background, which is consistent with the spectral behavior of oil stains. The pixel position is then labeled as an oil stain attribute.

[0045] When the vertical axis deviation of a pixel is negative (i.e., the pixel is lower than the background reference point in the vertical direction) and the absolute value of the horizontal axis deviation is less than the horizontal axis deviation reference value (i.e., the degree of deviation of the pixel from the background in the horizontal direction is lower than the average level of the whole image), it indicates that the second response value of the pixel is low and the first response value is close to the background, which is consistent with the spectral behavior of cracks. The pixel position is then marked as a crack attribute.

[0046] S28. After traversing all pixel positions and completing attribute marking, the connected regions formed by pixels with the same oil stain attribute and spatially adjacent pixels are marked as oil stain dominant marking regions, and the connected regions formed by pixels with the same crack attribute and spatially adjacent pixels are marked as crack candidate marking regions.

[0047] To implement the above steps, the grid size is determined as follows: First, the first and second response values ​​of each distribution point in the response plane are normalized to eliminate the interference of the difference in the numerical range of the two coordinate axes on the distance calculation.

[0048] Then, the spatial distance between each point in the normalized response plane and its nearest neighbor is calculated, and the distribution of all nearest neighbor distances is statistically analyzed. The median value of all nearest neighbor distances is used as the grid size. This is because in road scenes, normal road surface pixels account for the majority, and their distribution points are concentrated to form the largest cluster. The median value reflects the average degree of clustering of the distribution points. Using it as the grid size can ensure that the largest cluster representing the background falls within the same grid to accurately locate the background reference point, and it will not obscure the distribution differences between different areas due to the excessive size.

[0049] S3. Inversely project the oil-stain-dominated marked area into three-dimensional space to extract the corresponding local point cloud subset, and extract the point cloud of the clean road surface outside the oil-stain boundary.

[0050] After S2 region segmentation, the oil-dominated marked area has been identified in the image domain. However, the marking results of this area are based solely on two-dimensional spectral information, and it is impossible to determine whether there are hidden cracks underneath. Given that cracks are structural damage, most cracks are accompanied by local subsidence or misalignment of the road surface elevation. Oil, on the other hand, adheres to the road surface and is typically on the order of micrometers to millimeters thick, which is negligible relative to the accuracy of road elevation detection and hardly changes the road surface elevation. This difference makes the elevation information in three-dimensional space a physical basis for distinguishing between the two. Therefore, it is necessary to reverse-project the oil-dominated marked area in the image domain to three-dimensional space to extract the corresponding local point cloud data of this area, and simultaneously extract the point cloud of the clean road surface outside the oil-stained boundary as an elevation reference. By comparing the elevation distribution differences between the inside and outside of the oil-stained area, it is possible to verify whether there are structural cracks.

[0051] S31, extract a subset of local point clouds of oil stains.

[0052] S311. Extract all boundary pixel positions of the oil pollution dominant marker area in the two-dimensional multispectral image. These boundary positions are derived from the outer contour of the set of pixel positions corresponding to the oil pollution response clusters after clustering in step S2. Using the pixel coordinates of each boundary pixel position as input, according to the intrinsic parameter calibration relationship of the multispectral imaging device (including the mapping relationship between the pixel coordinate system and the image physical coordinate system with the camera optical center as the origin), the coordinates of each boundary pixel are converted into a direction vector with the imaging device optical center as the origin. This direction vector represents the spatial ray direction from the camera optical center and pointing to the actual road surface position corresponding to the boundary pixel position, which is used for spatial pointing guidance when tracing the two-dimensional image boundary to three-dimensional space.

[0053] S312. Based on the rigid installation parameters between the multispectral imaging device and the laser scanning device, i.e., the spatial relative positional relationship between the coordinate systems of the two devices, including the translation along the three coordinate axes and the rotation around the three coordinate axes, the direction vectors corresponding to the boundary pixel positions obtained in S311 are transformed one by one to the laser scanning coordinate system with the laser scanning device as the origin, so that the direction vectors are transformed from pointing relative to the imaging device to pointing relative to the laser scanning device; during the transformation, rotation is applied in sequence to unify the pointing of the coordinate axes of the two devices, and then translation is applied to unify the origin position of the coordinate systems of the two devices, so that the direction vector of each boundary pixel position is expressed in the laser scanning coordinate system.

[0054] Then, combining the spatial location identifier recorded at the synchronous acquisition time, that is, the real-time position of the acquisition platform in the world coordinate system, the real-time position of the acquisition platform in the world coordinate system is used as the starting point of the ray, and the direction vector of each boundary pixel position transformed into the laser scanning coordinate system is used as the direction of the ray. The ray is traced along the ray direction to the spatial position where it intersects with the road surface elevation surface reflected by the three-dimensional laser point cloud data. This elevation surface is directly provided by the spatial coordinates of the point cloud data. The obtained spatial position is the projection point of the boundary pixel position in the world coordinate system.

[0055] Finally, after traversing all boundary pixel positions, all projection points are connected sequentially to form the spatial projection boundary of the oil-dominated marked area in the world coordinate system.

[0056] S313. Using the spatial projection boundary as the intercept contour, extract the point cloud data falling within the coverage area of ​​the intercept contour from the three-dimensional laser point cloud data of the entire road section, and form a local point cloud subset corresponding to the oil pollution-dominated marking area.

[0057] S32, extract point clouds of clean road surface outside the boundary of oil contamination.

[0058] S321. Outside the spatial projection boundary formed in S313, a buffer width is configured to extend outward along the horizontal plane to form an annular interception window located outside the spatial projection boundary. The inner boundary of the window is the spatial projection boundary of the oil-stain-dominated marking area, and the outer boundary is the outer boundary after the buffer width is extended outward. The buffer width is used to ensure that the interception range covers a sufficient area of ​​clean road surface point cloud outside the oil-stain-dominated marking area. Specifically, the buffer width can be 10 times the average distance between adjacent point cloud points in the three-dimensional laser point cloud.

[0059] S322. Extract all point cloud data falling within the range of the annular interception window from the three-dimensional laser point cloud data of the entire road section. Since the annular window is located completely outside the spatial projection boundary of the oil pollution-dominated marking area and is separated from the oil pollution-covered area by the spatial projection boundary, the point cloud falling within the range of the annular window is not affected by the oil pollution coverage and belongs to the clean road surface point cloud outside the oil pollution boundary. It is used as the reference sample for constructing the local elevation baseline in subsequent S4.

[0060] S4. Based on the elevation distribution of the point cloud of the clean road surface outside the oil pollution boundary, construct a local elevation baseline. Subtract the point cloud elevation inside the oil pollution-dominated marked area from the local elevation baseline to obtain the relative elevation difference distribution field. Extract the locations where the relative elevation difference is negative and the absolute value of the elevation difference exceeds the limit as relative elevation difference anomalies.

[0061] After capturing the point cloud of the clean road surface outside the oil spill boundary, to determine whether there are hidden cracks beneath the oil spill-dominant marked area, it is necessary to compare the point cloud elevation inside the oil spill-dominant marked area with the elevation of the point cloud of the clean road surface. However, due to the bumps during the acquisition platform's movement and the design slope of the road itself, there is a systematic deviation in the absolute elevation of the point cloud at different spatial locations within the same road segment, making it impossible to directly use fixed elevation values ​​for comparison. Therefore, it is necessary to construct a local elevation baseline covering all spatial locations within the oil spill-dominant marked area, using the elevation distribution of the clean road surface point cloud outside the oil spill boundary as a reference. This baseline represents the theoretical reference surface for the elevation of the area if no cracks occur. By comparing the actual elevation inside the oil spill area with the baseline, the absolute elevation drift caused by vehicle bumps and road slope can be eliminated, highlighting the local relative subsidence caused by cracks.

[0062] As a preferred embodiment of the present invention, the local elevation baseline is constructed as follows: the point cloud of the clean road surface outside the oil pollution boundary is divided into multiple azimuth sectors according to its spatial orientation around the oil pollution-dominant marking area. The purpose of dividing the sector is to obtain elevation references from each direction surrounding the oil pollution area, so as to avoid the uneven offset of the baseline inside the oil pollution area caused by relying on only a single direction. In each azimuth sector, the point cloud point closest to the boundary of the oil pollution-dominant marking area is selected as the elevation control point of that azimuth. Selecting the closest point is to make the elevation reference points of each direction close to the boundary of the oil pollution area, so as to minimize the error accumulation caused by the intermediate transition zone when transferring to the inside of the oil pollution area.

[0063] Since the point cloud within the oil-contaminated marked area does not possess the elevation attributes of the clean road surface, its theoretical elevation value cannot be directly obtained. However, the elevation value of the point cloud of the clean road surface at the boundary of the oil-contaminated area is known, and there is spatial continuity rather than abrupt change between the spatial locations within the oil-contaminated area and the boundary clean road surface. Therefore, an elevation transfer method is adopted, using the geometric center of the oil-contaminated marked area as the convergence reference point for each transfer direction. The geometric center position is the average value of the coordinates of all boundary pixels of the oil-contaminated marked area. The direction of the spatial line connecting each elevation control point to the geometric center is taken as the elevation transfer direction. This direction represents the path of the clean road surface elevation information extending step by step from the boundary to the interior of the oil-contaminated area.

[0064] Along each elevation transfer direction, the elevation values ​​of the elevation control points are transferred point by point into the oil-contaminated marked area, so that each spatial location within the oil-contaminated area obtains a theoretical baseline value calculated from the elevation of the boundary clean road surface. During the transfer process, the elevation values ​​of each elevation control point are used as the starting elevation. The closer each point cloud is to the starting boundary, the closer the assigned elevation value is to the starting elevation. The farther away it is, the smaller the constraint effect of the starting elevation. The assigned elevation value gradually approaches the endpoint reference value when the transfer direction extends into the oil-contaminated area.

[0065] The transfer is carried out in batches as follows: First batch transfer - select the elevation transfer direction of the point cloud with clean road surface on the opposite side as the first batch transfer direction, and use the elevation value of the elevation control point at the boundary on the opposite side as the endpoint reference value of the direction. Complete the assignment of all point cloud points in the direction according to steps 1 to 3 below to form elevation coverage of part of the internal area of ​​the oil pollution-dominated marking area.

[0066] Subsequent transfer—For elevation transfer directions where there is no clean road surface point cloud on the opposite side, after the first batch of transfer directions form an overlapping coverage area within the oil-dominated marking area, the median value of the elevation values ​​of the point cloud points repeatedly assigned by multiple directions within the overlapping area is taken as the endpoint reference value, and then the transfer in that direction is performed according to steps 1 to 3.

[0067] The specific transfer assignment is implemented according to the following steps: Step 1: For any point cloud point in the current transfer direction, obtain the spatial distance of the point cloud point from the elevation control point along the transfer direction, and at the same time obtain the total transfer distance from the starting boundary to the end point of the oil pollution-dominated marking area in the transfer direction.

[0068] Step 2: Determine the assigned elevation value of the point cloud based on the proportion of the spatial distance obtained in Step 1 to the total transmission distance.

[0069] In one operational example, the difference between the starting elevation and the endpoint reference value is first calculated, and this difference is multiplied by the ratio to obtain the transition offset.

[0070] Then, the initial elevation is added to the transition offset to obtain the assigned elevation value for the point cloud point.

[0071] When the point cloud point is located at the starting boundary, the scale is zero, and the assigned elevation value is equal to the starting elevation; when the point cloud point is located at the extension end point, the scale is 1, and the assigned elevation value is equal to the end reference value; when the point cloud point is located in the middle position, the assigned elevation value transitions linearly with distance between the starting elevation and the end reference value.

[0072] Step 3: Perform Step 1 and Step 2 independently for each point cloud point encountered along the transmission direction until a point cloud point that has been assigned a value by another direction is encountered, at which point the transmission in that direction is stopped. If a point cloud point that has been assigned a value by another direction is encountered at the starting boundary, it means that the point cloud point in that direction has been covered by the transmission in other directions, and there is no need to repeat the execution. That direction is automatically skipped. The elevation value of each assigned point cloud point is calculated independently based on its own position from the starting boundary, without depending on the assigned value of the previous point cloud point, thereby avoiding the accumulation of errors point by point along the transmission path.

[0073] After traversing all transmission directions, for point cloud points within the oil-dominated marked area that have been repeatedly assigned values ​​in multiple directions, the median value of the assigned values ​​in each direction is taken as the final elevation value of the point cloud point to eliminate possible single-direction transmission deviations; for point cloud points that have not been assigned values ​​in any direction, the elevation values ​​of adjacent assigned point cloud points are taken to complete the data, ensuring that every spatial location within the oil-polluted area has a corresponding baseline elevation value.

[0074] In the example applied to the above elevation completion, for point cloud points that have not been assigned values ​​in any direction, a search neighborhood is constructed with the point cloud point as the center and the configured search radius. The search radius is determined based on the average distance between adjacent point cloud points in the 3D laser point cloud, specifically twice the average distance. The elevation values ​​of the point cloud points with assigned values ​​in the search neighborhood are taken, and the baseline elevation value for completion is calculated by linear weighting using the reciprocal of the spatial distance between each point and the point to be completed. If there are no point cloud points with assigned values ​​in the search neighborhood, the search radius is expanded by 1.5 times the current search radius until at least three point cloud points with assigned values ​​are found before completion.

[0075] The final baseline elevation values ​​of all point cloud points are combined to form the local elevation baseline.

[0076] After the local elevation baseline is constructed, a profile line can be selected along the spatial direction of the suspected crack outcrop, and the measured elevation inside the oil-contaminated area along this profile line can be compared with the local elevation baseline. In one implementation example, the elevation data and relative elevation difference determination results obtained along the profile line are shown in Table 2, and the comparison curve between the local elevation baseline and the measured elevation is shown in Table 2. Figure 5 As shown, the relative elevation difference and abnormal threshold judgment curves are as follows: Figure 6 As shown.

[0077] Table 2. Data on the determination of elevation and relative elevation difference of oil spill area profile.

[0078]

[0079] From Table 2 and Figure 5 It can be seen that the local elevation baseline changes continuously with the road slope, while the measured elevation of the oil-contaminated area shows a continuous downward deviation in the middle of the profile. Further combining... Figure 6 It can be seen that when the relative elevation difference is negative and its absolute value exceeds the limit of smoothness fluctuation of the clean road surface, the location is marked as a point of abnormal relative elevation difference. Therefore, this invention does not rely solely on the spectral dark spots of oil stains as the basis for disease judgment, but rather performs a structural-level physical verification of the spectral marking results by measuring the degree of depression of the actual point cloud elevation within the oil stain area relative to the local elevation baseline.

[0080] Further, see Figure 3 As shown, after the local elevation baseline is constructed, elevation difference analysis can be performed on the spatial locations within the oil-contaminated marked area based on this baseline. The specific process is as follows: Using the point cloud of the clean road surface outside the oil contamination boundary as the reference area for road surface flatness, all point cloud points in this area are traversed to obtain the elevation value of each point cloud point. The difference between the maximum and minimum values ​​is used as the road surface flatness fluctuation limit. This limit reflects the normal elevation fluctuation range of the clean road surface itself caused by factors such as aggregate unevenness and construction unevenness. It is used to define the boundary between normal road surface fluctuation and structural subsidence, and serves as the criterion for subsequent judgment on whether the point cloud in the oil contamination area belongs to abnormal subsidence.

[0081] For each spatial location within the oil-contaminated marked area, obtain the actual point cloud elevation value at that location, and find the baseline elevation value corresponding to the same horizontal plane coordinates on the local elevation baseline.

[0082] Calculate the difference between the actual point cloud elevation value and the baseline elevation value to obtain the relative height difference at that location.

[0083] The relative height differences of all spatial locations are organized into a relative height difference distribution field. This distribution field reflects the elevation deviation of each location within the oil-polluted marked area relative to the theoretical reference surface, where negative values ​​indicate that the location is below the theoretical reference surface.

[0084] By traversing every location in the relative elevation difference distribution field, locations where the relative elevation difference is negative and the absolute value of the relative elevation difference exceeds the road surface smoothness fluctuation limit are marked as relative elevation difference anomaly points. These anomaly points are the spatial locations where structural subsidence may exist.

[0085] It should be noted that 3D laser point cloud measurements measure the elevation of the road surface reflective surface. Although oil stains form a covering film when they adhere to the road surface, oil stains in road scenes usually originate from vehicle drips, and the thickness of the oil film is typically in the micrometer range. Compared to the ranging accuracy of 3D lidar, its impact on elevation measurement is negligible. Furthermore, even considering the influence of the oil film on laser ranging, the oil stain coverage causes the elevation measured by the laser point cloud to be higher than the actual road surface, while the cracks cause the road structure to sink, making the point cloud elevation lower than the normal road surface. That is, the elevation deviation introduced by the oil film is opposite to the direction of structural sinking caused by the cracks, and will not lead to the cracks being misjudged as normal road surfaces.

[0086] S5. Extract the spatial orientation of the crack outcrop adjacent to the oil-dominant marking area from the crack candidate marking area, obtain the continuous distribution length of relatively high difference points along the spatial orientation, and determine whether there are cracks under the oil cover.

[0087] After obtaining relatively high anomalies in S4, although these anomalies reflect the elevation subsidence within the oil-dominated marked area, the subsidence points may originate from cracks or defects caused by aggregate loss, uneven local compaction, etc. The elevation anomalies of a single point cannot distinguish between these two situations. Considering that the characteristic of cracks is their continuous linear extension in spatial distribution, this invention uses the direction of the exposed cracks in the crack candidate marked area as a directional guide to verify whether the high anomalies within the oil-contaminated area form a continuous linear arrangement along this direction in order to confirm the existence of cracks. The specific verification process is as follows: First, in the crack candidate marked area, the pixels that are connected to the boundary of the oil-dominated marked area are selected as the adjacent pixel set. This set represents the exposed segment of the crack extending from the outside of the oil-contaminated area to the boundary of the oil-contaminated area, providing a reference for subsequently determining the direction of the crack crossing the oil-contaminated area.

[0088] Then, based on the set of adjacent pixels, the process extends point by point along the spatial connectivity between pixels, recording the number of pixels passed from the starting point to the ending point as the extension length. This length reflects the scale of each connectivity direction. The longer the extension length of the connectivity direction, the higher the reliability of its direction. The connectivity direction with the longest extension length is selected as the crack outcrop segment, and its overall extension direction is determined as the spatial direction of the crack verification inside the oil pollution area.

[0089] Next, within the oil-contaminated marked area, multiple parallel scanning trajectory lines are set along the aforementioned spatial direction, with each scanning trajectory line arranged row by row along this direction. The purpose of scanning is to cover the entire spatial range within the oil-contaminated area, avoiding any omission of possible crack responses. Along each scanning trajectory line, each spatial location passed through in that row is checked sequentially to see if it is marked as a relatively high anomaly point. High anomaly points that appear consecutively on the same scanning trajectory line are grouped into the same continuous segment, and the number of anomalies contained in each continuous segment is recorded as the continuous distribution length of that row.

[0090] Finally, the continuous distribution lengths recorded on each scanning trajectory line are accumulated in the order of extension along the direction of the crack to obtain the cumulative extension length along the direction of the crack. This cumulative extension length reflects the overall distribution scale of high-difference points along the crack direction. If high-difference points are scattered in isolation and do not extend continuously along the direction of the crack, the accumulation result will not form an effective length. When the cumulative extension length exceeds the configured length threshold, it indicates that the high-difference points have formed a sufficiently long continuous linear arrangement along the crack direction, and their spatial distribution characteristics are consistent with the morphological characteristics of the crack. Based on this, it is determined that there is a crack under the oil pollution cover. The length threshold can be determined according to the smallest statistical unit of crack length in the maintenance assessment standard corresponding to the grade of the road to be identified. Conversely, it indicates that the high-difference points have not formed a continuous linear arrangement along the direction of the crack, which does not meet the spatial distribution characteristics of the crack. Based on this, it is determined that there is no crack under the oil pollution cover.

[0091] After extracting relatively high-discrepancy anomalies, further scanning can be performed along the spatial orientation of the crack outcrop to determine whether the anomalies form a continuous linear extension. In one implementation example, the continuous distribution length and cumulative extension length obtained along multiple scanning trajectories are shown in Table 3, and the corresponding cumulative extension length analysis curves are shown in Table 3. Figure 7 As shown.

[0092] Table 3. Continuity determination data for anomalies along the crack direction.

[0093]

[0094] From Table 3 and Figure 7 It can be seen that if the relatively high-difference anomaly points are only isolated depressions, their single-line continuous length and cumulative extension length are difficult to increase continuously; if the anomaly points are continuously arranged along the spatial direction of the exposed section of the crack, the cumulative extension length will gradually increase with the scanning trajectory. When the cumulative extension length exceeds the configured length threshold, it can be determined that there is a hidden crack continuous with the exposed crack under the oil stain cover; otherwise, it can be determined that no continuous structural depression that meets the crack morphology requirements has been formed in the oil stain cover area.

[0095] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, in the form of a computer program product.

[0096] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0097] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for intelligent identification of road defects based on multimodal perception, characterized in that, include: Acquire multi-band two-dimensional multispectral images and three-dimensional laser point cloud data of the same road surface location; The inter-band pixel brightness ratio of each pixel position is extracted as the response combination quantity. Based on the distribution characteristics of the response combination quantity of each pixel position in the two-dimensional response plane, the image domain is divided into oil-stain dominant marking area and crack candidate marking area. The oil-contaminated marked area is inversely projected into three-dimensional space to extract the corresponding local point cloud subset, and the point cloud of the clean road surface outside the oil contamination boundary is also extracted. Based on the elevation distribution of the point cloud of the clean road surface outside the oil pollution boundary, a local elevation baseline is constructed. The relative elevation difference distribution field is obtained by subtracting the point cloud elevation inside the oil pollution-dominated marked area from the local elevation baseline. The locations where the relative elevation difference is negative and the absolute value of the elevation difference exceeds the limit are extracted as relative elevation difference anomalies. Extract the spatial orientation of the outcrop segment of the crack adjacent to the oil-dominated marking area from the crack candidate marking area, obtain the continuous distribution length of the relatively high anomalous points along the spatial orientation, and determine whether there are cracks under the oil cover based on this.

2. The intelligent road defect identification method based on multimodal perception as described in claim 1, characterized in that: The multi-band two-dimensional multispectral images and three-dimensional laser point cloud data were obtained using the following methods: The multispectral imaging equipment and the laser scanning equipment are installed on the same acquisition platform and kept in a fixed spatial position. The acquisition platform moves along the road and triggers a synchronous acquisition once every fixed spatial interval. The data of each band channel of each frame of multispectral image is associated and stored with the corresponding three-dimensional laser point data at the time of acquisition, using the acquisition time identifier and spatial location identifier as indexes.

3. The intelligent road defect identification method based on multimodal perception as described in claim 1, characterized in that: The combined response quantities are extracted according to the following process: For each pixel location in a two-dimensional multispectral image, obtain its respective pixel brightness value in the short-wave infrared band, near-infrared band, and visible red light band; Divide the brightness value of each pixel location in the short-wave infrared band by the brightness value in the near-infrared band to obtain the first response value, and divide the brightness value of the same pixel location in the near-infrared band by the brightness value in the visible red light band to obtain the second response value. The first and second response values ​​are used as the combined response values ​​for the pixel position.

4. The intelligent road defect identification method based on multimodal perception as described in claim 3, characterized in that: The image domain is divided into an oil-dominated marker region and a crack candidate marker region. The process includes the following: Using the first response value of each pixel location as the horizontal axis coordinate and the second response value as the vertical axis coordinate, all pixel locations are mapped onto the two-dimensional response plane to form a distribution point set; The response plane is divided into equally spaced grids. The number of distribution points falling into each grid is counted. The geometric center of the grid containing the most distribution points is selected as the background reference point. For each pixel position, calculate the horizontal and vertical deviations of its distribution points relative to the background reference point. Calculate the average of the absolute values ​​of the deviations of all distribution points in the horizontal direction as the horizontal deviation reference value, and the average of the absolute values ​​of the deviations in the vertical direction as the vertical deviation reference value. When the horizontal axis deviation of a pixel position is negative and the absolute value of the vertical axis deviation is less than the vertical axis deviation reference value, it is marked as an oil stain attribute. When the vertical axis deviation of a pixel position is negative and the absolute value of the horizontal axis deviation is less than the horizontal axis deviation reference value, it is marked as a crack attribute. After traversing all pixel positions and completing attribute labeling, the connected regions formed by spatially adjacent pixels with the same oil stain attribute are labeled as oil stain dominant labeling regions, and the connected regions formed by spatially adjacent pixels with the same crack attribute are labeled as crack candidate labeling regions.

5. The intelligent road defect identification method based on multimodal perception as described in claim 1, characterized in that: The specific steps of extracting the corresponding local point cloud subset include: Based on the intrinsic parameter calibration relationship of the multispectral imaging device, the position coordinates of each boundary pixel of the oil-dominated marked area in the two-dimensional multispectral image are converted into direction vectors with the optical center of the imaging device as the origin. Based on the rigid installation parameters between the multispectral imaging device and the laser scanning device, the direction vectors corresponding to the positions of each boundary pixel are transformed into the scanning coordinate system of the laser scanning device. Combined with the spatial position markers recorded at the synchronous acquisition time, the projection points of each boundary pixel position in the world coordinate system are determined. All projection points are connected in sequence to form the spatial projection boundary of the oil-dominated marking area in the world coordinate system. Using the spatial projection boundary as the intercept contour, point cloud data falling within the intercept contour range are extracted from the three-dimensional laser point cloud data of the entire road section to form a local point cloud subset.

6. The intelligent road defect identification method based on multimodal perception as described in claim 5, characterized in that: The specific methods for capturing the point cloud of clean road surface outside the oil spill boundary include: A buffer width is extended along the horizontal plane around the spatial projection boundary to form an annular intercept window located outside the projection boundary. Point clouds falling within the annular interception window are extracted from the three-dimensional laser point cloud data of the entire road section and used as the point cloud of the clean road surface outside the oil pollution boundary.

7. The intelligent road defect identification method based on multimodal perception as described in claim 1, characterized in that: The construction of local elevation baselines specifically includes: The point cloud of the clean road surface outside the oil pollution boundary is divided into multiple directional sectors according to the spatial orientation surrounding the oil pollution-dominant marking area. The point cloud point closest to the boundary of the oil pollution-dominant marking area in each directional sector is selected as the elevation control point of that directional sector. The direction of the line connecting each elevation control point to the geometric center of the oil pollution-dominant marking area is taken as the elevation transfer direction. The elevation value of the elevation control point is transferred point by point from the boundary of the oil pollution-dominant marking area inward along each transfer direction until the extension end point inside the oil pollution-dominant marking area is reached or an already assigned point cloud point is encountered, at which point the transfer in that direction is stopped. For point cloud points that have been repeatedly assigned values ​​in multiple directions, the median value of the assigned values ​​in each direction is taken as the final elevation value; for point cloud points that have not been assigned values, the elevation values ​​of adjacent point cloud points that have already been assigned values ​​are used to complete the elevation value. The final baseline elevation values ​​of all point cloud points are combined to form the local elevation baseline.

8. The intelligent road defect identification method based on multimodal perception as described in claim 1, characterized in that: The extraction process for the relatively high anomalous points is as follows: The point cloud of the clean road surface outside the oil contamination boundary is used as the reference area for road surface smoothness, and the difference between the maximum and minimum elevation values ​​of all point cloud points in this area is used as the limit for road surface smoothness fluctuation. For each spatial location in the oil-polluted marked area, the actual point cloud elevation value is obtained, and the baseline elevation value corresponding to the same horizontal plane coordinate is found on the dynamic elevation baseline. The difference between the two is used to obtain the relative elevation difference value at that location. The relative elevation differences of all locations are organized into a relative elevation difference distribution field according to spatial distribution. Traverse the relative elevation difference distribution field and mark the locations where the relative elevation difference is negative and the absolute value of the relative elevation difference exceeds the road surface smoothness fluctuation limit as relative elevation difference anomalous points.

9. The intelligent road defect identification method based on multimodal perception as described in claim 1, characterized in that: The process of extracting the spatial orientation of the outcrop segment of the crack adjacent to the oil-dominant marker area from the crack candidate marker area includes the following steps: In the candidate marking area of ​​the crack, the pixels that are connected to the boundary of the oil-stain-dominated marking area are selected as the adjacent pixel set, and the longest extension of the adjacent pixel set is taken as the spatial direction of the crack outcrop segment.

10. The intelligent road defect identification method based on multimodal perception as described in claim 9, characterized in that: The process for determining whether cracks exist under the oil stain cover is as follows: Scan line by line along the spatial direction within the oil-dominated marked area and record the continuous distribution length of relatively high anomalous points in each line; The cumulative extension length along the direction is obtained by summing the continuous distribution lengths in each row. When the cumulative extension length exceeds the length threshold, it is determined that there is a crack under the oil stain cover; otherwise, it is determined that there is no crack.

Images