Road damage identification and positioning method fused with multi-source information

Abstract

The invention discloses a road damage identification and positioning method fused with multi-source information, and relates to the technical field of road and bridge facility health monitoring. The method comprises the following steps: S1, collecting multi-source image data of a road bridge through an unmanned aerial vehicle, a wall-climbing robot and / or an industrial camera; and S2, preprocessing the multi-source image data, respectively inputting the multi-source image data into a pre-trained deep learning recognition model, and outputting the type and initial position information of a disease. According to the method, the global view angle of the unmanned aerial vehicle and the local high-definition data of the wall-climbing robot / industrial camera are deeply fused, and the deep learning-based semantic segmentation model is utilized to perform pixel-level recognition on tiny diseases such as cracks, so that the recognition accuracy and reliability are greatly improved, and meanwhile, the recognition efficiency is improved. Image coordinates are accurately mapped to the three-dimensional live-action model through a coordinate transformation formula, centimeter-level space positioning of the disease is achieved, and detection accuracy and operation safety are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of road and bridge facility health monitoring technology, and in particular to a method for road damage identification and location that integrates multi-source information. Background Technology

[0002] As core components of transportation infrastructure, the structural health of roads and bridges directly affects public safety and economic efficiency. Traditional manual inspection methods are not only inefficient and costly, but also highly susceptible to subjective factors, making it difficult to detect hidden defects, especially in dangerous areas such as high-altitude and underwater sections of large bridge structures. With the development of technology, automated inspection technology based on digital image processing has emerged. By using cameras mounted on vehicles or drones to collect images and deep learning algorithms to identify damage, the objectivity and efficiency of inspection have been significantly improved, and this has become an important development direction for smart infrastructure maintenance.

[0003] Existing automated inspection technologies mostly rely on a single data source. Among them, drone-based aerial photography can quickly cover a large area and acquire macroscopic overall images, which is suitable for the initial screening of road surface defects such as cracks and potholes. However, this method is limited by flight altitude and camera resolution, making it difficult to conduct detailed observations of key parts such as bridge supports, truss interiors, and beam bottoms. Furthermore, the acquired two-dimensional images lack precise spatial dimensional information and cannot be directly used for three-dimensional location and quantitative analysis of damage. This results in assessments that often remain at the level of "whether there is damage" and are insufficient to support repair decisions that require precise spatial information.

[0004] To compensate for the shortcomings of drones in macroscopic inspection, methods have emerged in recent years that use wall-climbing robots or high-resolution fixed industrial cameras to conduct close-range inspections of key parts. These methods can acquire high-definition local images with millimeter-level precision, greatly improving the ability to identify subtle defects. However, these local inspection methods also have obvious drawbacks: their field of view is narrow, they lack global spatial reference, and it is difficult to accurately correspond the discovered damage to the overall structure; data collected from different platforms and at different times are independent of each other, forming "information silos" that cannot be fused, analyzed, and comprehensively evaluated under a unified spatiotemporal benchmark.

[0005] To address these issues, we provide a method for road damage identification and localization that integrates multi-source information. Summary of the Invention

[0006] The purpose of this invention is to provide a road damage identification and localization method that integrates multi-source information. By combining a multimodal deep learning framework with a unified spatiotemporal reference for precise mapping, it solves the problems of isolated and difficult-to-integrate multi-source detection data and lack of accurate three-dimensional spatial localization of damage in the prior art.

[0007] To solve the above-mentioned technical problems, the present invention is achieved through the following technical solution:

[0008] This invention discloses a method for road damage identification and localization that integrates multi-source information, comprising the following steps: S1: acquiring multi-source image data of roads and bridges using drones, wall-climbing robots, and / or industrial cameras; S2: preprocessing the multi-source image data and inputting it into a pre-trained deep learning recognition model to output the type and initial location information of the damage; S3: constructing a three-dimensional real-scene model of the road and bridge based on the drone data and using it as a global spatial reference; S4: spatially matching and fusing the initial location information of the damage obtained in S2 from the wall-climbing robot and industrial camera with the three-dimensional real-scene model in S3 to calculate the precise three-dimensional coordinates of the damage under the global spatial reference; S5: automatically calculating the technical condition indicators of the road or bridge based on the fused damage information and outputting a visual evaluation report.

[0009] The present invention is further configured such that the multi-source image data in S1 includes: global orthophoto images collected by a drone, oblique photogrammetric images, and local high-definition images of key parts collected by a wall-climbing robot or an industrial camera.

[0010] The present invention is further configured such that the deep learning recognition model in S2 includes: a convolutional neural network for target detection for processing global UAV images, and a convolutional neural network for semantic segmentation for processing local high-definition images.

[0011] The present invention is further configured such that the semantic segmentation convolutional neural network uses a Focal Loss function during training to address the class imbalance problem between defective pixels and background pixels. The specific formula of the Focal Loss function is as follows:

[0012] FL(p t )=-α t *(1-p t ) γ *log(p t )

[0013] Where, p t α represents the model's predicted probability of the true class. t γ is the class weight balancing factor, and γ is the hard sample focusing parameter.

[0014] The present invention is further configured such that the spatial matching and fusion in S4 specifically involves: solving the transformation parameters from the local image coordinate system to the global three-dimensional model coordinate system using a computer vision image registration algorithm, and then mapping the coordinates of the disease points in the local image to the global coordinate system using a coordinate transformation formula; the coordinate transformation formula is:

[0015] P global =s*R*P local +T

[0016] Among them, P local Let P be the homogeneous coordinates of the disease point in the local image coordinate system. global R represents the transformed global 3D coordinates, T is the rotation matrix, and s is the translation vector.

[0017] The present invention is further configured such that the object detection convolutional neural network and the semantic segmentation convolutional neural network are fused at the feature level; the fusion method is: the extracted global features F global and local features F local The weighted concatenation is performed using the following formula:

[0018]

[0019] Among them, W g and W l The weight matrix is ​​a learnable matrix. F represents the vector concatenation operation. fused This is the fused multimodal feature vector.

[0020] The present invention is further configured such that the spatial matching of the wall-climbing robot preferentially uses the positioning module data to directly align with the three-dimensional model, and assists in the verification and refinement of the coordinate transformation formula.

[0021] The present invention is further configured such that, during the training of the deep learning recognition model in S2, environmental temperature data is introduced as one of the input features to optimize the model's recognition performance for typical diseases in cold regions.

[0022] The present invention is further configured such that, in step S2, a generative adversarial network (GAN) is used to augment the acquired images to expand the training sample set and improve the model's generalization ability.

[0023] The present invention is further configured such that the technical condition indicators in S5 are at least one of the Road Surface Damage Index (PCI) and the Ride Quality Index (RQI) for roads, and an overall technical condition score calculated based on the scores of the superstructure, substructure, and bridge deck system for bridges.

[0024] The present invention has the following beneficial effects:

[0025] 1. This invention deeply integrates the global perspective of a drone with local high-definition data from a wall-climbing robot / industrial camera, and uses a semantic segmentation model based on deep learning to perform pixel-level identification of minor defects such as cracks, which greatly improves the accuracy and reliability of identification. At the same time, by using coordinate transformation formulas to accurately map image coordinates to a three-dimensional real-world model, centimeter-level spatial positioning of defects is achieved, which significantly improves the accuracy of detection and the safety of operation.

[0026] 2. This invention integrates multi-source sensing, computer vision, and artificial intelligence algorithms. It adaptively integrates global and local features through feature-level fusion formulas and introduces an environmental parameter optimization model, giving the system excellent robustness and scene adaptability. All damage information is integrated into a unified three-dimensional digital model and automatically generates a quantitative assessment report that conforms to the standards, greatly improving the scientific nature and efficiency of maintenance decisions. Attached Figure Description

[0027] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below.

[0028] Figure 1 This is an overall flowchart of a road damage identification and localization method that integrates multi-source information;

[0029] Figure 2 This is a flowchart of a deep learning recognition model in a road damage identification and localization method that integrates multi-source information.

[0030] Figure 3 This is a flowchart of the spatial matching and fusion sub-process in a road damage identification and localization method that integrates multi-source information. Detailed Implementation

[0031] The technical solutions of the present invention will be described below with reference to the accompanying drawings. The described embodiments are only some embodiments of the present invention, and not all embodiments.

[0032] Example 1

[0033] Please see Figure 1-3A method for road damage identification and localization by fusing multi-source information includes the following steps: S1: Collecting multi-source image data of roads and bridges using drones, wall-climbing robots, and / or industrial cameras; S2: Preprocessing the multi-source image data and inputting them into pre-trained deep learning recognition models to output the type and initial location information of the damage; S3: Constructing a three-dimensional real-scene model of the road and bridge based on the drone data and using it as a global spatial reference; S4: Spatially matching and fusing the initial location information of the damage obtained in S2 from the wall-climbing robot and industrial camera with the three-dimensional real-scene model in S3 to calculate the precise three-dimensional coordinates of the damage under the global spatial reference; S5: Automatically calculating the technical condition indicators of the road or bridge based on the fused damage information and outputting a visual evaluation report.

[0034] Example 2

[0035] Please see Figure 1-3 A method for road damage identification and localization integrating multi-source information includes the following steps: S1: Collecting multi-source image data of roads and bridges through drones, wall-climbing robots, and / or industrial cameras. The multi-source image data in S1 includes: global orthophoto images and oblique photogrammetric images collected by drones, and local high-definition images of key parts collected by wall-climbing robots or industrial cameras; S2: Preprocessing the multi-source image data and inputting them into pre-trained deep learning recognition models to output the type and initial location information of the damage. The deep learning recognition model in S2 includes: a target detection convolutional neural network for processing global drone images and a semantic segmentation convolutional neural network for processing local high-definition images. The semantic segmentation convolutional neural network uses the Focal Loss function during training to solve the problem of class imbalance between damage pixels and background pixels. The specific formula of the Focal Loss function is:

[0036] FL(p t )=-α t *(1-p t ) γ *log(p t )

[0037] Where, p t α represents the model's predicted probability of the true class. t γ is the class weight balancing factor, and γ is the hard sample focusing parameter. The object detection convolutional neural network and the semantic segmentation convolutional neural network are fused at the feature level. The fusion method is as follows: the extracted global features F... global and local features F local The weighted concatenation is performed using the following formula:

[0038]

[0039] Among them, W g and W l The weight matrix is ​​a learnable matrix. F represents the vector concatenation operation. fused To generate the fused multimodal feature vector, the deep learning recognition model in S2 incorporates ambient temperature data as one of its input features during training to optimize its performance in recognizing typical diseases in cold regions. S2 also includes using a Generative Adversarial Network (GAN) to augment the collected images, expanding the training sample set and improving the model's generalization ability. S3: A 3D real-world model of roads and bridges is constructed based on UAV data and used as the global spatial reference. S4: The initial location information of the diseases obtained from the wall-climbing robot and industrial camera in S2 is spatially matched and fused with the 3D real-world model in S3 to calculate the precise 3D coordinates of the diseases under the global spatial reference. Specifically, the spatial matching and fusion in S4 involves using a computer vision image registration algorithm to solve for the transformation parameters from the local image coordinate system to the global 3D model coordinate system, and then using the coordinate transformation formula to map the coordinates of the disease points in the local image to the global coordinate system. The coordinate transformation formula is:

[0040] P global =s*R*P local +T

[0041] Among them, P local Let P be the homogeneous coordinates of the disease point in the local image coordinate system. global R is the transformed global 3D coordinate, T is the rotation matrix, and s is the scale factor. The wall-climbing robot prioritizes direct alignment of the positioning module data with the 3D model for spatial matching, and uses the auxiliary coordinate transformation formula for verification and refinement. S5: Based on the fused defect information, the technical condition indicators of roads or bridges are automatically calculated and a visual evaluation report is output. The technical condition indicators in S5 are at least one of the road surface damage index (PCI) and ride quality index (RQI) for roads, and an overall technical condition score calculated based on the scores of the superstructure, substructure, and bridge deck system for bridges.

[0042] Example 3

[0043] Regular intelligent inspection and precise maintenance decision-making for highway pavement

[0044] During the data acquisition phase, drones equipped with five-lens oblique photography cameras were used to collect high-precision global orthophotos and oblique photography images to construct a complete digital road base map. Simultaneously, wall-climbing robots were deployed to acquire high-resolution local images of key suspected areas (such as bridge expansion joints and ramp connections), achieving millimeter-level resolution.

[0045] The pre-processed UAV global imagery is input into a target detection convolutional neural network (such as YOLO or Faster R-CNN) to quickly locate cracks, pits, repair marks, etc.

[0046] High-resolution images of the wall-climbing robot are input into a semantic segmentation convolutional neural network (such as U-Net or DeepLab) for pixel-level fine segmentation. To address the class imbalance problem where defect pixels are far fewer than background pixels, the network is trained using the Focal Loss loss function, which significantly improves the ability to identify minute cracks.

[0047] To improve the model's generalization ability, a generative adversarial network (GAN) was used during the training phase to augment the limited original disease images, simulating images under different lighting and wear conditions, which effectively expanded the training sample set.

[0048] The model output includes the disease type, severity, and initial location in the local image.

[0049] Using oblique photography data from drones, a high-precision 3D reality model is generated through real-scene 3D modeling technology. This model comes with global geographic coordinates, serving as a global spatial reference for all information.

[0050] For wall-climbing robot data: Since it has a built-in high-precision positioning module (such as UWB or laser SLAM), the spatial pose of each frame of the acquired image is directly aligned with the 3D model to achieve preliminary positioning.

[0051] For industrial camera data: The transformation parameters from the local image coordinate system to the global 3D model coordinate system are accurately solved using computer vision image registration algorithms (such as SIFT and ORB feature matching).

[0052] Using coordinate transformation formula P global =s*R*P local +T maps the local coordinates of all disease points identified in step S2 to the global three-dimensional coordinate system, obtaining their precise latitude, longitude, and elevation information. The positioning data of the wall-climbing robot can be used to verify and refine the transformation result, ensuring positioning accuracy.

[0053] The system automatically integrates all located defect information and calculates the Pavement Damage Index (PCI) and Ride Quality Index (RQI) according to industry standards. The final generated visual assessment report not only includes data reports but also overlays various defects onto the real-world model in the form of 3D point clouds or annotations, providing an immersive "one road, one file" digital asset view and providing accurate basis for maintenance decisions.

[0054] Example 4

[0055] Comprehensive health monitoring and precise management of large urban overpass complexes

[0056] The approach employs a combination of "global UAV scanning + detailed local inspection by robots / fixed cameras." UAVs execute automated flight paths to collect oblique photographic images of the overpass, acquiring macroscopic information such as the bridge deck, beams, and pier tops. Simultaneously, wall-climbing robots are attached to the sides of the piers to detect vertical cracks, and industrial cameras are used for close-up photography of critical load-bearing components such as supports and rivets.

[0057] In terms of model architecture, a feature-level fusion strategy is adopted. The global features (F) extracted by the target detection network processing UAV imagery are then used. global (e.g., overall deformation trend of the bridge deck, distribution pattern of defects) and local features extracted by semantic segmentation networks for processing local images (F lobal (e.g., crack width, details of concrete spalling) weighted splicing This fusion allows the model to incorporate both macro-context and micro-details, improving the accuracy of disease identification in complex contexts.

[0058] To address the characteristics of winter in northern cities, historical ambient temperature data from the same period was incorporated as input features when training the recognition model. This enabled the model to better learn the correlation between temperature stress and typical bridge defects (such as freeze-thaw cracks), thereby optimizing its recognition performance in low-temperature environments.

[0059] We constructed a millimeter-precise 3D model of the overpass complex, clearly showing all structural components.

[0060] Every defect (such as support displacement or pier crack) discovered by the robot and fixed camera is accurately mapped into the 3D model through image registration and coordinate transformation, realizing a one-to-one correspondence between defects and structural components and 3D spatial positioning.

[0061] Based on the "Standards for the Assessment of Technical Condition of Highway Bridges," the system calculates scores for the three major modules: superstructure, substructure, and bridge deck system, and automatically calculates the overall technical condition score. The report clearly marks the location, type, size, and impact level of each defect on structural safety, achieving precise management from "overall rating" to "defect level."

[0062] Example 5

[0063] Special Inspection and Causal Analysis of Winter Freezing Damage to Highways in Cold Regions

[0064] During the early spring when temperatures rise but there is still a window of opportunity where daytime thawing and nighttime freezing occur, drones are used to quickly scan the entire route and acquire macroscopic images of the road surface. Low-temperature resistant climbing robots are deployed to collect local high-definition images of sections known to be prone to frost heave and frost heave (such as the junction of roadbed fill and cut, and sections with poor drainage) to capture early subtle changes.

[0065] Temperature data from the cold regions (such as historical and real-time ground and air temperatures) were used as key prior knowledge and input into the deep learning recognition model. By learning the strong correlation between temperature and diseases (such as frost heave, surface spalling, and cracks of specific shapes), the model significantly improved its ability to identify typical diseases in cold regions and effectively distinguished between frost damage and mechanical damage.

[0066] Establish a three-dimensional model of the highway with elevation information, with particular emphasis on representing the changes in road surface smoothness.

[0067] All suspected frost damage points were precisely located in three dimensions, recording not only their planar positions but also their elevation changes (such as the height of the uplift point and the depth of the subsidence point).

[0068] The system not only outputs the Pavement Damage Index (PCI), but more importantly, it generates a special analysis report on frost damage. This report visualizes the distribution patterns and severity of frost damage by overlaying and analyzing the distribution of damage with temperature data and geographical information (such as slope and orientation), scientifically inferring the causes (such as insufficient drainage, insulation failure, etc.), and providing direct data support for taking targeted and predictive maintenance measures (such as adding drainage facilities and replacing insulation materials).

[0069] The preferred embodiments of the present invention disclosed above are only for the purpose of illustrating the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation described herein. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can better understand and utilize the present invention.

Claims

1. A method for road damage identification and localization by fusing multi-source information, characterized in that: Includes the following steps: S1: Acquire multi-source image data of roads and bridges using drones, wall-climbing robots, and / or industrial cameras; S2: Preprocess the multi-source image data and input them into the pre-trained deep learning recognition model to output the type and initial location information of the disease; S3: Construct a three-dimensional real-scene model of the road and bridge based on the UAV data, and use it as a global spatial reference; S4: Spatial matching and fusion of the initial location information of the disease obtained from the wall-climbing robot and industrial camera in S2 with the 3D real scene model in S3, to calculate the precise 3D coordinates of the disease under the global spatial reference. S5: Based on the fused defect information, automatically calculate the technical condition indicators of roads or bridges and output a visual assessment report.

2. The method for road damage identification and localization by fusing multi-source information according to claim 1, characterized in that: The multi-source image data in S1 includes: global orthophotos and oblique photographs collected by UAVs, and high-resolution local images of key parts collected by wall-climbing robots or industrial cameras.

3. The method for road damage identification and localization by fusing multi-source information according to claim 1, characterized in that: The deep learning recognition model in S2 includes: a convolutional neural network for target detection to process global UAV images, and a convolutional neural network for semantic segmentation to process local high-definition images.

4. The road damage identification and localization method fusing multi-source information according to claim 3, characterized in that: The semantic segmentation convolutional neural network uses the Focal Loss function during training to address the class imbalance between defective pixels and background pixels. The specific formula for the Focal Loss function is as follows: FL(p t )=-a t *(1-p t ) γ *log(p t ) Where, p t α represents the model's predicted probability of the true class. t γ is the class weight balancing factor, and γ is the hard sample focusing parameter.

5. The method for road damage identification and localization by fusing multi-source information according to claim 1, characterized in that: The spatial matching and fusion in S4 specifically involves: using a computer vision image registration algorithm to solve for the transformation parameters from the local image coordinate system to the global 3D model coordinate system; and then using the coordinate transformation formula to map the coordinates of the disease points in the local image to the global coordinate system. The coordinate transformation formula is as follows: P global =s*R*P local +T Among them, P local Let P be the homogeneous coordinates of the disease point in the local image coordinate system. global R represents the transformed global 3D coordinates, T is the rotation matrix, and s is the translation vector.

6. The method for road damage identification and localization by fusing multi-source information according to claim 3, characterized in that: The target detection convolutional neural network and the semantic segmentation convolutional neural network are fused at the feature level; the fusion method is as follows: the extracted global features F global and local features F local The weighted concatenation is performed using the following formula: F fused =W g ·F global ⊕W l ·F local Among them, W g and W l Let F be a learnable weight matrix, ⊕ denotes vector concatenation operation, and F fused This is the fused multimodal feature vector.

7. The method for road damage identification and localization by fusing multi-source information according to claim 5, characterized in that: The spatial matching of the wall-climbing robot prioritizes direct alignment between the positioning module data and the 3D model, and assists in the verification and refinement of the coordinate transformation formula.

8. The method for road damage identification and localization by fusing multi-source information according to claim 1, characterized in that: In the S2 deep learning recognition model, environmental temperature data was introduced as one of the input features during training to optimize the model's recognition performance for typical diseases in cold regions.

9. The method for road damage identification and localization by fusing multi-source information according to claim 1, characterized in that: S2 also includes using a generative adversarial network (GAN) to augment the acquired images, thereby expanding the training sample set and improving the model's generalization ability.

10. The method for road damage identification and localization by fusing multi-source information according to claim 1, characterized in that: The technical condition indicators in S5 are, for roads, at least one of the Road Surface Damage Index (PCI) and the Ride Quality Index (RQI), and for bridges, an overall technical condition score calculated based on the scores of the three modules: superstructure, substructure, and bridge deck system.

Images