Unmanned aerial vehicle roadbed hidden danger cooperative detection method and device

Abstract

The application provides a kind of unmanned aerial vehicle roadbed hidden danger cooperative detection method and device, method includes: obtaining polarization binocular image sequence and carrying out dynamic radiance correction, generates left and right view image sequence of anti-glare;Macroscopic disease identification is carried out to diffuse reflection image, and macroscopic semantic feature field is generated;Diffuse reflection image is carried out natural texture enhancement processing, and natural speckle field and corresponding matchable area are generated;Based on the left and right view image sequence of anti-glare, three-dimensional reconstruction is carried out in combination with scale constraint and double-axis cross flight path constraint, and multi-period three-dimensional ground model is generated;Based on multi-period three-dimensional ground model and natural speckle field, deformation solution is carried out to the same target area of adjacent inspection period in matchable area, and deformation field information is obtained;Macroscopic semantic feature field and deformation field information are mapped to the same three-dimensional coordinate system for cooperative cross-validation, and hidden danger detection result is output.The way, improve the feasibility of monitoring in wet environment and the reconstruction accuracy under the condition of no control point.

Description

Technical Field

[0001] This invention relates to the field of collaborative detection technology, and in particular to a method and device for collaborative detection of roadbed hazards using unmanned aerial vehicles (UAVs). Background Technology

[0002] Currently, non-contact monitoring of highway slope and subgrade deformation and landslide hazards mainly employs a combination of UAV photogrammetry and 3D digital image correlation (3D-DIC). The conventional implementation process requires establishing ground control points (GCPs) in the area to be monitored to constrain the 3D reconstruction scale, and spraying artificial speckle patterns onto the target surface to meet the texture requirements of DIC matching. However, rural roads have numerous points, long lines, and wide areas, with damage points often distributed in high-risk areas such as steep slopes and sections near water. Large-scale deployment of GCPs and spraying of speckle patterns is costly, inefficient, and poses safety risks.

[0003] Furthermore, after rain or in damp environments, water accumulation on the road surface and slippery slopes can cause specular reflection, leading to localized overexposure and texture loss in ordinary visible light cameras. This undermines the assumption of constant brightness relied upon by traditional DIC methods, making feature matching prone to failure. Without GCP (Geometric Content Detection), relying solely on UAV positioning and image matching for 3D reconstruction is prone to scaling errors and Z-axis dome effects, making it difficult to meet the accuracy requirements for identifying minor roadbed settlement or initial slope slippage. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide a method and device for collaborative detection of roadbed hazards by UAVs, which eliminates the need for GCP deployment and speckle spraying, suppresses reflection through polarization imaging, and combines natural texture enhancement and scale constraint to reduce field costs and improve the feasibility of monitoring in humid environments and reconstruction accuracy under conditions without control points.

[0005] In a first aspect, embodiments of the present invention provide a method for collaborative detection of roadbed hazards using unmanned aerial vehicles (UAVs). The method includes: acquiring a sequence of polarized binocular images; performing dynamic radiometry correction on the polarized binocular image sequence to generate a deglare-free left and right view image sequence; performing macroscopic hazard identification on diffuse reflection images in the deglare-free left and right view image sequence to generate a macroscopic semantic feature field; performing natural texture enhancement processing on the diffuse reflection images to generate a natural speckle field and corresponding matchable regions; performing three-dimensional reconstruction based on the deglare-free left and right view image sequence, combined with the scale constraints of a portable spatial 3D scale and dual-axis cross-flight constraints, to generate a multi-phase three-dimensional surface model; performing deformation calculation on the same target area in adjacent inspection cycles within the matchable region based on the multi-phase three-dimensional surface model and the natural speckle field to obtain deformation field information; mapping the macroscopic semantic feature field and the deformation field information to the same three-dimensional coordinate system for collaborative cross-validation, and outputting hazard detection results.

[0006] In a preferred embodiment of the present invention, the above-described acquisition of a polarized binocular image sequence, dynamic radiosity correction of the polarized binocular image sequence, and generation of a deglare-free left and right view image sequence includes: calibrating the transmission axis direction of the polarizer before inspection, so that the transmission axis direction of the polarizer is orthogonal to the polarization direction of reflections generated by water surfaces, slippery road surfaces, or damp slopes, in order to filter out specular reflection glare; and performing vignetting correction, histogram normalization, local brightness equalization, and cloud shadow compensation on the polarized binocular image sequence after glare removal to generate a deglare-free left and right view image sequence.

[0007] In a preferred embodiment of the present invention, the above-described method of performing macroscopic disease identification on diffuse reflection images in the deglare left and right view image sequence to generate a macroscopic semantic feature field includes: performing multi-level feature extraction on the diffuse reflection images to obtain disease feature maps at different scales; performing cross-scale adaptive fusion on the disease feature maps at different scales, and performing channel enhancement on the fused features to obtain enhanced feature maps; performing stepwise upsampling and category mapping on the enhanced feature maps to generate disease category probability maps; determining the disease category of each pixel based on the disease category probability maps to generate various disease masks; and performing post-processing on the various disease masks to extract the geometric features and confidence information of each disease region to form a structured macroscopic semantic feature field.

[0008] In a preferred embodiment of the present invention, the above-described natural texture enhancement processing of the diffuse image to generate a natural speckle field and a corresponding matchable region includes: performing contrast-limited adaptive histogram equalization on the diffuse image to obtain an enhanced image; performing sharpening processing on the enhanced image to obtain a sharpened image; calculating local gray-level variance, gradient energy, feature point density, and correlation peak confidence within a local window of the sharpened image; performing weighted fusion on the local gray-level variance, gradient energy, feature point density, and correlation peak confidence to obtain a texture quality score; and marking the area covered by the local window as a matchable region when the texture quality score is not less than a preset score threshold.

[0009] In a preferred embodiment of the present invention, the above-described method of performing 3D reconstruction based on the deglare left and right view image sequence, combined with the scale constraints of a portable spatial 3D scale and the dual-axis intersecting flight path constraints, to generate a multi-stage 3D surface model includes: controlling a UAV to acquire multi-view images along two intersecting flight paths along the route direction and perpendicular to the route direction; placing a portable spatial 3D scale in the area to be measured; the portable spatial 3D scale containing at least two endpoint targets; detecting the endpoint targets of the portable spatial 3D scale in the multi-view images, and obtaining the 3D coordinates of the spatial endpoints corresponding to each endpoint target in the reconstruction model through multi-view geometric calculations; determining scale constraint terms based on the spatial endpoints and their corresponding physical distances, and adding the scale constraint terms to the optimization objective function of the motion-restoring structured bundle adjustment model to constrain the global scale of the 3D reconstruction.

[0010] In a preferred embodiment of the present invention, the above-described deformation calculation of the same target area in adjacent inspection cycles within the matchable area, based on the multi-phase three-dimensional surface model and the natural speckle field, to obtain deformation field information, includes: within the matchable area, performing correlation matching between the local windows corresponding to each pixel of the previous phase three-dimensional surface model and the corresponding positions of the subsequent phase three-dimensional surface model to determine the optimal matching position of each pixel; determining the three-dimensional spatial coordinate difference of the pixel between the previous and subsequent phases based on the optimal matching position to obtain a three-dimensional displacement vector; calculating the displacement gradient based on the three-dimensional displacement vectors of multiple pixels, and obtaining the strain component from the displacement gradient to obtain the strain distribution field; the three-dimensional displacement vector and the strain distribution field together constitute the deformation field information.

[0011] In a preferred embodiment of the present invention, the step of mapping the macroscopic semantic feature field and the deformation field information to the same three-dimensional coordinate system for collaborative cross-validation and outputting the hazard detection result includes: projecting a two-dimensional disease mask onto the surface of a three-dimensional ground model to form a three-dimensional disease region with disease category, confidence level, and spatial range; calculating the macroscopic disease confidence level and microscopic deformation risk value of each three-dimensional disease region; and outputting the hazard detection result as a high-level hazard region when the macroscopic disease confidence level is not less than a preset first confidence level threshold and the microscopic deformation risk value is not less than a preset second confidence level threshold. When the confidence level of the macroscopic disease is not less than the preset first confidence threshold and the microscopic deformation risk value is less than the preset second confidence threshold, the hidden danger detection result is output as a general disease area; when the confidence level of the macroscopic disease is less than the preset first confidence threshold and the microscopic deformation risk value is not less than the preset second confidence threshold, the hidden danger detection result is output as an early structural hidden danger area; when the confidence level of the macroscopic disease is less than the preset first confidence threshold and the microscopic deformation risk value is less than the preset second confidence threshold, the hidden danger detection result is output as a normal area.

[0012] Secondly, embodiments of the present invention also provide a collaborative detection device for roadbed hazards using unmanned aerial vehicles (UAVs). The device includes: a dynamic radiosity correction module for acquiring a polarized binocular image sequence, performing dynamic radiosity correction on the polarized binocular image sequence, and generating a deglare-free left and right view image sequence; a macroscopic hazard identification module for identifying macroscopic hazards in the diffuse reflection images of the deglare-free left and right view image sequence, generating a macroscopic semantic feature field; a matchable region generation module for performing natural texture enhancement processing on the diffuse reflection images, generating a natural speckle field and corresponding matchable regions; a 3D reconstruction module for performing 3D reconstruction based on the deglare-free left and right view image sequence, combined with the scale constraints of a portable spatial 3D scale and dual-axis cross-flight constraints, generating a multi-phase 3D surface model; a deformation calculation module for performing deformation calculation on the same target area in adjacent inspection cycles within the matchable region based on the multi-phase 3D surface model and the natural speckle field, obtaining deformation field information; and a collaborative cross-validation module for mapping the macroscopic semantic feature field and the deformation field information to the same 3D coordinate system for collaborative cross-validation, outputting hazard detection results.

[0013] Thirdly, embodiments of the present invention also provide an electronic device, including a processor and a memory, wherein the memory stores computer-executable instructions that can be executed by the processor, and the processor executes the computer-executable instructions to implement the UAV roadbed hazard collaborative detection method of the first aspect described above.

[0014] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing computer-executable instructions. When the computer-executable instructions are invoked and executed by a processor, the computer-executable instructions cause the processor to implement the UAV roadbed hazard collaborative detection method described in the first aspect.

[0015] The embodiments of the present invention bring the following beneficial effects: This invention provides a method and apparatus for collaborative detection of roadbed hazards using unmanned aerial vehicles (UAVs). The method involves acquiring a sequence of polarized binocular images, performing dynamic radiometry correction on the sequence to generate deglare-free left and right view image sequences, identifying macroscopic defects in the diffuse reflection images within these sequences to generate a macroscopic semantic feature field, enhancing the natural texture of the diffuse reflection images to generate a natural speckle field and corresponding matchable regions, and performing 3D reconstruction based on the deglare-free left and right view image sequences, combined with scale constraints from a portable 3D scale and dual-axis cross-flight constraints to generate multi-phase 3D surface models. Based on these models and the natural speckle field, deformation calculations are performed on the same target area in adjacent inspection cycles within the matchable regions to obtain deformation field information. The macroscopic semantic feature field and the deformation field information are mapped to the same 3D coordinate system for collaborative cross-validation, outputting the hazard detection results. This method eliminates the need for GCP (Gas-Chip-Based Protection) deployment and speckle spraying, suppresses reflection through polarized imaging, and combines natural texture enhancement with scale constraints to reduce fieldwork costs and improve the feasibility of monitoring in humid environments and reconstruction accuracy under control point-free conditions.

[0016] Other features and advantages of this disclosure will be set forth in the following description, or some features and advantages may be inferred from the description or determined without doubt, or may be learned by practicing the techniques described above.

[0017] To make the above-mentioned objects, features and advantages of this disclosure more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0018] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0019] Figure 1 A flowchart of a method for collaborative detection of roadbed hazards using unmanned aerial vehicles (UAVs) provided in an embodiment of the present invention; Figure 2A flowchart of another method for collaborative detection of roadbed hazards by unmanned aerial vehicles provided in an embodiment of the present invention; Figure 3 A structural diagram of a lightweight semantic segmentation network provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of a UAV-based collaborative detection device for roadbed hazards provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] Currently, existing technologies for non-contact monitoring of highway slopes, roadbed deformation, shoulder retaining wall defects, and landslide hazards mainly include the following categories: 3D terrain reconstruction technology based on UAV motion-reconstructed structure photogrammetry (UAV-SfM). This technology typically recovers camera pose from multi-view UAV imagery and generates digital surface models, 3D point clouds, or mesh models of the area to be surveyed.

[0022] Deformation measurement technology based on three-dimensional digital image correlation (3D-DIC). This technology calculates the displacement field of the target surface by comparing the changes in texture subsets in images from different periods, or by combining the changes in corresponding regions in three-dimensional topography data, and further obtains the strain field when the accuracy conditions are met.

[0023] This is a fusion monitoring technology that combines UAV photogrammetry, ground control point (GCP) or real-time dynamic differential global navigation satellite system (RTK-GNSS) positioning with DIC post-processing. This type of technology is used to improve the scale accuracy and spatial registration accuracy of 3D reconstructions and to acquire large-scale deformation information of roadbed or slope surfaces.

[0024] Visual inspection technologies based on visible light RGB images, 3D point cloud difference, or digital surface model (DSM) change detection. These technologies are mainly used to identify surface defects such as cracks, erosion gullies, subsidence, and retaining wall defects.

[0025] When using UAV-SfM, DIC, and their fusion methods for deformation monitoring, the standard implementation process includes: First, ground control points are manually established and their spatial coordinates are determined. Surveyors set up targets in the areas of the roadbed, slope, or shoulder to be measured, and use a real-time dynamic differential global navigation satellite system (RTK-GNSS) or a total station to measure the three-dimensional coordinates of the targets. These coordinates are used as the spatial scale reference for the three-dimensional reconstruction of the structure of motion (SfM) to improve the scale accuracy and georegistration accuracy of the three-dimensional reconstruction, and reduce model scale drift and overall deviation in the elevation direction.

[0026] Second, target surface texture preprocessing. In order to meet the requirements of digital image correlation algorithms for subset texture randomness and grayscale stability, when the natural texture of the target surface is insufficient or the texture stability is poor, it is usually necessary to spray or lay high-contrast random textures on the surface of the soil, roadbed or retaining wall to be tested. For example, a white background is first formed, and then black matte speckle is sprayed to form an artificial speckle field.

[0027] Third, visible light image sequence acquisition. The UAV is equipped with a visible light RGB camera to acquire multiple images according to a preset flight path, flight altitude, and lateral and forward overlap rates. The acquired image sequences are used as input data for 3D reconstruction, image correlation analysis, or change detection.

[0028] Fourth, 3D surface model reconstruction. The image sequence and GCP coordinates are imported into photogrammetric software, and the SfM algorithm is used to recover the camera's interior and exterior orientation elements. Furthermore, a digital surface model, 3D point cloud, or mesh model with geographic coordinate information is generated through multi-view stereo (MVS).

[0029] Fifth, deformation or change calculation. In image-related analysis, the displacement distribution of the target surface is obtained by comparing the positional changes of texture subsets in images from different periods using the sum of absolute differences (SAD), normalized cross-correlation, or other correlation matching criteria. In three-dimensional model-based analysis, the morphological changes between different periods are obtained through point cloud registration, mesh difference, or DSM difference, and the strain distribution is further calculated when the accuracy and continuity conditions are met.

[0030] Current technologies have the following shortcomings: (1) Poor robustness to ambient light, especially susceptible to interference from reflections from standing water. After rain, freeze-thaw cycles, or in damp and muddy conditions, rural roads experience strong specular reflections from standing water and slippery slopes, leading to localized overexposure, texture loss, or grayscale saturation in ordinary visible light cameras. Traditional DIC methods rely on the assumptions of constant brightness and traceable texture. Specular reflections can disrupt the grayscale consistency between adjacent periods, making feature matching unstable or even ineffective.

[0031] (2) Over-reliance on artificial speckle and survey targets. Rural roads have many points, long lines and wide areas. Slope and roadbed defects are often distributed in high-risk areas such as steep slopes, valleys and water-adjacent sections. Large-scale deployment of GCP and spraying speckle is not only costly and inefficient, but also poses field safety risks. In areas with vegetation cover, loose soil, exposed sand and gravel or on roads, artificial speckle is also difficult to maintain for a long time.

[0032] (3) Significant elevation drift and scale error without GCP. When GCP cannot be deployed, aerial triangulation reconstruction relying solely on UAV-borne positioning and image matching is prone to overall scale scaling error, camera focal length and flight altitude coupling error, and Z-axis dome effect, resulting in the three-dimensional model failing to achieve the accuracy required for identifying minor roadbed settlement or initial slope slip.

[0033] (4) The results of surface defect identification and deformation monitoring lack a unified spatial correlation. Existing visual inspections mainly focus on macroscopic surface phenomena such as cracks, scour ditches, road shoulder damage, and retaining wall defects, while DIC deformation monitoring focuses on displacement and strain fields. The two types of information usually belong to different systems and different coordinate references, making it difficult to conduct joint analysis in the same three-dimensional space. It is also difficult to make auxiliary judgments on potential risks of voiding, softening, or slippage based on the spatial correspondence between surface defects and deformation trends.

[0034] Based on this, the present invention provides a method and apparatus for collaborative detection of roadbed hazards using unmanned aerial vehicles (UAVs). This method involves acquiring a polarized binocular image sequence, performing dynamic radiometry correction on the polarized binocular image sequence to generate a deglare-free left and right view image sequence, identifying macroscopic defects in the diffuse reflection images of the deglare-free left and right view image sequence to generate a macroscopic semantic feature field, performing natural texture enhancement processing on the diffuse reflection images to generate a natural speckle field and corresponding matchable regions, and performing 3D reconstruction based on the deglare-free left and right view image sequence, combined with the scale constraints of a portable 3D spatial scale and dual-axis cross-flight constraints, to generate a multi-phase 3D surface model. Based on the multi-phase 3D surface model and the natural speckle field, deformation calculations are performed on the same target area in adjacent inspection cycles within the matchable region to obtain deformation field information. The macroscopic semantic feature field and the deformation field information are mapped to the same 3D coordinate system for collaborative cross-validation, and the hazard detection result is output. This method eliminates the need for GCP deployment and speckle spraying. It suppresses reflections through polarization imaging, and combines natural texture enhancement with scale constraints to reduce field costs and improve the feasibility of monitoring in humid environments and reconstruction accuracy under conditions without control points.

[0035] To facilitate understanding of this embodiment, a method for collaborative detection of roadbed hazards using unmanned aerial vehicles (UAVs) will first be described in detail.

[0036] Example 1 This invention provides a method for collaborative detection of roadbed hazards using unmanned aerial vehicles (UAVs). Figure 1 This is a flowchart illustrating a collaborative detection method for roadbed hazards using unmanned aerial vehicles (UAVs) provided in an embodiment of the present invention. Figure 1 As shown, the collaborative detection method for roadbed hazards using unmanned aerial vehicles (UAVs) may include the following steps: Step S101: Obtain a polarized binocular image sequence, perform dynamic radiosity correction on the polarized binocular image sequence, and generate a deglare left and right view image sequence.

[0037] Specifically, acquiring a polarized binocular image sequence and performing dynamic radiosity correction on the polarized binocular image sequence to generate a deglare-free left and right view image sequence may include: calibrating the transmission axis direction of the polarizer before inspection so that the transmission axis direction of the polarizer is orthogonal to the polarization direction of reflections generated by water surfaces, slippery road surfaces, or damp slopes, in order to filter out specular reflection glare; and performing vignetting correction, histogram normalization, local brightness equalization, and cloud shadow compensation on the polarized binocular image sequence after glare removal to generate a deglare-free left and right view image sequence.

[0038] Among them, the drone is equipped with a polarization binocular camera. Before the inspection, the polarizer's transmission axis direction is determined through a calibration program, and it is made orthogonal to the main reflection polarization direction generated by water surface, slippery road surface or wet slope, so as to physically filter the specular reflection glare generated by water accumulation, muddy road surface and slippery slope.

[0039] The binocular cameras simultaneously acquire left and right view images. The system records camera intrinsic and extrinsic parameters, exposure time, attitude, real-time dynamic differential / post-processing dynamic differential (RTK / PPK) positioning information, and flight path information. Dynamic radiosity correction is performed on the raw images, including vignetting correction, histogram normalization, local brightness equalization, and cloud shadow compensation, to reduce global grayscale drift between different inspection periods. This step outputs a deglare-free left and right view image sequence and its corresponding acquisition metadata.

[0040] As one approach, for anti-reflective data acquisition, in addition to installing an orthogonal linear polarizer in front of a binocular camera, polarization cameras, multispectral cameras, shortwave infrared (SWIR) cameras, or tunable filters can be used to extract wavelengths that are not sensitive to water reflections; partial reflection suppression can also be achieved through multi-exposure fusion, high dynamic range imaging, or deep learning anti-reflection algorithms.

[0041] As one implementation method, for the binocular imaging module, it can be replaced by a multi-view camera, a structured light camera, an event camera, a visible light-thermal infrared fusion camera, or a monocular camera combined with a multi-view SfM reconstruction method, as long as it can output an image sequence that meets the requirements of subsequent 3D reconstruction and DIC calculation.

[0042] Step S102: Perform macroscopic defect identification on the diffuse reflection images in the deglare left and right view image sequence to generate a macroscopic semantic feature field.

[0043] Among them, a lightweight semantic segmentation network is used for macroscopic disease identification. The output diffuse reflection image is input into the pre-trained lightweight semantic segmentation network to perform pixel-level identification of macroscopic visible diseases in the areas of highway slopes, roadbeds, shoulders and retaining walls.

[0044] Step S103: Perform natural texture enhancement processing on the diffuse image to generate a natural speckle field and the corresponding matching region.

[0045] Specifically, performing natural texture enhancement processing on the diffuse image to generate a natural speckle field and corresponding matchable regions can include: performing contrast-limited adaptive histogram equalization on the diffuse image to obtain an enhanced image; sharpening the enhanced image to obtain a sharpened image; calculating the local gray-level variance, gradient energy, feature point density, and related peak confidence within a local window of the sharpened image; performing weighted fusion of the local gray-level variance, gradient energy, feature point density, and related peak confidence to obtain a texture quality score; and marking the area covered by the local window as a matchable region when the texture quality score is not less than a preset score threshold.

[0046] The obtained deglare diffuse reflection image is The system first performs Contrast Limited Adaptive Histogram Equalization (CLAHE) on the image to obtain the enhanced image. Then, the Laplacian operator is used for sharpening to obtain the sharpened image. :

[0047] in, Represents the Laplace operator. This represents the sharpening factor, set to 0.5.

[0048] Subsequently, the system calculates the image gradient magnitude. :

[0049] in, and These represent the gradient responses of the image in the horizontal and vertical directions, respectively. Through the above processing, the outlines of tiny soil particles, the edges of gravel, and the micro-textures of the roadbed surface are amplified, forming a high-contrast natural speckle field that can be used for DIC subset matching.

[0050] To determine whether a natural texture meets the DIC matching requirements, the system uses pixels as the basis for the determination. local window centered Internally calculate texture quality metrics. Texture quality metrics include at least local grayscale variance. Gradient energy Feature point density and related peak confidence The local gray-level variance and gradient energy are expressed as follows:

[0051]

[0052] in, This indicates the number of pixels within a local window. This represents the average grayscale value within a local window.

[0053] The system calculates the texture quality score based on the above indicators. :

[0054] in, These are the weighting coefficients, and ; , , These are used to normalize the local gray-level variance, gradient energy, and feature point density, respectively. All four weights are set to 0.25.

[0055] When the texture quality score of a certain area Not less than the preset threshold When, mark the region as a matchable region; when At that time, the system will change the DIC subset window from its initial size. The pixels gradually increase to Pixels or The region is counted as a pixel, and the relevant peak confidence is recalculated. If the region is still below the threshold after increasing the window size, it is marked as a low-confidence region and is not included in subsequent high-precision displacement calculations or its weight is reduced in the results.

[0056] Step S104: Based on the deglare left and right view image sequence, combined with the scale constraints of the portable spatial 3D scale and the dual-axis cross-line constraints, three-dimensional reconstruction is performed to generate a multi-stage three-dimensional surface model.

[0057] Specifically, based on the deglare-free left and right view image sequence, and combined with the scale constraints of a portable spatial 3D scale and the dual-axis intersecting flight path constraints, a multi-stage 3D surface model is generated through 3D reconstruction. This can include: controlling a UAV to acquire multi-view images along two intersecting flight paths along the route direction and perpendicular to the route direction; placing a portable spatial 3D scale in the area to be measured; the portable spatial 3D scale containing at least two endpoint targets; detecting the endpoint targets of the portable spatial 3D scale in the multi-view images, and obtaining the 3D coordinates of the spatial endpoints corresponding to each endpoint target in the reconstructed model through multi-view geometric calculations; determining scale constraints based on the spatial endpoints and their corresponding physical distances, and adding the scale constraints to the optimization objective function of the motion-restored structured bundle adjustment model to constrain the global scale of the 3D reconstruction.

[0058] In order to eliminate the overall scale drift and Z-axis dome effect under the condition of no ground control point (GCP), this invention adopts a combination constraint method of "dual-axis intersecting route + portable spatial 3D scale" for three-dimensional reconstruction.

[0059] The UAV acquired images using two sets of intersecting flight paths: one along the route and one perpendicular to the route. The route-along path ensured continuous coverage of the roadbed, shoulders, and retaining walls, while the perpendicular path increased the observation angles of the slopes and roadbed cross-sections. The forward overlap rate of the two sets of flight paths was set to 75%, and the lateral overlap rate to 65%. This intersecting flight path acquisition allowed for imaging of the same slope or roadbed area from different perspectives, reducing coupling errors between camera focal length, flight altitude, and terrain undulations.

[0060] A portable 3D spatial scale is placed in the high-risk area to be measured, the starting point of the inspection, or the ending point of the inspection. The portable 3D spatial scale is made of rigid material and includes at least two endpoint targets, with the physical distance between the endpoints specified. Pre-calibrated precisely. Endpoint targets can use automatically recognizable patterns such as high-contrast crosses, checkerboard patterns, AprilTag visual reference markers, or augmented reality square reference markers (ArUco codes).

[0061] During the 3D reconstruction process, the system first automatically detects the target endpoints of the scale in the multi-view images, and then obtains the 3D coordinates of the two endpoints in the reconstructed model through multi-view geometric calculations, denoted as follows: and The reconstructed distance between the endpoints is:

[0062] in, This represents the Euclidean distance.

[0063] The system will know the physical distance The scale constraint is incorporated into the Structure from Motion (SfM) bundle adjustment model. The optimization objective function including the scale constraint is expressed as:

[0064] in, This represents the set of parameters to be optimized, including camera intrinsics, camera extrinsics, and 3D point coordinates. This represents the observed image point of the j-th 3D point in the i-th image; Represents the camera projection function; This represents the intrinsic parameter matrix of the camera corresponding to the i-th image; and These represent the camera rotation matrix and translation vector, respectively. This represents the 3D points to be optimized; This indicates the scale constraint weight.

[0065] Through the above constraints, the system uses the known physical length of the scale to constrain the global scale of 3D reconstruction without measuring absolute geographic coordinates, thereby reducing scale drift and Z-axis dome effect under GCP-free conditions and improving the spatial scale consistency of subsequent 3D-DIC displacement calculations.

[0066] Step S105: Based on the multi-phase three-dimensional surface model and the natural speckle field, the deformation of the same target area in adjacent inspection cycles is calculated within the matching area to obtain deformation field information.

[0067] Specifically, based on multi-phase 3D surface models and natural speckle fields, deformation calculations are performed on the same target area in adjacent inspection cycles within a matchable region to obtain deformation field information. This can include: within the matchable region, correlation matching is performed between the local windows corresponding to each pixel in the previous phase 3D surface model and the corresponding positions in the subsequent phase 3D surface model to determine the optimal matching position for each pixel; based on the optimal matching position, the 3D spatial coordinate difference between the pixel in the previous and subsequent phases is determined to obtain the 3D displacement vector; the displacement gradient is calculated based on the 3D displacement vectors of multiple pixels, and the strain components are obtained from the displacement gradient to obtain the strain distribution field; the 3D displacement vector and the strain distribution field together constitute the deformation field information.

[0068] The system utilizes multi-stage 3D surface models and natural speckle fields to perform 3D-DIC matching on the same target area in adjacent inspection cycles. The Zero-mean Normalized Cross-Correlation (ZMNCC) criterion is preferred over the Sum of Absolute Differences (SAD) criterion to reduce the impact of local linear variations in illumination, humidity changes, and residual reflections on the matching results.

[0069] Let the grayscale sequence of the previous image subset be... The grayscale sequence of the candidate matching image subset in the next phase is ,in N represents the number of pixels in the subset of the image; and These represent the average gray values ​​of the two image subsets:

[0070]

[0071] The ZMNCC correlation coefficient C is expressed as:

[0072] The value of C ranges from -1 to 1. The system determines the optimal matching position between image subsets from two consecutive periods by searching and maximizing C. When the maximum correlation coefficient is less than a preset correlation threshold... When this happens, the corresponding matching point is marked as a low-confidence matching point; relevant threshold Take 0.85.

[0073] For points selected through matching confidence, the system obtains their spatial coordinates in the previous and next 3D coordinate systems based on the binocular camera calibration parameters, camera pose, and 3D reconstruction model. and The three-dimensional displacement vector U at this point is expressed as:

[0074] in, and Represents the horizontal displacement component. It indicates the vertical settlement or uplift component.

[0075] Furthermore, the system calculates the displacement gradient based on the displacement changes of multiple matching points within the local neighborhood, and then obtains the strain components from the displacement gradient. Let the displacement vector be... Then, the principal strain, shear strain, settlement, and slip direction can be calculated. The vertical settlement can be taken as:

[0076] when The time indicates that the point has subsided relative to the previous period. The horizontal slip can be expressed as:

[0077] The slip direction angle can be expressed as:

[0078] Through the above calculations, the system obtains the three-dimensional displacement vector field, settlement field, horizontal slip field, and strain distribution field of the highway slope, subgrade, shoulder retaining wall area to be tested.

[0079] Step S106: Map the macroscopic semantic feature field and deformation field information to the same three-dimensional coordinate system for collaborative cross-validation, and output the hidden danger detection results.

[0080] Specifically, mapping macroscopic semantic feature fields and deformation field information to the same three-dimensional coordinate system for collaborative cross-validation and outputting hazard detection results can include: projecting a two-dimensional disease mask onto the surface of a three-dimensional ground model to form a three-dimensional disease region with disease category, confidence level, and spatial range; calculating the macroscopic disease confidence level and microscopic deformation risk value for each three-dimensional disease region; when the macroscopic disease confidence level is not less than a preset first confidence level threshold and the microscopic deformation risk value is not less than a preset second confidence level threshold, the output hazard detection result is a high-level hazard region; when the macroscopic disease confidence level is not less than the preset first confidence level threshold and the microscopic deformation risk value is less than the preset second confidence level threshold, the output hazard detection result is a general disease region; when the macroscopic disease confidence level is less than the preset first confidence level threshold and the microscopic deformation risk value is not less than the preset second confidence level threshold, the output hazard detection result is an early structural hazard region; when the macroscopic disease confidence level is less than the preset first confidence level threshold and the microscopic deformation risk value is less than the preset second confidence level threshold, the output hazard detection result is a normal region.

[0081] Specifically, the system maps the macroscopic disease semantic field with the microscopic displacement field, settlement field, and strain field to the same three-dimensional coordinate system. In particular, the system uses camera pose, point cloud model, and image-3D point correspondence to project a two-dimensional disease mask onto a three-dimensional surface, forming a three-dimensional disease region with disease category, confidence level, and spatial range.

[0082] Let a certain three-dimensional spatial region be Its macroscopic disease confidence level is The micro-deformation risk value is Among them, the confidence level of macroscopic defects can be determined by the average confidence level of defect segments within the region; the risk value of microscopic deformation can be obtained by normalized weighting of settlement, horizontal slip, and maximum principal strain. in, Indicates the area Representative settlement within, This represents the representative horizontal slip. Indicates the maximum principal strain; , , These represent the settlement threshold, horizontal slip threshold, and strain threshold, respectively. These are the weighting coefficients, and .

[0083] The system is constructed with the following collaborative discrimination logic: when ,and When this occurs, it indicates that the area simultaneously exhibits both macroscopic visible defects and microscopic abnormal deformations, classifying it as a high-level hidden danger area, and issuing risk warnings for roadbed voids, deep soil erosion, slope instability, or uneven settlement. when ,but When the area has macroscopic surface defects such as cracks, erosion, water accumulation, exposed slopes or cracked retaining walls, but no significant microscopic deformation has been detected, it is judged as a general defect area. when ,but At this time, it indicates that the surface defects in the area are not yet obvious, but abnormal strain concentration, slight subsidence, horizontal slip or shear bands have already appeared, and it is determined to be an area with early structural hidden dangers. when ,and At that time, it was determined to be a normal or low-risk area.

[0084] in, This represents the confidence threshold for macroscopic diseases. This represents the threshold for microscopic deformation risk, which can be set based on historical inspection data, road grade, slope type, and maintenance requirements.

[0085] The system ultimately outputs the location, type, risk level, 3D displacement cloud map, settlement cloud map, and strain cloud map of the potential hazard. Through cross-validation of the macroscopic disease semantic field and the microscopic deformation field, false alarms caused by single visual disease identification can be reduced, and the early identification capability of hidden risks such as roadbed voids, slope slippage, and retaining wall instability can be improved.

[0086] In this embodiment, instead of manually spraying speckle patterns or setting up dedicated markers, natural textures such as sand and gravel particles, soil aggregates, asphalt textures, exposed rock and soil, vegetation edges, and water erosion textures are used as DIC matching features. Through glare removal image enhancement, sharpening, gradient calculation, and texture quality evaluation, a natural speckle field suitable for DIC subset matching is constructed, and high-confidence regions suitable for displacement calculation are selected, thereby reducing manual fieldwork costs and the need for operations in high-risk areas.

[0087] The UAV-based collaborative detection method for roadbed hazards provided in this invention can acquire a polarized binocular image sequence, perform dynamic radiometry correction on the polarized binocular image sequence to generate a deglare-free left and right view image sequence, perform macroscopic hazard identification on the diffuse reflection image in the deglare-free left and right view image sequence to generate a macroscopic semantic feature field, perform natural texture enhancement processing on the diffuse reflection image to generate a natural speckle field and corresponding matchable regions, and perform 3D reconstruction based on the deglare-free left and right view image sequence, combined with the scale constraints of a portable spatial 3D scale and dual-axis cross-flight constraints, to generate a multi-phase 3D surface model. Based on the multi-phase 3D surface model and the natural speckle field, deformation calculation is performed on the same target area in adjacent inspection cycles within the matchable region to obtain deformation field information. The macroscopic semantic feature field and deformation field information are mapped to the same 3D coordinate system for collaborative cross-validation, and the hazard detection results are output. In this method, there is no need to deploy GCPs and spray speckle, and the reflection is suppressed by polarized imaging. Combined with natural texture enhancement and scale constraints, the fieldwork costs are reduced, and the feasibility of monitoring in humid environments and the reconstruction accuracy under conditions without control points are improved.

[0088] Example 2 This invention also provides another method for collaborative detection of roadbed hazards by unmanned aerial vehicles (UAVs); this method is implemented based on the method in the above embodiments; this method focuses on describing the specific implementation of macroscopic hazard identification and generation of macroscopic semantic feature fields from diffuse reflection images in deglare left and right view image sequences.

[0089] Figure 2 A flowchart of another UAV-based collaborative detection method for roadbed hazards provided in an embodiment of the present invention is shown below. Figure 2 As shown, the process of identifying macroscopic defects in diffuse reflection images from a sequence of deglare left and right view images and generating a macroscopic semantic feature field may include the following steps: Step S201: Perform multi-level feature extraction on the diffuse reflection image to obtain disease feature maps at different scales.

[0090] Specifically, let the output diffuse reflection image be... Its original size is First, scale the image to a fixed input size. ,in Set W to 512. Then normalize the image pixel values ​​to obtain the network input image X, which is calculated as follows:

[0091] in, This represents the channel mean of the training sample image. This represents the channel standard deviation of the training sample images. The normalized image X is used to reduce the impact of differences in light intensity during different inspection periods on the disease identification results.

[0092] The preprocessed image X is input into a lightweight encoder for multi-layer feature extraction. The encoder consists of four consecutive feature extraction stages, each composed of depthwise separable convolutions, batch normalization layers, and nonlinear activation functions to reduce the computational cost of the model. The feature maps output from each stage are denoted as follows:

[0093] in, Indicates encoder, This is a shallow texture feature map. For edge feature maps, This is a local morphological feature map. These are deep semantic feature maps. The spatial resolution of the above feature maps decreases sequentially, and can be 1 / 2, 1 / 4, 1 / 8 and 1 / 16 of the input image size, respectively, with the number of channels set to 32, 64, 128 and 256, respectively.

[0094] Among them, shallow texture feature map Used to characterize the texture variations of roadbed, slope, shoulder, and retaining wall surfaces; edge feature map Used to characterize crack boundaries, scour ditch boundaries, settlement boundaries, and retaining wall crack edges; local morphological feature map. Used to characterize the morphology of areas such as potholes, collapses, and missing shoulders; deep semantic feature map It is used to characterize the spatial relationship between the diseased area and the surrounding roads, slopes and vegetation background.

[0095] Step S202: Perform cross-scale adaptive fusion of disease feature maps at different scales, and perform channel enhancement on the fused features to obtain enhanced feature maps.

[0096] Specifically, the multi-layer feature map output by the encoder Input a multi-scale feature adaptive fusion module. First, the feature maps at each scale are unified to the same spatial resolution through upsampling or downsampling, and then... Convolution with a uniform number of channels yields:

[0097] in, This indicates bilinear interpolation upsampling or stride convolution downsampling. express Convolution operation, This represents the feature map after scale alignment.

[0098] Then, the adaptive fusion weights for feature maps at each scale are calculated:

[0099] in, Indicates global average pooling. This represents the fusion weight of the feature map at the i-th scale. The multi-scale features are weighted and fused according to the fusion weight to obtain the fused feature map. :

[0100] In this way, the network can automatically adjust the contribution of features at different levels according to the scale of the disease, thereby improving its ability to identify diseases with significant scale differences, such as thin cracks, small-scale shoulder defects, local scour ditches, retaining wall cracks, and settlement boundaries.

[0101] Fuse feature maps The input channel attention module calculates the contribution weight of each channel to disease identification. First, global average pooling is performed on the fused feature map to obtain the channel description vector z:

[0102] in, This indicates that the c-th channel in the fused feature map is located at coordinates... eigenvalues ​​at that location and These represent the height and width of the fused feature map, respectively.

[0103] Then, channel attention weights s are generated through two fully connected transformations and a sigmoid activation function:

[0104] in, and Represents the learnable weight matrix. Represents the ReLU (Linear Rectifier Unit) activation function. This represents the Sigmoid function. Finally, the channel attention weights are multiplied channel by channel by the fused feature map to obtain the enhanced feature map. :

[0105] in, This indicates channel-wise multiplication. Through this channel attention enhancement process, the network enhances effective feature channels related to cracks, erosion, water accumulation, collapse, retaining wall cracking, and vegetation anomalies, while suppressing interfering features generated by shadows, ordinary texture noise, vegetation background, and non-disease areas.

[0106] Step S203: Perform step-by-step upsampling and category mapping on the enhanced feature map to generate a disease category probability map.

[0107] Specifically, enhance the feature map The input decoder performs step-by-step upsampling to restore the spatial resolution of the input image. The decoder outputs a disease category probability map P:

[0108] in, Indicates decoder, Represents the category mapping convolution, This represents the multi-class probability normalization function. The size of the disease category probability map P is... Where C represents the number of categories. The categories include at least the background, cracks, scour ditches, waterlogged areas, exposed slope areas, damaged shoulder areas, cracked retaining wall areas, subsidence pothole areas, subsidence boundary areas, and areas with abnormal vegetation.

[0109] For any pixel in the image Its disease categories Determine as follows:

[0110] in, Represents pixels The probability of belonging to the k-th disease category. If the probability of the largest category is less than the preset confidence threshold. If the confidence threshold is not met, the pixel is classified as background or a low-confidence region. Take 0.6.

[0111] Step S204: Determine the disease category of each pixel based on the disease category probability map, and generate various disease masks.

[0112] Specifically, a macroscopic disease mask is generated based on the disease category probability map. :

[0113] in, Represents pixels Does it belong to the first Class of diseases; when pixels Prediction category equal to the Class, and the probability of its corresponding class. Not less than the preset confidence threshold season Otherwise The system performs morphological opening and closing operations and small connected component removal on various defect masks to remove isolated noise points and fill in local broken areas. For areas smaller than a preset area threshold... Connected components are removed, and the area threshold is applied. The resolution can be set from 20 to 100 pixels depending on the image resolution.

[0114] Subsequently, contour extraction was performed on the preserved connected components to obtain the contour lines of each diseased area. Circumscribed rectangle Area , main direction Aspect Ratio and regional confidence The region confidence score can be calculated based on the average class probability of all pixels within that region.

[0115] in, This represents the connected region of the nth disease. This indicates the number of pixels in the area.

[0116] Step S205 involves post-processing various disease masks to extract the geometric features and confidence information of each disease area, forming a structured macroscopic semantic feature field.

[0117] Specifically, the category, mask, outline, circumscribed rectangle, area, main direction, aspect ratio, and confidence level of each diseased area are structurally represented to form a macroscopic semantic feature field. :

[0118] in, This represents the structured semantic information of the nth diseased region. Indicates the type of disease. Indicates a masking layer for disease. Indicates the outline. Indicates the circumscribed rectangle. Indicates the area of ​​the diseased region. Indicates the main direction of the diseased area. Indicates the aspect ratio of the affected area. The region confidence level is represented by t, the image acquisition time is represented by id, the image number or camera number is represented by N, and the number of diseased areas detected is represented by N.

[0119] Therefore, the system converts the macroscopic disease identification results in the two-dimensional image into a macroscopic semantic feature field that can be used for subsequent three-dimensional registration, displacement field superposition, and macro-micro collaborative early warning.

[0120] During the model training phase, a training sample set containing images of highway slopes, roadbeds, shoulders, and retaining walls was constructed. Pixel-level annotations were performed on cracks, scour ditches, waterlogged areas, exposed slope areas, damaged shoulder areas, cracked retaining wall areas, subsidence pit areas, settlement boundary areas, and areas with abnormal vegetation. A weighted combination of cross-entropy loss and Dice loss was used during training to simultaneously improve class discrimination ability and the accuracy of defect boundary segmentation. The total loss function is:

[0121] in, Represents cross-entropy loss, This indicates Dice's loss. and For loss weights, preferably, , .

[0122] The cross-entropy loss is calculated as follows:

[0123] The Dice loss is calculated as follows:

[0124] in, Represents pixels The corresponding real category label, To prevent the smoothing coefficient from being zero in the denominator, take... .

[0125] The model training parameters are set as follows: input image size Batch size is 16, initial learning rate is The optimizer used is the Adam optimizer, with a weight decay factor of 1. The training epochs range from 100 to 200, and the learning rate decay strategy employs cosine annealing or step decay. After training, the model parameters with the highest segmentation accuracy on the validation set are used as the parameters for the macroscopic disease semantic segmentation model in actual inspections.

[0126] For ease of understanding, Figure 3 This is a structural diagram of a lightweight semantic segmentation network provided in an embodiment of the present invention.

[0127] The lightweight semantic segmentation network includes an encoder, a decoder, a multi-scale feature adaptive fusion module, and a channel attention module, which output macroscopic lesion masks, lesion categories, image coordinates, bounding boxes, contour lines, and confidence scores. The specific implementation details have been described in the above embodiments and will not be repeated here.

[0128] The specific effects of this application are as follows: Overcoming the limitations of image acquisition in wet and reflective environments. By using orthogonal polarization extinction and dynamic radiosity correction, specular reflections caused by water accumulation, water film, and slippery slopes are physically suppressed, reducing the probability of local overexposure and texture loss in images. Furthermore, by combining the ZMNCC criterion to resist residual brightness changes, the DIC algorithm can be extended from ideal laboratory environments to rural road field environments such as after rain, dampness, and mud.

[0129] Reduce manual deployment costs and field risks. Replace manual spraying with enhanced natural textures and large-scale GCP mapping networks with portable 3D scale rulers, reducing the need for surveyors to deploy and spray in steep slopes, near water, and in areas prone to landslides, thereby improving inspection efficiency and safety.

[0130] Improve the stability of 3D reconstruction in scenarios without GCP. The dual-axis intersecting route increases multi-view constraints on the target area, and the portable spatial 3D scale provides hard constraints on known physical lengths, which can suppress scale drift and Z-axis dome effect in SfM reconstruction, and improve the measurability of small deformations such as roadbed settlement and slope slip.

[0131] This upgrade moves from surface identification to structural early warning. Macroscopic defect AI segmentation is used to detect exposed cracks, erosion, water accumulation, and defects, while 3D-DIC is used to capture creep, settlement, and shear bands that have not yet been exposed. The two are cross-validated in a unified three-dimensional coordinate system, which can reduce the risk of missing deep voids or early instability by a single visual inspection.

[0132] This facilitates the formation of a closed-loop result for maintenance management. The system directly outputs the location, type, level, and three-dimensional deformation of potential hazards, making it easy to interface with rural road maintenance systems, GIS platforms, or digital twin platforms, thereby improving the closed-loop efficiency of inspection, verification, handling, and re-measurement.

[0133] Example 3 Corresponding to the above method embodiments, this invention provides a collaborative detection device for unmanned aerial vehicle (UAV) roadbed hazards. Figure 4 This is a schematic diagram of the structure of a UAV-based collaborative detection device for roadbed hazards provided in an embodiment of the present invention, as shown below. Figure 4 As shown, the UAV-based collaborative detection device for roadbed hazards may include: The dynamic radiosity correction module 401 is used to acquire a polarized binocular image sequence, perform dynamic radiosity correction on the polarized binocular image sequence, and generate a deglare left and right view image sequence.

[0134] The macroscopic disease identification module 402 is used to identify macroscopic diseases in the diffuse reflection images in the deglare left and right view image sequence and generate a macroscopic semantic feature field.

[0135] The matching region generation module 403 is used to perform natural texture enhancement processing on the diffuse image to generate a natural speckle field and the corresponding matching region.

[0136] The 3D reconstruction module 404 is used to perform 3D reconstruction based on the deglare left and right view image sequence, combined with the scale constraints of the portable spatial 3D scale and the dual-axis cross-line constraints, to generate a multi-stage 3D surface model.

[0137] The deformation calculation module 405 is used to perform deformation calculation on the same target area in adjacent inspection cycles within a matchable area based on a multi-phase three-dimensional surface model and natural speckle field, and obtain deformation field information.

[0138] The collaborative cross-validation module 406 is used to map macroscopic semantic feature fields and deformation field information to the same three-dimensional coordinate system for collaborative cross-validation and output the hazard detection results.

[0139] The UAV-based roadbed hazard collaborative detection device provided in this invention can acquire a polarized binocular image sequence, perform dynamic radiometry correction on the polarized binocular image sequence to generate a deglare-free left and right view image sequence, perform macroscopic hazard identification on the diffuse reflection image in the deglare-free left and right view image sequence to generate a macroscopic semantic feature field, perform natural texture enhancement processing on the diffuse reflection image to generate a natural speckle field and corresponding matchable regions, and perform 3D reconstruction based on the deglare-free left and right view image sequence, combined with the scale constraints of a portable spatial 3D scale and dual-axis cross-flight constraints, to generate a multi-phase 3D surface model. Based on the multi-phase 3D surface model and the natural speckle field, deformation calculation is performed on the same target area in adjacent inspection cycles within the matchable region to obtain deformation field information. The macroscopic semantic feature field and deformation field information are mapped to the same 3D coordinate system for collaborative cross-validation, and the hazard detection results are output. In this method, there is no need to deploy GCPs and spray speckle, and the reflection is suppressed by polarized imaging. Combined with natural texture enhancement and scale constraints, the fieldwork costs are reduced, and the feasibility of monitoring in humid environments and the reconstruction accuracy under conditions without control points are improved.

[0140] In some embodiments, the dynamic radiometry correction module is also used to calibrate the polarizer's transmission axis direction before inspection, so that the polarizer's transmission axis direction is orthogonal to the reflected polarization direction generated by the water surface, slippery road surface, or wet slope, so as to filter out specular reflection glare; and to perform vignetting correction, histogram normalization, local brightness equalization, and cloud shadow compensation on the polarized binocular image sequence after glare removal, to generate a glare-free left and right view image sequence.

[0141] In some embodiments, the macroscopic disease identification module is further configured to perform multi-level feature extraction on diffuse reflection images to obtain disease feature maps at different scales; perform cross-scale adaptive fusion on disease feature maps at different scales, and perform channel enhancement on the fused features to obtain enhanced feature maps; perform stepwise upsampling and category mapping on the enhanced feature maps to generate disease category probability maps; determine the disease category of each pixel based on the disease category probability maps to generate various disease masks; and perform post-processing on various disease masks to extract the geometric features and confidence information of each disease region to form a structured macroscopic semantic feature field.

[0142] In some embodiments, the matching region generation module is further configured to perform contrast-limited adaptive histogram equalization on the diffuse image to obtain an enhanced image; perform sharpening on the enhanced image to obtain a sharpened image; calculate the local gray-level variance, gradient energy, feature point density, and correlation peak confidence within a local window of the sharpened image; perform weighted fusion on the local gray-level variance, gradient energy, feature point density, and correlation peak confidence to obtain a texture quality score; and mark the area covered by the local window as a matching region when the texture quality score is not less than a preset score threshold.

[0143] In some embodiments, the 3D reconstruction module is further configured to control the UAV to acquire multi-view images along two intersecting routes: along the route direction and perpendicular to the route direction; place a portable spatial 3D scale in the area to be measured; the portable spatial 3D scale contains at least two endpoint targets; detect the endpoint targets of the portable spatial 3D scale in the multi-view images; obtain the 3D coordinates of the spatial endpoints corresponding to each endpoint target in the reconstruction model through multi-view geometric calculation; determine the scale constraint term based on the spatial endpoints and the corresponding physical distance; and add the scale constraint term to the optimization objective function of the motion-restored structure bundle adjustment model to constrain the global scale of the 3D reconstruction.

[0144] In some embodiments, the deformation calculation module is further configured to perform correlation matching between the local windows corresponding to each pixel of the previous three-dimensional surface model and the corresponding positions of the subsequent three-dimensional surface model within the matchable region, and determine the optimal matching position of each pixel; based on the optimal matching position, determine the three-dimensional spatial coordinate difference between the pixel in the previous and subsequent periods to obtain the three-dimensional displacement vector; calculate the displacement gradient based on the three-dimensional displacement vector of multiple pixels, and obtain the strain component from the displacement gradient to obtain the strain distribution field; the three-dimensional displacement vector and the strain distribution field together constitute the deformation field information.

[0145] In some embodiments, the collaborative cross-validation module is further configured to project a two-dimensional disease mask onto the surface of a three-dimensional land model to form a three-dimensional disease region with disease category, confidence level, and spatial range; calculate the macroscopic disease confidence level and microscopic deformation risk value for each three-dimensional disease region; when the macroscopic disease confidence level is not less than a preset first confidence level threshold and the microscopic deformation risk value is not less than a preset second confidence level threshold, the output hazard detection result is a high-level hazard region; when the macroscopic disease confidence level is not less than the preset first confidence level threshold and the microscopic deformation risk value is less than the preset second confidence level threshold, the output hazard detection result is a general disease region; when the macroscopic disease confidence level is less than the preset first confidence level threshold and the microscopic deformation risk value is not less than the preset second confidence level threshold, the output hazard detection result is an early structural hazard region; when the macroscopic disease confidence level is less than the preset first confidence level threshold and the microscopic deformation risk value is less than the preset second confidence level threshold, the output hazard detection result is a normal region.

[0146] The device provided in this embodiment of the invention has the same implementation principle and technical effect as the aforementioned method embodiment. For the sake of brevity, any parts not mentioned in the device embodiment can be referred to the corresponding content in the aforementioned method embodiment.

[0147] Example 5 This invention also provides an electronic device for running the above-described UAV-based collaborative detection method for roadbed hazards; see [link to previous document]. Figure 5 The diagram shows the structure of an electronic device, which includes a memory 500 and a processor 501. The memory 500 stores one or more computer instructions, which are executed by the processor 501 to realize the above-mentioned collaborative detection method for roadbed hazards by UAVs.

[0148] Furthermore, Figure 5 The electronic device shown also includes a bus 502 and a communication interface 503. The processor 501, the communication interface 503, and the memory 500 are connected via the bus 502.

[0149] The memory 500 may include high-speed random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 503 (which can be wired or wireless), such as the Internet, wide area network, local area network, metropolitan area network, etc. The bus 502 can be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 5 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0150] Processor 501 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of processor 501 or by instructions in software form. Processor 501 can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this invention can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software module can reside in a readily available storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory 500, and processor 501 reads information from memory 500 and, in conjunction with its hardware, completes the steps of the method described in the foregoing embodiments.

[0151] This invention also provides a computer-readable storage medium storing computer-executable instructions. When these computer-executable instructions are called and executed by a processor, they cause the processor to implement the aforementioned UAV-based collaborative detection method for roadbed hazards. For specific implementation details, please refer to the method embodiments, which will not be repeated here.

[0152] The computer program product for the collaborative detection method of roadbed hazards by UAV provided in the embodiments of the present invention includes a computer-readable storage medium storing non-volatile program code executable by a processor. The instructions included in the program code can be used to execute the methods in the preceding method embodiments. For specific implementation, please refer to the method embodiments, which will not be repeated here.

[0153] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0154] In the several embodiments provided by this invention, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the coupling or direct coupling or communication connection shown or discussed may be through some communication interface; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0155] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0156] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0157] If the functionality is implemented as a software functional unit and sold or used as an independent product, it can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0158] Finally, it should be noted that the above embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A collaborative detection method for roadbed hazards using unmanned aerial vehicles (UAVs), characterized in that, The method includes: A polarized binocular image sequence is acquired, and dynamic radiosity correction is performed on the polarized binocular image sequence to generate a deglare left and right view image sequence. Macroscopic defects are identified in the diffuse reflection images in the deglare left and right view image sequence to generate a macroscopic semantic feature field; The diffuse image is subjected to natural texture enhancement processing to generate a natural speckle field and a corresponding matchable region; Based on the deglare left and right view image sequence, combined with the scale constraints of the portable spatial 3D scale and the dual-axis cross-line constraints, three-dimensional reconstruction is performed to generate a multi-stage three-dimensional surface model. Based on the multi-phase three-dimensional surface model and the natural speckle field, deformation calculations are performed on the same target area in adjacent inspection cycles within the matching area to obtain deformation field information; The macroscopic semantic feature field and the deformation field information are mapped to the same three-dimensional coordinate system for collaborative cross-validation, and the hazard detection results are output.

2. The method according to claim 1, characterized in that, The process of acquiring a polarized binocular image sequence, performing dynamic radiosity correction on the polarized binocular image sequence, and generating a deglare-free left and right view image sequence includes: Before inspection, the direction of the polarizer's transmission axis is calibrated so that it is orthogonal to the polarization direction of the reflection generated by the water surface, wet and slippery road surface or damp slope, so as to filter out the glare from the mirror reflection. The polarized binocular image sequence after glare removal is subjected to vignetting correction, histogram normalization, local brightness equalization, and cloud shadow compensation to generate a glare-free left and right view image sequence.

3. The method according to claim 1, characterized in that, The step of performing macroscopic defect identification on diffuse reflection images in the deglare left and right view image sequence and generating a macroscopic semantic feature field includes: Multi-level feature extraction is performed on the diffuse reflection image to obtain disease feature maps at different scales; Cross-scale adaptive fusion is performed on the disease feature maps at different scales, and channel enhancement is performed on the fused features to obtain enhanced feature maps; The enhanced feature map is upsampled and mapped to categories step by step to generate a disease category probability map; Based on the disease category probability map, the disease category of each pixel is determined, and various disease masks are generated. The various disease masks are post-processed to extract the geometric features and confidence information of each disease area, forming a structured macroscopic semantic feature field.

4. The method according to claim 1, characterized in that, The step of performing natural texture enhancement processing on the diffuse image to generate a natural speckle field and corresponding matchable regions includes: The diffuse image is subjected to contrast-limited adaptive histogram equalization to obtain an enhanced image; The enhanced image is then sharpened to obtain a sharpened image; Within a local window of the sharpened image, calculate the local gray-level variance, gradient energy, feature point density, and related peak confidence. The local gray-level variance, gradient energy, feature point density, and relevant peak confidence are weighted and fused to obtain a texture quality score. When the texture quality score is not less than a preset score threshold, the area covered by the local window is marked as a matchable area.

5. The method according to claim 1, characterized in that, The process involves performing 3D reconstruction based on the deglare-reduced left and right view image sequences, combined with scale constraints from a portable 3D spatial scale and dual-axis cross-path constraints, to generate multi-stage 3D surface models, including: Control the drone to collect multi-view images along two intersecting routes: one along the route direction and one perpendicular to the route direction; A portable 3D spatial scale is placed in the area to be measured; the portable 3D spatial scale includes at least two endpoint targets. In the multi-view images, the endpoint targets of the portable spatial 3D scale are detected, and the three-dimensional coordinates of the spatial endpoints corresponding to each endpoint target in the reconstruction model are obtained through multi-view geometric calculation. The scale constraint term is determined based on the spatial endpoint and the corresponding physical distance. The scale constraint term is then added to the optimization objective function of the motion-reconstructed structured bundle adjustment model to constrain the global scale of the 3D reconstruction.

6. The method according to claim 1, characterized in that, Based on the multi-phase three-dimensional surface model and the natural speckle field, deformation calculations are performed on the same target area in adjacent inspection cycles within the matching area to obtain deformation field information, including: Within the matchable area, the local windows corresponding to each pixel of the previous 3D surface model are correlated with the corresponding positions of the next 3D surface model to determine the optimal matching position of each pixel. Based on the optimal matching position, the difference in three-dimensional spatial coordinates of the pixel between the previous and next periods is determined to obtain a three-dimensional displacement vector. The displacement gradient is calculated based on the three-dimensional displacement vector of multiple pixels, and the strain component is obtained from the displacement gradient to obtain the strain distribution field; the three-dimensional displacement vector and the strain distribution field together constitute the deformation field information.

7. The method according to claim 1, characterized in that, The step of mapping the macroscopic semantic feature field and the deformation field information to the same three-dimensional coordinate system for collaborative cross-validation and outputting the hidden danger detection result includes: The two-dimensional disease mask is projected onto the surface of the three-dimensional ground model to form a three-dimensional disease area with disease type, confidence level and spatial range; Calculate the macroscopic disease confidence level and microscopic deformation risk value for each of the three-dimensional disease areas; When the confidence level of the macroscopic disease is not less than the preset first confidence level threshold and the risk value of the microscopic deformation is not less than the preset second confidence level threshold, the hazard detection result is output as a high-level hazard area. When the confidence level of the macroscopic disease is not less than the preset first confidence level threshold and the risk value of the microscopic deformation is less than the preset second confidence level threshold, the hidden danger detection result is output as a general disease area. When the confidence level of the macroscopic defects is less than the preset first confidence threshold and the risk value of the microscopic deformation is not less than the preset second confidence threshold, the hidden danger detection result is output as an early structural hidden danger area. When the confidence level of the macroscopic disease is less than the preset first confidence level threshold and the risk value of the microscopic deformation is less than the preset second confidence level threshold, the hazard detection result is output as a normal area.

8. A collaborative detection device for roadbed hazards using unmanned aerial vehicles (UAVs), characterized in that, The device includes: The dynamic radiosity correction module is used to acquire a polarized binocular image sequence, perform dynamic radiosity correction on the polarized binocular image sequence, and generate a deglare left and right view image sequence. The macroscopic disease identification module is used to identify macroscopic diseases in the diffuse reflection images in the deglare left and right view image sequence and generate a macroscopic semantic feature field. The matching region generation module is used to perform natural texture enhancement processing on the diffuse image to generate a natural speckle field and a corresponding matching region. The 3D reconstruction module is used to perform 3D reconstruction based on the deglare left and right view image sequence, combined with the scale constraints of the portable spatial 3D scale and the dual-axis cross-line constraints, to generate a multi-stage 3D surface model. The deformation calculation module is used to perform deformation calculation on the same target area in adjacent inspection cycles within the matching area based on the multi-phase three-dimensional surface model and the natural speckle field, so as to obtain deformation field information. The collaborative cross-validation module is used to map the macroscopic semantic feature field and the deformation field information to the same three-dimensional coordinate system for collaborative cross-validation and output the hazard detection results.

9. An electronic device, characterized in that, The method includes a processor and a memory, wherein the memory stores computer-executable instructions that can be executed by the processor, and the processor executes the computer-executable instructions to implement the UAV roadbed hazard collaborative detection method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when invoked and executed by a processor, cause the processor to implement the UAV roadbed hazard collaborative detection method according to any one of claims 1 to 7.

Images