Unmanned aerial vehicle and video monitoring cooperative water conservancy engineering stereoscopic perception method and system

Abstract

The present application relates to the technical field of intelligent monitoring of water conservancy projects, and in particular to a water conservancy project stereoscopic perception method and system cooperating with a UAV and video monitoring, the method comprising: constructing a unified spatial geometric constraint relationship of air-ground joint perception; converting cross-view geometric inconsistency into scene three-dimensional depth perception signals; performing weighted attention fusion based on the geometric consistency confidence of cross-view features to generate unified construction scene perception representation; performing multi-dimensional evaluation of construction safety, quality and progress based on the fusion representation, mapping to a digital twin model and generating management and control decisions, the present application solves the problem of missing three-dimensional quantification capability caused by independent processing of air and ground views in the prior art, and realizes cross-view cooperation at the perception level.

Description

Technical Field

[0001] This invention relates to the field of intelligent monitoring technology for water conservancy projects, and in particular to a method and system for three-dimensional perception of water conservancy projects that combines unmanned aerial vehicles (UAVs) and video surveillance. Background Technology

[0002] With the rapid development of drone remote sensing technology, video surveillance technology, and artificial intelligence algorithms, the traditional construction management model relying on manual inspection is transforming towards intelligent operation. However, existing technologies still face technical bottlenecks in achieving intelligent control of all elements of construction using multi-source sensing methods.

[0003] Chinese patent application CN120782216A discloses an artificial intelligence-based method and system for monitoring the construction quality and progress of building engineering. This method uses a drone to capture images of the construction site from five directions: front, back, left, right, and bottom. The images are then matched with project drawings using the SIFT algorithm to construct a BIM model. Visual SLAM technology is then used to generate a dynamic 3D point cloud map, which is compared with the volumetric overlap of the BIM model to assess construction progress. A grayscale prediction model is also introduced to predict progress deviations. However, in this scheme, the drone is only used in the initial modeling stage. Continuous monitoring after modeling is completed relies entirely on fixed ground sensors and cameras. These two sensing methods work independently and are simply stitched together at the data layer, failing to establish a geometric relationship between the aerial and ground perspectives at the perception level. This results in the 3D reconstruction accuracy being limited by projection distortion from a single viewpoint and the inability to eliminate blind spots.

[0004] Chinese patent application CN120047873A discloses a digital twin-based progress monitoring method and BIM information management system. This method identifies construction progress by deploying fixed cameras at the construction site to collect images, preprocessing them, extracting regions of interest, and then performing semantic matching. The identification results are then mapped to a digital twin model. However, this solution relies entirely on a single sensing source—the fixed camera—lacking global macroscopic coverage of large-scale construction scenarios and focusing only on the progress dimension without addressing collaborative analysis of safety and quality.

[0005] Chinese patent application CN118628052A discloses a smart construction site management platform based on a public cloud architecture. This platform integrates multiple functional modules, including personnel management, safety management, quality management, progress management, video surveillance management, and machinery and equipment management, using a four-layer cloud architecture. However, the functional modules operate independently in a parallel manner, lacking a coupling analysis mechanism between the three core control dimensions of safety, quality, and progress. This fails to reveal the implicit cross-dimensional correlations such as increased quality risks during rush periods and progress setbacks during quality rectification. Furthermore, the platform does not address the perception-level collaboration between drones and ground monitoring.

[0006] A comprehensive analysis of existing technologies reveals that current solutions generally treat drone aerial photography and ground-based fixed monitoring as two independent sensing tools, performing image analysis in their respective processing flows before information aggregation at the data layer. This approach overlooks a crucial fact: in large-scale open environments with complex terrain and dynamic water areas, such as water conservancy construction projects, the geometric inconsistency between aerial and ground perspectives encodes rich 3D structural information about the scene. Traditional methods treat this perspective inconsistency as alignment errors and attempt to eliminate it in post-processing. However, this approach discards the depth information contained in cross-view parallax, fundamentally limiting the 3D quantization and occlusion removal capabilities of the sensing system. Furthermore, the large water areas unique to water conservancy construction scenarios introduce spurious correspondences due to water surface refraction and specular reflection into cross-view feature matching, challenges specific to this scenario that current technologies fail to address. Finally, existing systems face common problems such as high false negative rates in security hazard detection, lack of three-dimensional measurement methods for quality defect detection, and limited accuracy of progress assessment due to single-view volume estimation errors. At the same time, the three control dimensions of safety, quality, and progress are independent of each other and lack coupled analysis, making it impossible to reveal the implicit correlations and chain risks between them on a unified perception basis. Summary of the Invention

[0007] Addressing the core bottleneck of existing multi-source perception systems for water conservancy engineering construction, which handle aerial and ground perspectives independently and lack cross-perspective 3D quantitative perception capabilities, this invention provides a method and system for 3D perception of water conservancy engineering through collaboration between unmanned aerial vehicles (UAVs) and video surveillance. By constructing joint air-ground geometric constraints at the perception level and inverting cross-perspective inconsistencies into depth perception signals, under the constraints of complex terrain and dynamic water areas in construction scenarios, this invention achieves 3D quantitative perception and multi-scale occlusion elimination of construction elements from the perspective of projective geometry principles. This provides a unified, high-precision perception foundation for safety hazard detection, quality defect measurement, and schedule deviation assessment.

[0008] The technical solution of this invention is as follows:

[0009] A method for three-dimensional perception of water conservancy projects through collaboration between drones and video surveillance includes the following steps:

[0010] Step S1: Construct a unified spatial geometric constraint relationship for air-ground joint perception. The UAV's flight attitude parameters are obtained through the fusion of the inertial measurement unit and satellite navigation system onboard the UAV. External parameter calibration parameters of the ground-based fixed video surveillance equipment are obtained through pre-calibration. Based on the UAV's flight attitude parameters and the ground-based fixed video surveillance equipment's external parameter calibration parameters, the aerial camera projection matrix and the ground camera projection matrix are calculated respectively. According to the aerial camera projection matrix and the ground camera projection matrix, the epipolar geometric constraint relationship between the aerial viewpoint and the ground viewpoint is determined. This epipolar geometric constraint relationship serves as the unified spatial geometric constraint relationship for the air-ground joint perception.

[0011] Step S2: Convert the geometric inconsistency between the aerial viewpoint and the ground viewpoint into a scene 3D depth perception signal, calculate the cross-viewpoint parallax through the epipolar geometric constraints, and invert the depth information of each area of ​​the construction scene.

[0012] Step S3: Multi-scale feature extraction is performed on the aerial view and the ground view respectively. The depth information of each area of ​​the construction scene obtained in step S2 is back-projected onto the ground view image plane and the ground view feature matching point position is measured by Euclidean distance and mapped by Gaussian function to obtain the geometric consistency confidence of cross-view features. Based on the geometric consistency confidence, weighted attention fusion is performed to generate a unified multi-scale construction scene perception representation.

[0013] Step S4: Based on the unified multi-scale construction scene perception representation and the depth information of each area of ​​the construction scene obtained by inversion in step two, construction safety hazard identification, engineering quality defect detection and construction progress deviation assessment are performed respectively. The assessment results are mapped to the digital twin model and control decisions are generated.

[0014] This invention also provides a three-dimensional perception system for water conservancy projects that integrates unmanned aerial vehicles (UAVs) and video surveillance, comprising:

[0015] The air-ground joint calibration module is used to construct the epipolar geometric constraint relationship between the aerial viewpoint and the ground viewpoint based on the UAV flight attitude parameters and the extrinsic calibration parameters of the ground fixed video surveillance equipment.

[0016] The cross-view depth inversion module is used to convert the geometric inconsistency between the aerial view and the ground view into a scene 3D depth perception signal;

[0017] The cross-view fusion perception module is used to extract multi-scale features from the aerial and ground views respectively, and perform weighted attention fusion based on the geometric consistency confidence of the cross-view features to generate a unified multi-scale construction scene perception representation.

[0018] The multi-dimensional management and assessment module is used to perform construction safety hazard identification, engineering quality defect detection and construction progress deviation assessment based on the unified multi-scale construction scene perception representation and the depth information of each area of ​​the construction scene obtained by the cross-view depth inversion module.

[0019] The digital twin mapping and decision-making module is used to map the evaluation results to the digital twin model and generate control decisions.

[0020] The beneficial effects of this invention are as follows:

[0021] First, this invention constructs a unified spatial geometric constraint relationship for joint air-ground perception and reverses cross-view inconsistencies into depth perception signals, enabling the system to naturally acquire three-dimensional quantization capabilities at the perception level. The mechanism is that the geometric differences in imaging of the same scene target from the air viewpoint and the ground viewpoint contain the target's depth information. By mapping parallax to depth values ​​through epipolar geometric constraints, three-dimensional measurement of the construction scene can be achieved without additional depth sensors. Compared with the two-dimensional image analysis scheme where UAVs are only used for initial modeling and subsequent monitoring relies solely on the ground viewpoint, this invention fundamentally solves the problem of insufficient quantization accuracy caused by single-view projection distortion.

[0022] Second, this invention achieves reliable perception fusion of multi-source sensing data rather than simple splicing through a weighted attention fusion mechanism based on geometric consistency confidence. The mechanism is that the geometric consistency confidence of cross-view features provides an intrinsic fusion quality metric. Regions with high confidence receive greater fusion weights, while regions with low confidence are downweighted. This allows the fusion result to adaptively cope with complex scene challenges such as occlusion, reflection, and texture loss. Compared with solutions that rely solely on a single fixed camera, this invention achieves robust perception through multi-view complementarity.

[0023] Third, there is a significant synergistic effect among the three technical features of this invention: air-to-ground cross-view geometric calibration, depth signal inversion, and consistency-weighted fusion. Geometric calibration alone can only achieve coordinate alignment, depth inversion alone can only obtain sparse depth maps, and attention fusion alone lacks a geometric quality metric. The synergistic combination of the three technologies gives rise to system-level capabilities that are impossible to achieve with a single technology, enabling multi-scale occlusion elimination and dynamic target stereo tracking based on three-dimensional quantization perception. The mechanism is that geometric calibration provides the mathematical basis for depth inversion, the depth signal provides a consistency metric for fusion quality assessment, and the consistency metric, in turn, guides the adaptive allocation of fusion weights, forming a positive feedback synergistic loop. Attached Figure Description

[0024] Figure 1 This is a flowchart illustrating the three-dimensional perception method for water conservancy projects that combines drones and video surveillance, as provided in an embodiment of the present invention.

[0025] Figure 2 This is a schematic diagram of the architecture of a three-dimensional perception system for water conservancy projects that integrates drones and video surveillance, provided in an embodiment of the present invention. Detailed Implementation

[0026] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0027] Reference Figure 1 This invention provides a method for three-dimensional perception of water conservancy projects through collaboration between unmanned aerial vehicles (UAVs) and video surveillance. The method is applied to water conservancy project construction sites, where at least one inspection UAV and at least two fixed video surveillance cameras are deployed. The UAV is equipped with a high-resolution aerial camera and an inertial measurement unit (IMU). The fixed video surveillance cameras undergo pre-calibration to determine their intrinsic parameter matrix and extrinsic parameter matrix in the world coordinate system. The method includes the following steps.

[0028] Step S1: Construct unified spatial geometric constraints for air-ground joint perception. The core objective of this step is to establish a strict geometric relationship between the UAV's aerial view and the view of the ground-based fixed camera, so that subsequent steps can utilize the geometric differences between the two viewpoints to extract 3D information of the scene.

[0029] Specifically, the real-time pose parameters of the UAV during its inspection flight are first acquired. These pose parameters include the UAV's three-dimensional position coordinates and three-axis attitude angles in the world coordinate system, calculated by the fusion of the UAV's onboard inertial measurement unit and satellite navigation system, with a position accuracy better than 0.05m and an attitude angle accuracy better than 0.1 degrees. By combining these pose parameters with the intrinsic parameter matrix of the aerial camera, the projection matrix of the UAV's aerial camera at each moment can be calculated. The intrinsic parameter matrix includes focal length, principal point coordinates, and distortion coefficients, which are pre-calibrated using a checkerboard calibration method before UAV deployment.

[0030] Simultaneously, the intrinsic and extrinsic parameter matrices of the pre-calibrated ground-based fixed video surveillance cameras are read, and the projection matrix of the ground cameras is calculated. The ground cameras are calibrated using a total station during deployment at the construction site to determine their position and orientation in the world coordinate system. After obtaining the projection matrices of both the aerial and ground cameras, the fundamental matrix between them is calculated based on epipolar geometry principles. The fundamental matrix establishes the constraint relationship between each pixel in the aerial view image and the corresponding epipolar line in the ground view image. It should be noted that because the UAV's pose continuously changes during flight, the fundamental matrix is ​​dynamically updated over time; it is recalculated whenever the UAV's pose parameters are updated to maintain the real-time effectiveness of the geometric constraints.

[0031] In actual water conservancy construction scenarios, drones inevitably experience airflow disturbances during flight, especially in typical water conservancy construction environments such as dam crests, canyon channels, and open water surfaces. Valley winds, thermal convection, and humidity gradients generated by water surface evaporation all exacerbate the aircraft's attitude fluctuations. Traditional methods treat these fluctuations as error sources and attempt to eliminate them through electronic image stabilization or mechanical stabilization. This invention takes the opposite approach, utilizing the image geometric distortion caused by drone attitude fluctuations as an active parallax perturbation. The physical basis of this inverted strategy is that attitude fluctuations are equivalent to multiple samplings of the same scene with minute viewpoint changes within a very short time interval. Although the parallax information generated by these minute viewpoint changes is small in magnitude, it has a significant enhancement effect on detecting slow-moving dynamic targets at the construction site.

[0032] Specifically, the changes in attitude angular velocity during the UAV's flight were extracted, and frequency domain analysis was performed on the aerial image sequence. In the frequency domain, the image motion component caused by the aircraft's own motion has characteristic frequencies related to the aircraft's inherent vibration modes, typically concentrated in the 2Hz to 15Hz frequency band; while the image motion component caused by the dynamic target motion at the construction site is distributed across different frequency bands. Adaptive bandpass filtering was used to separate the two types of motion components. After separation, the geometric distortion caused by the aircraft's attitude fluctuations was extracted as an independent motion field, which contains additional parallax information generated by small changes in viewing angle.

[0033] The additional disparity information is then subjected to motion enhancement processing. The motion enhancement coefficient is defined as follows:

[0034] ,

[0035] in: For pixels The motion enhancement coefficient at a given pixel is a scalar, ranging from 0 to positive infinity, dimensionless, and calculated using this formula. It characterizes the significance of the target's motion relative to the aircraft's own motion at that pixel location. A higher value indicates a greater likelihood that there are dynamic targets in the scene at that location that are independent of the aircraft's motion; For pixels The total motion vector of the scene at that location is a two-dimensional vector with units of pixels / frame. It is estimated by optical flow and includes the superposition of the aircraft motion and the scene target motion. For pixels The motion component generated by the aircraft's own motion is a two-dimensional vector with units of pixels / frame, and is determined by the aircraft's motion field after frequency domain separation. To prevent division by zero, the stability constant is a scalar with a value of [value missing]. It is dimensionless. If the value is too large, it will reduce the sensitivity of the enhancement coefficient. If the value is too small, it will lead to numerical instability. This represents the L2 norm of a two-dimensional vector.

[0036] Exercise enhancement coefficient Exceeding the preset threshold The pixel region is marked as a candidate dynamic target region. The value ranges from 1.5 to 5.0, and 3.0 is preferred in this embodiment. This value is determined based on the ratio of the movement speed of typical dynamic targets at the construction site to the attitude fluctuation amplitude of the UAV. If the value is too low, the false detection rate will increase, and if the value is too high, the slow-moving targets will be missed.

[0037] Step S2: Convert cross-view geometric inconsistencies into scene 3D depth sensing signals. Based on the epipolar geometric constraints established in Step S1, this step utilizes the parallax between the aerial view and the ground view to invert the 3D depth information of the construction scene.

[0038] Specifically, for each feature point in the aerial view image, a corresponding matching point is searched in the ground view image along the epipolar line direction determined by the epipolar geometric constraints. During the search, the normalized cross-correlation coefficient (CCC) is used as the matching metric. The CCC is calculated between the neighborhood image patch of the feature point in the aerial view and the neighborhood image patch of the candidate position on the epipolar line in the ground view. The position with the highest CCC that exceeds the matching threshold is selected as the matching point. When a match is successful, the difference in pixel coordinates between the corresponding points in the two views is the cross-view parallax. Based on the principle of triangulation, the three-dimensional coordinates of the target point are derived from the parallax and the baseline distance between the two cameras. The baseline distance is the Euclidean distance between the optical center of the UAV aerial camera and the optical center of the ground camera, typically ranging from 10m to 200m, depending on the UAV's flight altitude and the position of the ground camera.

[0039] In water conservancy engineering construction scenarios, the presence of water surfaces presents unique challenges to cross-view feature matching. When the light path from either an aerial or ground-based perspective passes over the water surface, the incident light refracts at the water-air interface, causing the standard epipolar geometry constraints to fail. Specifically, specular reflection from the water surface generates virtual images in the image. These virtual images constitute spurious feature correspondences, which, if not identified and eliminated, will severely interfere with the accuracy of depth inversion. Furthermore, for construction targets below or close to the water surface (such as cofferdam foundations and exposed underwater pile foundations), refraction causes their apparent position in the aerial perspective to deviate from their true projected position, with the offset exhibiting a non-linear relationship with water depth and incident angle. This invention constructs a water surface refraction perception geometry model to address this problem.

[0040] Specifically, the spatial location and normal vector of the water surface are first determined based on the water distribution information and water level monitoring data of the construction area. For target points observed through the water surface refraction path, their imaging position in the aerial view is affected by refraction offset, requiring correction of the standard epipolar geometry constraint.

[0041] Define the cross-view correspondence after refraction correction:

[0042] ,

[0043] in: , is the aerial viewpoint pixel coordinate after refraction correction, is a two-dimensional vector, is calculated by this formula, and represents the equivalent imaging position of the target in the aerial viewpoint image after considering the water surface refraction effect. These are the original pixel coordinates observed in the aerial view image. They are two-dimensional vectors, with units of pixels, and are obtained directly from image acquisition. This represents the pixel offset caused by water surface refraction. It is a two-dimensional vector in pixels and is determined by geometric calculations of the refracted light path.

[0044] The calculation of the refracted pixel offset is based on Snell's law of refraction:

[0045] ,

[0046] in: The focal length of the aerial camera is a scalar value in pixels, ranging from 500 to 5000. It is obtained from camera calibration. The larger the focal length, the greater the projection of the refraction offset onto the image plane. Let be the depth of the target point below the water surface, be a scalar in meters (m), and range from 0 to 20 m. This depth is estimated from prior water depth information or through multi-time iterative observations. Considering the potential cyclic dependency between refraction correction and water depth inversion, this embodiment employs a dual-track mechanism combining zero-order cold start and multi-time iterative convergence. The first track is a zero-order cold start: for water conservancy project construction areas where water level monitoring stations, hydrological measurement buoys, or prior underwater topographic mapping data have been deployed, The initial value is directly taken from the difference between the real-time water level output by the water level monitoring system and the pre-mapped underwater topographic elevation, without relying on cross-view inversion; the second track is multi-time iterative convergence: for temporary construction areas lacking water level monitoring facilities, the first iteration sets... =0 means assuming no refraction, and the initial depth is obtained through uncorrected epipolar geometry constraints. Then Substitute the values ​​into Snell's refraction formula to calculate the first refraction shift and update the equivalent imaging position. The second iteration is based on Re-perform feature matching and deep inversion to obtain This process is repeated until two consecutive iterations yield the result. The iteration terminates when the L2 norm of the difference is less than a preset convergence threshold, which is set to 0.05m in this embodiment. Under typical water texture conditions at a water conservancy construction site, the iteration process usually converges within 3 to 5 rounds. The larger the value, the more significant the refractive shift. The angle between the incident ray and the normal vector to the water surface is called the angle of incidence. It is a scalar with units of rad and a value range of 0 to 1. rad, derived from the position of the aerial camera and the water surface normal vector. The geometric relationship between the target point and its location is determined; Let be the refractive index of air, a scalar with a value of 1.000, dimensionless, and a physical constant. Let be the refractive index of water, be a scalar with a value of 1.333, dimensionless, and a physical constant. is the unit vector of the refraction offset direction. It is a two-dimensional vector, dimensionless, and is determined by the direction of the intersection of the incident plane and the image plane.

[0047] After refraction correction, use the corrected coordinates. Replace the original coordinates Participates in cross-view feature matching and depth inversion. For regular target points that do not follow the water surface refraction path, The correction does not affect the normal calculation process. By correcting the cross-viewpoint epipolar geometric constraints of paths reflected or refracted through the water surface, a refraction-corrected epipolar geometric constraint is obtained. This eliminates the false cross-viewpoint feature correspondences caused by water surface specular reflection, allowing underwater or near-water surface construction targets observed through refraction paths to be included in the unified spatial geometric constraint relationship of the air-ground joint perception. This refraction perception geometric model enables water surface areas and near-water surface construction targets to be effectively incorporated into the stereo perception framework.

[0048] Step S3: Weighted attention fusion based on geometric consistency confidence of cross-view features. Building upon the depth inversion completed in Step S2, this step performs reliability fusion on the feature maps from the aerial and ground views to generate a unified perception representation of the construction scene.

[0049] First, multi-scale feature maps are extracted from both aerial and ground-view images using a pyramid-type multi-resolution encoder. The pyramid-type encoder extracts features at three levels: original resolution, 2x downsampling resolution, and 4x downsampling resolution. A learnable scale alignment module then aligns the features from different resolution levels to the same spatial scale.

[0050] Then, the geometric consistency confidence of the cross-view features is calculated. For each spatial location in the fusion region, the corresponding features of that location in the aerial and ground views are matched and evaluated. The evaluation criterion is whether the cross-view feature pairs of that location satisfy the geometric consistency under the depth information constraints established in step S2.

[0051] Define the geometric consistency confidence level:

[0052] ,

[0053] in: For spatial location The geometric consistency confidence of the cross-view feature at a location is a scalar with a value range of 0 to 1. It is dimensionless and is calculated by this formula. It represents the reliability of the geometric correspondence between the aerial view and ground view feature pairs at that location. The closer the value is to 1, the higher the geometric consistency. For spatial location The three-dimensional coordinates are three-dimensional vectors with units of m, and are calculated jointly from the depth inversion results of step S2 and the camera's intrinsic and extrinsic parameters; This is a projection function that projects three-dimensional coordinates onto the ground-view image plane. The output is a two-dimensional vector in pixels, determined by the projection matrix of the ground camera. The coordinates of the corresponding pixels in the ground view image are obtained through feature matching. They are two-dimensional vectors with units of pixels. The scale parameter for geometric consistency is a scalar with a unit of pixel and a value range of 1.0 to 10.0 pixels. In this embodiment, 3.0 pixels is preferred. This value is determined based on the camera calibration accuracy and depth inversion error statistics. If the value is too small, a large number of valid matches will be judged as low confidence. If the value is too large, the ability to filter unreliable matches will be reduced. This represents the L2 norm of a two-dimensional vector.

[0054] The geometric consistency confidence score is used to perform weighted fusion of multi-source features:

[0055] ,

[0056] in: For spatial location The fusion-sensory feature vector at the location is The dimensionless vector, calculated by this formula, represents the unified perception representation of the location that integrates information from both the air and ground perspectives. For aerial perspective at position The feature vector is . A dimensionless vector, extracted by a pyramid encoder; For ground perspective at position The feature vector is . A dimensionless vector, extracted by a pyramid encoder; , and These are the aerial view weight matrix, the ground view weight matrix, and the cross-view interaction weight matrix, respectively. The dimensional learnable parameter matrix is ​​dimensionless and is obtained through end-to-end training via an attention mechanism. The definition is the same as the formula mentioned above; This represents the element-wise multiplication operation of the eigenvectors; is the feature dimension, a positive integer scalar with a value of 256, and is dimensionless. (The third term in the formula...) The product reaches its maximum value when the consistency is at an intermediate level, at which point the cross-perspective interaction information is the richest. Its technical effect lies in enhancing the complementary analysis capability of two perspectives in uncertain regions.

[0057] During feature fusion, a sparse-dense progressive cross-view feature propagation strategy is adopted to expand the effective fusion area. Specifically, in the refraction correction and depth inversion stage of step S2, high-confidence depth estimates and feature correspondences are obtained only in regions with rich texture and reliable feature matching. These regions constitute a sparse set of seed nodes. In water conservancy construction sites, textured regions typically correspond to the surface of concrete pours, steel mesh structures, formwork joints, and mechanical equipment surfaces, while low-texture regions such as water surfaces, exposed soil surfaces, and large-area homogeneous concrete curing surfaces are areas where reliable feature matching is difficult to obtain in the initial stage. Starting from the seed nodes, feature correspondences and depth information are gradually propagated along the spatial neighborhood direction based on the scene geometric continuity assumption. The geometric continuity assumption holds that the depth values ​​and surface normals of adjacent spatial locations should change continuously in the absence of occlusion boundaries and abrupt depth changes. This assumption holds true in most construction scenarios, requiring special handling only at building edges and depth jumps.

[0058] Define the propagation consistency score:

[0059] ,

[0060] in: To start from the seed node to adjacent nodes The consistency score during propagation is a scalar, ranging from 0 to 1, dimensionless, calculated by this formula, and characterizes the consistency from... Towards Does the depth propagation conform to the local plane fitting constraint? To propagate from node Extrapolation to nodes The predicted depth value is a scalar in meters, generated by the seed node. The local planar model is determined; Seed node Local planes fitted to its neighborhood At the node The interpolation depth at the point is a scalar in meters, calculated by least-squares plane fitting. The scale parameter for propagation consistency is a scalar with units of meters and a value range of 0.05 to 0.5m. In this embodiment, 0.1m is preferred. If the value is too small, it will limit the propagation range and make the final dense graph too sparse. If the value is too large, it will allow unreliable depth propagation, leading to error accumulation.

[0061] when Below the propagation threshold Stop sending to the node directional diffusion. The value ranges from 0.3 to 0.8, with 0.5 being preferred in this embodiment. Values ​​that are too low can introduce errors when propagating to geometrically inconsistent areas, while values ​​that are too high can cause propagation to stop prematurely, resulting in insufficient dense coverage.

[0062] Building upon spatial propagation, a spatiotemporal consistency graph constraint is further constructed. Feature points at the same spatial location collected at different times are used as graph nodes, and the feature correspondences between temporally adjacent frames are used as graph edges. Utilizing a crucial physical characteristic of hydraulic engineering construction—the monotonically progressive nature of construction activities (i.e., poured concrete cannot be removed and installed structures cannot be dismantled)—the propagation direction on the graph is constrained. The temporal propagation direction constraint is defined as follows:

[0063] ,

[0064] in: From time Towards the moment The permission flag for propagation is a scalar with a value of 0 or 1. It is dimensionless and is calculated by this formula. A value of 1 indicates that propagation is permitted, and a value of 0 indicates that propagation is prohibited. and They are time points and The quantitative value of construction progress is a scalar, ranging from 0 to 1, dimensionless, and is calculated from the ratio of the volume of the completed engineering entities within the construction area to the total design volume, representing the degree of completion of the construction progress.

[0065] The monotonous progression of the construction schedule means It is a non-decreasing function of time. When this constraint is violated (i.e., the progress value at a later time step is lower than that at a previous time step), it indicates that the feature matching at the corresponding time step has been interfered with by factors such as changes in illumination, weather, or significant changes in the appearance of the construction, and the feature propagation path is unreliable. By truncating these unreliable propagation paths, the cumulative propagation of temporal errors is avoided, and ultimately, a spatiotemporally consistent, dense perception result across the entire domain is obtained.

[0066] Step S4: Multidimensional Management and Control Assessment and Digital Twin Mapping. Based on the fused perception representation obtained in Step S3 and the deep information obtained in Step S2, this step performs construction status assessment from three dimensions: safety, quality, and schedule, and maps the results to a digital twin model.

[0067] In terms of safety, the system detects and analyzes personnel behavior and the work environment at the construction site based on fused perception representation. It utilizes a target detection network to identify elements such as construction workers, machinery, and safety protection facilities, and combines depth information to determine safety indicators such as the 3D distance between personnel and hazardous areas, the safety belt wearing status of workers at heights, and the degree of overlap between the operating range of machinery and the personnel's activity area. In 3D space, when the shortest 3D distance between personnel and operating machinery is less than the safety warning distance, the system generates an immediate warning; when personnel enter the preset hazardous area on the water-facing edge of the cofferdam, the system marks a high-risk area. This safety detection benefits from the 3D depth information provided in steps S2 and S3, enabling the measurement of distance and spatial relationships in real 3D space rather than a 2D image plane, avoiding distance misjudgments caused by 2D projection. By integrating multiple safety indicators, a safety risk sub-index is generated through weighted normalization calculation. .

[0068] In terms of quality, geometric quantitative analysis of engineering entities is performed using fused perception representation and depth information. By comparing the 3D reconstruction results of the current construction state with the design BIM model, quality issues such as structural dimensional deviations, abnormal surface flatness, and appearance defects are detected. Specifically, the global dense depth map output in step S3 is converted into a 3D point cloud. An iterative nearest-point algorithm aligns the measured point cloud with the design BIM model to the same coordinate system. The normal distance between the measured point cloud and the surface of the design model is calculated point by point. When the normal distance exceeds the allowable quality deviation, it is marked as a quality deviation point. In water conservancy engineering construction, quality issues such as geometric deviations of concrete pours, flatness of formwork joints, cracks and honeycomb pitting on the dam surface can all be quantitatively detected using this method. A quality deviation sub-index is generated by integrating multiple quality indicators. .

[0069] In terms of schedule, the current 3D volume of the construction status is compared with the target volume of the design model, the deviation between the actual completion rate and the planned completion rate of each construction zone is calculated, and a schedule deviation sub-index is generated. Specifically, the construction area is divided into several independent construction zones according to the design drawings. For each zone, the ratio of the volume enclosed by the measured 3D point cloud to the target volume of the design model is calculated as the actual completion ratio of the zone. This ratio is then subtracted from the planned completion ratio of the zone in the project schedule to obtain the progress deviation of the zone. The progress deviations of all zones are weighted and summed to obtain the overall progress deviation sub-index.

[0070] Based on the three sub-indices mentioned above, a construction health coupling evaluation index is calculated. Unlike traditional methods that evaluate the three dimensions independently, this invention reveals and quantifies the implicit correlation between them. For example, improvements in schedule deviations during the rush phase are often accompanied by increases in quality and safety risks; during the quality rectification phase, schedules may temporarily regress, and safety risks may increase due to rework. This cross-dimensional coupling effect cannot be captured by independent evaluations of a single dimension.

[0071] Define the construction health coupling evaluation index:

[0072] ,

[0073] in: The construction health evaluation index is a scalar quantity with a value range of 0 to 1. It is dimensionless and is calculated by this formula. The closer the value is to 0, the healthier the construction status is, and the closer the value is to 1, the higher the risk. , , These are the safety risk sub-index, quality deviation sub-index, and schedule deviation sub-index, respectively. They are all scalars, ranging from 0 to 1, and are dimensionless. They are obtained by normalizing the evaluation results of each dimension. , , The linear weight coefficients for safety, quality, and schedule are scalars, ranging from 0 to 1, with the sum of the three being 1. They are dimensionless, and in this embodiment, they are preferably 0.4, 0.3, and 0.3, respectively. The weight allocation reflects the safety-first management strategy in water conservancy project construction. , , These are the coupling coefficients between safety-quality, safety-schedule, and quality-schedule, respectively. They are all scalars, ranging from -0.5 to 0.5, and are dimensionless. They are determined by statistical regression analysis of historical construction data. In this embodiment, they are preferably 0.15, 0.20, and 0.10, respectively. Positive values ​​indicate that there is a positive mutual stimulation effect between the two dimensions of risk, that is, an increase in the risk of one dimension will aggravate the risk of the other dimension. Negative values ​​indicate that there is a hedging effect between the two dimensions.

[0074] When the change of any sub-index causes When the response is non-proportional, it indicates that the change has triggered a chain reaction of risks in other dimensions through the coupling coefficient, thus triggering a cross-dimensional joint early warning. Specifically, the non-proportional response detection condition is defined as follows: when... Significantly deviates from its corresponding linear weight At that time, that is This triggers a cross-dimensional joint early warning. The value ranges from 0.05 to 0.20, and 0.10 is preferred in this embodiment.

[0075] In terms of setting early warning thresholds, a Bayesian adaptive update mechanism is adopted. Water conservancy project construction is a dynamic process spanning several months or even years, and the risk characteristics of different construction stages are significantly different. Using a fixed threshold will lead to a high false alarm rate in the early stage and a high false alarm rate in the later stage.

[0076] Define the Bayesian threshold update formula:

[0077] ,

[0078] in: For the first The first control dimension in the The warning threshold for each construction stage is a scalar with a value range of 0 to 1. It is dimensionless and is calculated by this formula. It represents the critical value for judging the risk anomaly in this stage. For the first The first control dimension in the The prior threshold for each construction stage is a scalar with a value range of 0 to 1. It is dimensionless and is calculated from the previous stage. The first stage takes the empirical initial value. For the first The first control dimension in the The mean of the actual deviation distribution of each construction stage is a scalar, ranging from 0 to 1, dimensionless, and is determined by the statistical mean of the perception assessment results of that stage. is the prior strength parameter, which is a scalar with a value range of 1 to 20 and is dimensionless. In this embodiment, 5 is preferred to control the weight of historical experience in threshold update. The larger the value, the more conservative the threshold update. The smaller the value, the easier it is for the threshold to be skewed by the current data. Here, is the likelihood intensity parameter, a scalar value ranging from 1 to 20, dimensionless, and preferably 3 in this embodiment, controlling the weight of the current observation data in the threshold update. (Subscript) Values , or These correspond to the three dimensions of safety, quality, and schedule, respectively.

[0079] The evaluation results and coupled evaluation indices of the above dimensions are mapped to a pre-constructed digital twin model of the water conservancy project. The digital twin model uses the BIM model as its geometric base and renders the real-time perception results in a color-coded manner onto the corresponding spatial location of the 3D model, realizing a visual presentation of the construction status.

[0080] In the feedback loop mechanism of the digital twin model, based on the weak perception areas and high-risk areas marked in the model, the next round of inspection flight routes of the UAV and the gimbal turning angle of the ground video monitoring equipment are adjusted in reverse. Specifically, the spatial coordinates of weak perception areas and high-risk areas are extracted into a set of key areas of focus. The flyby density and hovering time of these areas are increased in the UAV flight path planning, and the orientation of the adjustable gimbal is deflected to the key areas in the camera scheduling. This allows perception resources to adaptively tilt towards the areas with the most urgent current control needs, forming a closed-loop optimization of perception, assessment, decision-making, and re-perception.

[0081] The feedback closed-loop mechanism operates in sync with the construction inspection cycle. At the end of each inspection cycle, the system automatically summarizes the statistical results of the perception coverage and the changes in risk ratings for each area, generating a perception resource allocation plan for the next cycle. During regular construction, the inspection cycle is typically 1 to 2 times per day; during critical construction processes or when the risk level increases, the inspection cycle can be automatically increased to once every 2 to 4 hours. The UAV flight path planning algorithm adds circling flight segments to key areas of concern, reducing the segment spacing from the normal 50m to 20m to improve the image coverage density and depth inversion accuracy of these areas. The ground camera gimbal scheduling strategy calculates the optimal combination of gimbal yaw and pitch angles based on the angular relationship between key areas and cameras and the field of view coverage, maximizing the overall coverage area while meeting the coverage requirements of key areas. Through this closed-loop optimization mechanism, the system's perception capabilities can continuously and adaptively match the dynamically changing control needs of the construction site, avoiding ineffective and dispersed allocation of perception resources.

[0082] Reference Figure 2 The present invention also provides a three-dimensional perception system for water conservancy projects that integrates unmanned aerial vehicles (UAVs) and video surveillance, which is used to implement all the method steps in the above method embodiments, including the following modules.

[0083] The air-to-ground joint calibration module, corresponding to step S1 in the method embodiment, is used to construct the epipolar geometric constraint relationship between the aerial viewpoint and the ground viewpoint based on the UAV flight attitude parameters and the extrinsic calibration parameters of the ground-based fixed video surveillance equipment. This module receives attitude data output from the UAV's inertial measurement unit and satellite navigation system, as well as pre-calibrated ground camera parameters, calculates the fundamental matrix, and establishes epipolar constraints. Internally, this module maintains a dynamic parameter buffer to store all attitude time-series data of the UAV in the most recent flight cycle, supporting real-time updates of the fundamental matrix. When the UAV attitude data update frequency is lower than the ground camera frame rate, this module calculates intermediate attitude parameters precisely aligned with the ground video frame timestamps through attitude interpolation, ensuring strict synchronization of cross-viewpoint geometric constraints in the time dimension. This module also includes a motion enhancement subunit, used to separate the image distortion caused by UAV flight attitude fluctuations into the aircraft's own motion components and scene target motion components, and to mark candidate dynamic target regions according to the frequency domain separation and motion enhancement coefficient calculation methods described in the method embodiment.

[0084] The cross-view depth inversion module, corresponding to step S2 in the method embodiment, is used to convert the geometric inconsistency between the aerial view and the ground view into a scene 3D depth perception signal. This module performs feature matching search along the epipolar direction in the ground view image based on epipolar geometric constraints, and uses parallax and baseline distance for triangulation to invert depth. Internally, this module maintains a water area mask layer, labeling the current water area distribution range based on geographic information system data and real-time water level sensor data of the construction area. The module also includes a water surface refraction correction subunit, used to correct the epipolar geometric constraints in construction scenes with water areas based on the water surface normal vector and the law of refraction. It calculates pixel offsets according to the refraction correction formula described in the method embodiment, eliminating false feature correspondences caused by water surface reflection, and bringing near-water and underwater construction targets into the stereo perception range. The output of this subunit includes the refraction-corrected corresponding coordinates and refraction confidence markers.

[0085] The cross-view fusion perception module, corresponding to step S3 in the method embodiment, is used to extract multi-scale features from the aerial and ground views respectively, and perform weighted attention fusion based on the geometric consistency confidence of the cross-view features to generate a unified multi-scale construction scene perception representation. This module includes three sub-units: a pyramid multi-scale encoding sub-unit, used to extract aerial and ground view features at three resolution levels and perform cross-resolution alignment using a learnable alignment module, which uses a combination of convolutional layers and upsampling layers to achieve spatial alignment and channel mapping between feature maps of different resolutions; a geometric consistency evaluation sub-unit, used to calculate the cross-view consistency score for each spatial location using the depth inversion results according to the geometric consistency confidence formula described in the method embodiment; and a sparse-dense propagation sub-unit, used to perform progressive feature propagation and spatiotemporal consistency map constraints, gradually expanding the perception capability of sparse reliable areas to full-domain dense coverage according to the propagation consistency score and construction monotonicity constraints described in the method embodiment. The output of this module is a fused perception feature map and a full-domain dense depth map, which together constitute the input for subsequent multi-dimensional control and evaluation.

[0086] The multi-dimensional management and control assessment module, corresponding to the assessment part of step S4 in the method embodiment, is used to perform construction safety hazard identification, engineering quality defect detection, and construction progress deviation assessment based on the unified multi-scale construction scene perception representation and the depth information of each area of ​​the construction scene obtained by the cross-view depth inversion module. This module includes three parallel processing channels: a safety assessment sub-unit, a quality assessment sub-unit, and a progress assessment sub-unit. These sub-units calculate safety risk sub-indices according to the safety detection, quality difference comparison, and progress volume comparison methods described in the method embodiment. Quality deviation sub-index and schedule deviation sub-index The three sub-units share a fused sensing feature map and a depth map as a unified input, but each independently executes domain-specific analysis logic. This module also includes a coupling evaluation sub-unit for calculating the coupling evaluation index according to the construction health coupling evaluation formula described in the method embodiment. A joint early warning is triggered when a cross-dimensional non-proportional response is detected. The joint early warning information includes three fields: triggering dimension identifier, description of the coupled impact path, and suggested response measures.

[0087] The digital twin mapping and decision-making module, corresponding to the mapping and decision-making part of step S4 in the method embodiment, is used to map the evaluation results to the digital twin model and generate control decisions. This module uses the BIM model as its geometric base, and through coordinate transformation, maps the measured 3D point cloud and evaluation results from the perception coordinate system to the BIM model coordinate system. It renders the safety risk level, quality deviation degree, and progress completion status onto the corresponding spatial locations of the 3D model using color coding. Specifically, the safety level uses red-yellow-green three-color coding, the quality deviation uses a gradient of blue shades, and the progress status uses grayscale coding. Based on the weak perception areas and high-risk areas marked in the digital twin model, this module automatically generates recommended routes for the next round of UAV inspections using a route planning algorithm, prioritizing coverage of high-risk areas and areas lacking depth information from the previous inspection. Simultaneously, it sends orientation adjustment commands to the ground camera of the adjustable gimbal, achieving closed-loop optimization of perception, evaluation, decision-making, and re-perception. The module also includes a Bayesian threshold update subunit, which is used to dynamically adjust the warning thresholds of each dimension according to the deviation distribution of historical construction stages and the current observation data in accordance with the Bayesian update formula described in the method embodiment. The threshold update is automatically executed once when each construction stage is switched.

[0088] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for three-dimensional perception of water conservancy projects using a combination of unmanned aerial vehicles (UAVs) and video surveillance, characterized in that: Includes the following steps: Construct a unified spatial geometric constraint relationship for joint air-ground perception, obtain the UAV flight attitude parameters by fusing the inertial measurement unit and satellite navigation system carried by the UAV, and obtain the external parameter calibration parameters of the ground fixed video monitoring equipment through pre-calibration; Based on the flight attitude parameters of the UAV and the external parameter calibration parameters of the ground fixed video surveillance equipment, the projection matrix of the aerial camera and the projection matrix of the ground camera are calculated respectively. Based on the aerial camera projection matrix and the ground camera projection matrix, the epipolar geometric constraint relationship between the aerial viewpoint and the ground viewpoint is determined, and the epipolar geometric constraint relationship serves as the unified spatial geometric constraint relationship for the air-ground joint sensing. The geometric inconsistency between the aerial viewpoint and the ground viewpoint is converted into a scene 3D depth perception signal. The cross-viewpoint parallax is calculated through the epipolar geometric constraints, and the depth information of each area of ​​the construction scene is obtained by inversion. Multi-scale feature extraction is performed on the aerial and ground perspectives respectively. The depth information of each area of ​​the construction scene obtained by inversion is back-projected onto the ground perspective image plane and the location of the ground perspective feature matching point is measured by Euclidean distance. The geometric consistency confidence of the cross-view feature is obtained by mapping the Euclidean distance through a Gaussian function. Weighted attention fusion is performed based on the geometric consistency confidence to generate a unified multi-scale construction scene perception representation. Based on the unified multi-scale construction scene perception representation and the depth information of each area of ​​the construction scene obtained by inversion, construction safety hazard identification, engineering quality defect detection and construction progress deviation assessment are performed respectively. The assessment results are mapped to the digital twin model and control decisions are generated.

2. The method as described in claim 1, characterized in that, The construction of unified spatial geometric constraints for joint air-ground sensing also includes: The change in attitude angular velocity during the flight of the UAV is used as an active parallax disturbance source. The motion components of the UAV itself and the motion components of the scene target are separated in the frequency domain. The image geometric distortion caused by the instability of the UAV attitude is reversed into a motion enhancement detection signal for dynamic targets at the construction site.

3. The method as described in claim 2, characterized in that, Also includes: In construction scenarios involving water, a water surface refraction perception geometry model is constructed based on the estimation of the water surface normal vector and the refraction law of the incident-refracted light path. The cross-viewpoint epipolar geometric constraints of the reflected or refracted light path through the water surface are then corrected to obtain the refraction-corrected epipolar geometric constraints.

4. The method as described in claim 3, characterized in that, Weighted attention fusion based on the geometric consistency confidence also includes: A sparse-dense progressive cross-view feature propagation strategy is adopted. Based on the refraction-corrected epipolar geometry constraints, feature matching is performed along the epipolar direction between aerial and ground views. Matching points with normalized cross-correlation coefficients exceeding a preset matching threshold are designated as high-confidence sparse matching feature points and used as seed nodes. Based on scene geometric continuity constraints, feature correspondences and depth information are gradually propagated along the spatial neighborhood direction to the region to be propagated, beyond the high-confidence sparse matching feature points. During the propagation process, a propagation consistency score is calculated using local plane fitting constraints. The propagation consistency score is obtained by Gaussian function mapping the difference between the predicted depth value extrapolated from the seed node to the adjacent nodes in the region to be propagated and the interpolated depth of the local plane fitting at the adjacent nodes. When the propagation consistency score is lower than the preset propagation threshold, the propagation stops and the region is marked as unreliable.

5. The method as described in claim 4, characterized in that, The sparse-dense progressive cross-view feature propagation strategy also includes spatiotemporal consistency graph constraints: A cross-time feature association graph is constructed, with feature points at the same spatial location at different acquisition times as graph nodes and feature correspondences between temporally adjacent frames as graph edges. The monotonically progressive physical constraint of the construction state of the water conservancy project is used to set temporal direction restrictions on the propagation on the graph, prohibiting features from propagating along the backward direction of the construction state. This eliminates the interruption of temporal feature propagation caused by changes in lighting and construction appearance, and generates spatiotemporally consistent global dense perception results.

6. The method as described in claim 5, characterized in that, After performing the identification of construction safety hazards, detection of engineering quality defects, and assessment of construction progress deviations, the process also includes: Based on the spatiotemporally consistent global dense perception results, the construction health coupling evaluation index is calculated. The construction health coupling evaluation index is obtained through a nonlinear coupling function of the safety risk sub-index, the quality deviation sub-index, and the schedule deviation sub-index. The nonlinear coupling function captures the implicit correlation between the three dimensions. When a change in any sub-index causes a non-proportional response in the coupling evaluation index, a cross-dimensional joint early warning is triggered.

7. The method as described in claim 1, characterized in that, After mapping the evaluation results to a digital twin model and generating control decisions, the process also includes: Based on the weak perception areas marked in the digital twin model, the flight path of the UAV for the next round of inspection and the gimbal turning angle of the ground video monitoring equipment are adjusted in reverse, so that the perception resources are adaptively tilted towards the areas with the most urgent current control needs, forming a closed-loop optimization of perception, assessment, decision-making and re-perception.

8. The method as described in claim 1, characterized in that, The multi-scale feature extraction employs a pyramid-type multi-resolution encoder, extracting feature maps from the aerial and ground perspectives at the original resolution, double downsampled resolution, and quadruple downsampled resolution, respectively. Before performing the weighted attention fusion, the feature maps at different resolution levels are aligned across resolutions using a learnable scale alignment module.

9. The method as described in claim 1, characterized in that, The early warning thresholds in the control decision adopt a Bayesian adaptive update mechanism, which takes the actual deviation distribution of the historical construction stage as the prior distribution and the current perception result as the likelihood function. The safety hazard early warning threshold, quality defect judgment threshold and schedule deviation tolerance threshold are dynamically updated through Bayesian posterior inference, so that the thresholds automatically match the current working condition characteristics as the construction stage evolves.

10. A three-dimensional perception system for water conservancy projects that integrates unmanned aerial vehicles (UAVs) and video surveillance, used to implement the method as described in any one of claims 1 to 9, characterized in that, include: The air-to-ground joint calibration module is used to calculate the aerial camera projection matrix and the ground camera projection matrix respectively by using the flight attitude parameters of the UAV obtained by the inertial measurement unit and satellite navigation system on the UAV and the external parameter calibration parameters of the ground fixed video monitoring equipment obtained by pre-calibration. Based on the aerial camera projection matrix and the ground camera projection matrix, the epipolar geometric constraint relationship between the aerial view and the ground view is constructed. The epipolar geometric constraint relationship serves as the unified spatial geometric constraint relationship for air-to-ground joint perception. The cross-view depth inversion module is used to convert the geometric inconsistency between the aerial view and the ground view into a scene 3D depth perception signal; The cross-view fusion perception module is used to extract multi-scale features from the aerial view and the ground view respectively. By back-projecting the scene's three-dimensional depth perception signal to the ground view image plane and performing Euclidean distance measurement on the ground view feature matching point position, and then mapping it with a Gaussian function, the geometric consistency confidence of the cross-view features is obtained. Based on the geometric consistency confidence, weighted attention fusion is performed to generate a unified multi-scale construction scene perception representation. The multi-dimensional management and assessment module is used to perform construction safety hazard identification, engineering quality defect detection and construction progress deviation assessment based on the unified multi-scale construction scene perception representation and the depth information of each area of ​​the construction scene obtained by the cross-view depth inversion module. The digital twin mapping and decision-making module is used to map the evaluation results to the digital twin model and generate control decisions.

Images