Rockfill body image recognition tracking and earth-rock dam surface displacement monitoring method

Abstract

The present application relates to a kind of rockfill body image recognition tracking and earth-rockfill dam surface displacement monitoring method, the rockfill individual of foundation frame and contrast frame is identified, the image of rockfill individual appears in the association of foundation frame and contrast frame, the displacement value of each rockfill individual is calculated, based on GNSS data and unmanned aerial vehicle oblique photogrammetry rockfill body image processing method carries out earth-rockfill dam surface displacement monitoring.The present application uses non-contact monitoring, can avoid interference with field construction or operation;Through unmanned aerial vehicle oblique photogrammetry and image recognition and tracking algorithm, can obtain the high spatial resolution of earth-rockfill dam target area based on multiple " mass points " Face domain displacement monitoring data, solve the bottleneck of the multiple monitoring requirements of " face domain ", " multiple mass points " and " reference coordinate system axial direction " that current prior art cannot simultaneously consider, provide the monitoring data that can reflect displacement spatial distribution law for construction quality control and numerical simulation.

Description

Technical Field

[0001] This invention relates to dam displacement monitoring technology, and more particularly to a method for image recognition and tracking of rockfill bodies and surface displacement monitoring of earth-rock dams. Background Technology

[0002] Surface displacement of earth-rock dams is a "true deformation characteristic" that directly and comprehensively reflects the working state of the dam body, playing a crucial role in construction quality control and numerical inversion. Monitoring the surface displacement of earth-rock dams allows for understanding the displacement distribution patterns, reflecting the quality of filling and compaction, and timely detection of construction defects and hidden dangers, which is beneficial for controlling deformation during construction and ensuring the accuracy of dam formation. Simultaneously, monitoring data can be used for numerical model parameter inversion and calibration, predicting future deformation trends and safety of earth-rock dams. Currently, there are various forms of surface displacement monitoring for earth-rock dams, such as traditional monitoring technologies like inclinometers, automated monitoring via Global Navigation Satellite System (GNSS), spaceborne synthetic aperture radar interferometry (InSAR), ground-based interferometric synthetic aperture radar (GB-InSAR), and UAV remote sensing monitoring, each with its own advantages and disadvantages.

[0003] Traditional monitoring technologies, such as inclinometer monitoring, can acquire displacement data inside the dam body or at designated points. They are highly targeted, easy to operate, and low in cost. However, they are contact-based single-point monitoring and cannot acquire area data. They are prone to missing local deformations between measuring points, and manual operation has a significant impact on data accuracy, such as human reading errors.

[0004] GNSS automated monitoring can be deployed on the surface of the dam. Its advantage lies in its ability to acquire three-dimensional displacement monitoring data under a long-term reference geographic coordinate system with high data accuracy (millimeter-level accuracy). Its disadvantage is the same as that of traditional monitoring technology: it is a single-point monitoring system. Due to the limitations of on-site construction, the actual deployment points are few and sparse, and the monitoring data cannot reflect the spatial distribution.

[0005] Space-based satellite InSAR monitoring is divided into Differential Synthetic Aperture Radar Interferometry (D-InSAR) monitoring and Long-Term Synthetic Aperture Radar Interferometry (LT-InSAR) monitoring. Their common advantages are long-term stability, non-contact monitoring, and no interference with on-site construction. However, both are susceptible to atmospheric errors, spatiotemporal decoherence, and other factors, and have limited monitoring frequencies. D-InSAR has the advantage of a large monitoring range, capable of acquiring area monitoring data. Its disadvantages include only acquiring small deformation data within the radar line of sight, low spatial resolution (meter-level), and monitoring accuracy for single deformations primarily at the centimeter level. LT-InSAR monitoring can acquire traditional three-dimensional displacement monitoring data for multiple mass points, but its sensitivity to deformation varies across the three directions. LT-InSAR monitoring is most sensitive to vertical deformation, with monitoring accuracy reaching the millimeter level; the sensitivity to east-west water deformation is second, while the sensitivity to north-south horizontal deformation is the lowest, with the worst data accuracy, primarily at the centimeter level. Long-term InSAR relies on multiple, long-term radar images to invert long-term dam deformation data by identifying and tracking stable radar scattering points. However, due to limitations in the on-site construction environment, the number of radar scattering points acquired is relatively small, resulting in sparse data and making it difficult to reflect the spatial distribution pattern of earth-rock dam displacement.

[0006] Ground-based radar GB-InSAR monitoring has the advantages of acquiring area monitoring data with high accuracy, reaching sub-millimeter level in the radar line-of-sight direction. However, it has disadvantages such as expensive equipment, complex installation, and easily obstructed field of view. The deformation it monitors is displacement in the radar line-of-sight direction, which is technically difficult to convert into traditional three-dimensional displacement, making it difficult to meet the data requirements for dam quality control during the construction period.

[0007] Unmanned aerial vehicle (UAV) remote sensing monitoring, such as UAV oblique photogrammetry and UAV lidar scanning, can acquire the coordinates of multiple points on the surface of an earth-rock dam, and extract displacement data by subtracting these coordinates. Its advantages lie in high spatial resolution, enabling full-area perception and acquiring a continuous, planar displacement field. However, its main drawback is that the displacement monitoring data is based on "geographical locations" rather than "mass points" within the rockfill. For example, in the point cloud data acquired by UAV oblique photogrammetry, each pixel represents a geographic plane coordinate position (X, Y). Typically, one coordinate is fixed to calculate the displacement deformation in the other coordinate direction. Assuming the X direction is east-west and the Y direction is north-south, for example, by fixing an east-west coordinate α and then extracting the north-south coordinates β1 and β2 from two consecutive measurements, the north-south displacement based on the "geographical location" is ΔY = β2 - β1. However, in reality, a real point mass P (such as the centroid or corner points of a rockfill structure) will displace in both the east-west and north-south directions. Assuming it moves from position (X1, Y1) to (X2, Y2), the north-south displacement based on the "point mass" is ΔY = Y2 - Y1. Clearly, β1 and β2 are generally not equal to Y1 and Y2. To achieve displacement monitoring based on "point masses," manual identification of feature points is usually required, which is labor-intensive and limits the number of feature points that can be identified in the short term, making it difficult to quickly monitor the displacement of large areas of individual rockfill structures.

[0008] In summary, the current status of displacement monitoring technology for earth-rock dams can be summarized as follows:

[0009] (1) Traditional monitoring technology and GNSS automated monitoring belong to point monitoring. Compared with other technologies that can achieve area monitoring, its advantage is that it has higher accuracy and can obtain the deformation displacement in the axial direction of the traditional reference coordinate system. However, its disadvantage is that it is easy to miss the local deformation between measuring points. Moreover, due to the limitations of the construction site, the number of points is small and sparse, and the monitoring data cannot reflect the spatial distribution.

[0010] (2) D-InSAR and ground-based radar GB-InSAR can achieve area monitoring, but the detected deformation is displacement in the radar line of sight, which is difficult to adapt to the data requirements of dam quality control during the construction period; due to the changes in the construction site, the images are prone to spatiotemporal decoherence, so long-term InSAR monitoring may face the problem of insufficient number of measurement points in dam scene monitoring.

[0011] (3) UAV remote sensing monitoring can acquire high spatial resolution point cloud coordinate data of the target area, but when calculating displacement monitoring data, it is necessary to manually mark feature points, which is costly and the number of marked feature points is limited. At the same time, the accuracy of UAV measurement data is generally at the centimeter level, which is lower than that of GNSS automated monitoring technology.

[0012] The monitoring and numerical simulation of earth-rock dam construction primarily target "mass points" (e.g., monitoring equipment is mainly deployed at key locations to monitor triaxial displacement; numerical simulation mainly simulates the deformation patterns of multiple mass points). The deformation values ​​reflect information such as filling quality and deformation characteristics. Therefore, in earth-rock dam construction monitoring and numerical simulation, not only is "area" monitoring data reflecting the dam's working state needed, but also axial displacement monitoring data based on "rockfill mass points" within a reference coordinate system (such as the engineering coordinate system, the Geodetic 2000 coordinate system, etc.). Summary of the Invention

[0013] The purpose of this invention is to propose a method for image recognition and tracking of rockfill bodies and surface displacement monitoring of earth-rock dams. It adopts GNSS data and UAV oblique photogrammetry image processing method for rockfill bodies to achieve high-precision monitoring of displacement deformation of earth-rock dams over a large area and at multiple points.

[0014] To achieve the above objectives, the technical solution of the present invention is: a method for image recognition and tracking of riprap, the method comprising:

[0015] S1. Take two frames of images of the earth-rock dam rockfill body acquired at different times. The earlier image of the earth-rock dam rockfill body is marked as the base frame, and the later image of the earth-rock dam rockfill body is marked as the comparison frame. Identify the individual rockfill bodies in the base frame and the comparison frame, and obtain the graphic features of each individual rockfill body in the base frame and the comparison frame.

[0016] S2. Using a target matching strategy, the same physical feature point is selected as the reference point in the base frame and the comparison frame. Then, based on the graphic features of each individual stone pile, the images of the individual stone pile appearing in the base frame and the comparison frame are associated.

[0017] S3. For the interrelated rock pile individuals in the base frame and the comparison frame, calculate the displacement value of each rock pile individual relative to the reference point.

[0018] A further method is that, in step S1, a unified pixel coordinate system is established in the base frame and the comparison frame, with the reference point as the origin o. ;

[0019] A deep learning model algorithm is used to identify the base frame and the comparison frame, and to obtain the bounding rectangle bounding box of each of the stone piles in the base frame and the comparison frame. The graphic features of the stone piles are the graphic features of the bounding rectangle bounding box of the stone piles.

[0020] Step S2 includes:

[0021] S21. Construct the position vector T of the individual rock pile based on the circumscribed rectangular bounding box.

[0022]

[0023] In the formula, Let k represent the position vector of the bounding rectangle of the rubble pile individual numbered k. The feature value representing the bounding rectangle of an individual rock pile;

[0024] Construct the N-dimensional feature vector F of individual rubble piles.

[0025]

[0026] In the formula, This represents the feature vector of the rockfill body numbered k. Representing the eigenvector N-dimensional elements, where N is greater than or equal to 6;

[0027] The intersection-union ratio (IOU) and the cosine similarity C of the eigenvectors were used to calculate the matching score of individual rocks.

[0028] S22. Establish the matching cost matrix between each individual rock pile in the base frame and each individual rock pile in the comparison frame.

[0029] S23. Match and associate each individual rock pile in the base frame and the comparison frame.

[0030] A further approach is to include deep learning model algorithms such as Mask R-CNN, YOLO, and SegNet.

[0031] A further method is to use position vectors for the graphical features. for:

[0032]

[0033] In the formula, This represents the coordinates of the top-left corner of the bounding rectangle of the rubble pile individual numbered k; This represents the coordinates of the lower left corner of the bounding rectangle of the rubble pile individual numbered k; This represents the coordinates of the lower right corner of the bounding rectangle of the rubble pile individual numbered k; This represents the coordinates of the upper right corner of the bounding rectangle of the rubble pile individual numbered k;

[0034] Let the matching score between the i-th boulders in the base frame image and the j-th boulders in the comparison frame image be . ,

[0035]

[0036] In the formula, This represents the intersection-union ratio (IoU) between the bounding rectangle of the i-th boulders in the base frame image and the bounding rectangle of the j-th boulders in the comparison frame image, with a value range of [0, 1]. The cosine similarity of the feature vectors of the i-th boulders in the base frame image and the j-th boulders in the comparison frame image is used. This represents the preset positional weight coefficients between the i-th individual rock pile in the base frame image and the j-th individual rock pile in the comparison frame image. The appearance feature weight coefficients of the i-th boulders in the base frame image and the j-th boulders in the comparison frame image satisfy the constraints. ;

[0037] In equation (4):

[0038]

[0039] In equation (5), This represents the area of ​​the bounding rectangle of the i-th individual rock pile in the base frame image; This represents the area of ​​the bounding rectangle of the j-th individual rock pile in the comparison frame image; The area of ​​the intersection of the bounding rectangle of the i-th boulders in the base frame image and the bounding rectangle of the j-th boulders in the comparison frame image is represented.

[0040] Based on the position vector of the outer rectangle of each individual rock pile in the cell coordinate system ξoη The characteristic vector F of each individual rock pile is calculated as follows:

[0041]

[0042] In equation (6), This represents the feature vector of the individual rockfill unit numbered k; They represent the eigenvectors respectively. The six-dimensional elements: and These represent the coordinate values ​​of the centroid of the individual rock pile numbered k on the ξ-axis and η-axis of the pixel coordinate system ξoη, respectively. The centroid of the individual rock pile is the centroid of the circumscribed rectangular bounding box. The width of the boulders numbered k is indicated by its cell length along the ξ-axis. The height of the individual rock pile numbered k is indicated by its cell length along the η-axis. The aspect ratio of the rubble pile individual with the number k is defined as follows: ; The pixel area of ​​the boulders numbered k is the area enclosed by the bounding rectangle.

[0043] After calculating the feature vectors of all rockfill bodies, the feature vectors are then... The elements are standardized, that is...

[0044]

[0045] In equation (7), Let k represent the standardized feature vector of the individual rockfill unit numbered k. Let P and Q represent the standardized vector elements, respectively. Then, the specific formula for calculating the standardized vector elements is:

[0046]

[0047] In equation (8), Represents the standardized coefficient of the feature element, and Value makes all Not less than 1.0 × 10 -6 ;

[0048] Cosine similarity of feature vectors between the i-th riprap individual in the base frame image and the j-th riprap individual in the comparison frame image The calculation formula is:

[0049]

[0050] In equation (9), The numerical range of is [0, 1].

[0051] In S22, a matching cost matrix is ​​established between each individual stone pile in the base frame and each individual stone pile in the comparison frame. Let P be the number of individual stone piles after instance segmentation in the base frame and Q be the number of individual stone piles after instance segmentation in the comparison frame. A P×Q dimensional matching cost matrix B is constructed. If P and Q are not equal, matrix B is expanded to form an R-order square matrix as the matching cost matrix M, where R = max(P, Q). The elements of the expanded rows or columns are all assigned a large number. The formula for calculating the matching cost matrix M of order R is as follows:

[0052]

[0053] In the formula, This represents the matching cost between the i-th boulders in the base frame image and the j-th boulders in the comparison frame image. The calculation formula is:

[0054]

[0055] In the formula, The score represents the matching score between the i-th boulders in the base frame image and the j-th boulders in the comparison frame image. The calculated value is obtained according to equation (4). Large numbers .

[0056] In S23, each individual rock pile in the base frame and the comparison frame is matched and associated. The matching relationship is represented by a 0-1 allocation matrix E, which is an R-order square matrix corresponding to the matching matrix M.

[0057]

[0058] in:

[0059]

[0060] A single rock pile in the base frame can be matched with at most one rock pile in the comparison frame, and the matching matrix E satisfies the condition.

[0061]

[0062] A stone pile individual in the comparison frame can be matched and associated with at most one stone pile individual in the base frame, and the matching matrix E satisfies the condition.

[0063]

[0064] Construct the overall matching cost function H as follows:

[0065]

[0066] The one-to-one global optimal pairing of individual rocks is transformed into finding the minimum value of the overall matching cost function H, min H.

[0067] After solving for the matching matrix E, assign a unified identifier ID to the individual rocks that have been matched and associated in the base frame and comparison frame images.

[0068] A further method is that, in step S3, the method for calculating the displacement value of each individual rockfill unit includes:

[0069] The same physical feature point is selected as the reference point in both the base frame and the comparison frame, and then a unified pixel coordinate system is established with the reference point as the origin. Based on the interconnected boulders obtained in step S2, the centroid pixel coordinates of the boulders in the base frame are... The centroid pixel coordinates of the individual boulders in the comparison frame are: Then, the relative pixel displacement of the rubble pile individual k relative to the reference point o is:

[0070]

[0071] In the formula, and These represent the individual k in the pile of stones on the pixel coordinate axis. and pixel coordinate axes The relative pixel displacement in the direction relative to the reference point o; K represents the number of boulders that match each other in the base frame and the comparison frame;

[0072] Let D be the ratio of the distance on the geographic plane to the number of pixels in the image of the rockfill body of the earth-rock dam. Then, in the pixel coordinate system... The relative geographical displacement of the center relative to the reference point o is:

[0073]

[0074] In the formula, and These represent the individual k in the pile of stones on the pixel coordinate axis. and pixel coordinate axes The relative geographical displacement in the direction of the reference point o;

[0075] Transform the pixel coordinate system using coordinate transformation methods. Relative geographical displacement and Transformed into relative geographic displacement in geographic coordinate system xOy and ,Right now

[0076]

[0077] A method for monitoring surface displacement of an earth-rock dam, comprising: employing the aforementioned image recognition and tracking method for rockfill bodies, and using GNSS data and UAV oblique photogrammetry image processing methods for monitoring surface displacement of the earth-rock dam; the method includes:

[0078] I. Image processing method for rockfill bodies of earth-rock dams obtained by UAV oblique photogrammetry, wherein the rockfill body image of the earth-rock dam is an image of multiple rockfill individuals within the target area of ​​the earth-rock dam acquired using UAV oblique photogrammetry technology, including at least two frames of rockfill body images of the earth-rock dam acquired at different times.

[0079] 2. Deploy GNSS measuring points within the target area of ​​the earth-rock dam and establish a geographic coordinate system xOy;

[0080] 3. Take two frames of the earth-rock dam rockfill body image acquired at different times. The earlier earth-rock dam rockfill body image is marked as the base frame, and the later earth-rock dam rockfill body image is marked as the comparison frame. Establish the pixel coordinate system ξoη.

[0081] IV. Using the above-mentioned image recognition and tracking method for riprap, global matching of riprap is performed to track the change in the centroid position of riprap and calculate the relative geographical displacement of individual ripraps.

[0082] V. Calculate the absolute geographic displacement of the rockfill body based on GNSS data;

[0083] VI. Compare the results with GNSS monitoring values ​​to analyze the accuracy of the displacement calculation results of the rockfill.

[0084] A further method involves acquiring images of the rockfill body of the earth-rock dam using UAV oblique photogrammetry. This involves using the same UAV, flight speed, flight path, camera, camera focal length and aperture, and sensor of the same specifications to acquire images of the target area of ​​the earth-rock dam during multiple UAV aerial photography sessions. The images of the rockfill body of the earth-rock dam acquired by UAV oblique photogrammetry have the same scale, and a uniform sampling resolution is set when performing aerial triangulation and modeling on the target area of ​​the earth-rock dam.

[0085] A further approach is to deploy no fewer than four GNSS measurement points within the target area of ​​the earth-rock dam, including no fewer than three non-collinear GNSS calibration points and no fewer than one GNSS verification point, and convert the displacement of the GNSS measurement points into geographic displacement in the geographic coordinate system xOy.

[0086] A further method is to define the offset displacement of the t-th GNSS calibration point as:

[0087]

[0088] In the formula, and These represent the offset displacements at the t-th GNSS calibration point along the x and y axes of geographic coordinates, respectively. and These represent the GNSS displacement measurements of the t-th calibration point along the x and y axes of geographic coordinates, respectively. and The relative geographic displacement of the earth-rock dam surface at the t-th GNSS calibration point in the x and y directions is represented by the UAV oblique photogrammetry image processing method for rockfill bodies. and These represent the additional displacements caused by systematic errors in the x and y directions of the geographic coordinate axis, respectively; and These represent the absolute geographic displacement of the reference point o in the x and y directions of the geographic coordinate axes, respectively, in the UAV oblique photogrammetry image processing method for rockfill.

[0089] A further approach is to fit an offset displacement function based on the offset displacement of the GNSS calibration point. This offset displacement function is fitted using a polynomial of at least first order, expressed as follows:

[0090]

[0091] In the formula, Represents the geographic coordinates of the base frame; and These represent the offset displacement functions in the x and y directions of the geographic coordinate axis, respectively; The coefficients of the fitting polynomial represent the offset displacement function along the x-axis of the geographic coordinate system. This represents the coefficients of the fitting polynomial for the offset displacement function along the y-axis of the geographic coordinate system. The number of fitting polynomials for the offset displacement function is determined by the number of GNSS measurement points deployed in the actual project, but at least a first-order polynomial must be used for fitting. The polynomial coefficients... and The calculations and results are obtained based on a data fitting algorithm.

[0092] A further method is to calculate the absolute geographic displacement of each individual rockfill unit in the xOy geographic coordinate system:

[0093]

[0094] In the formula, and These represent the absolute displacements of individual rockfill k along the x and y axes of the geographic coordinate system, respectively. and Let x and y represent the offset displacement functions in the directions of geographic coordinate axis x and y, respectively, and calculate them according to equation (21).

[0095] The beneficial effects of this invention are: it adopts non-contact monitoring, which can avoid interference with on-site construction or operation; through UAV oblique photogrammetry technology and image recognition and tracking algorithms, it can acquire high spatial resolution surface displacement monitoring data of the target area of ​​earth-rock dam based on multiple "mass points", which solves the bottleneck of the current technology that cannot simultaneously take into account multiple monitoring needs of "surface area", "multiple mass points" and "reference coordinate system axial direction", improves the displacement measurement accuracy of the existing UAV oblique photogrammetry technology, and provides monitoring data that can reflect the spatial distribution law of displacement for construction quality control and numerical simulation.

[0096] The present invention will now be described in detail with reference to the accompanying drawings and embodiments. Attached Figure Description

[0097] Figure 1 This is a schematic diagram of the principle of the invention, and a schematic diagram of the displacement calculation of the rockfill body;

[0098] Figure 2 This is a schematic diagram of the invention, which is a schematic diagram of images collected by a drone at the earth-rock dam site;

[0099] Figure 3 This is a schematic diagram of the principle of the present invention, and a schematic diagram of the method for acquiring images of the rockfill body of an earth-rock dam;

[0100] Figure 4 This is a schematic diagram of the invention, showing a comparison between the base frame and the comparison frame in a rockfill image;

[0101] Figure 5 This is a schematic diagram of the invention, showing the matching results of individual rock piles in the base frame and the comparison frame;

[0102] Figure 6 This is a schematic diagram of the invention, showing the displacement cloud map of individual rock piles;

[0103] Figure 7 This is a schematic diagram of the drone flight path for a dam under construction.

[0104] Figure 8 This is an orthophoto of a section of the slope of an earth-rock dam under construction, with GNSS measuring points marked on the image.

[0105] Figure 9 This is an orthophoto image of a section of the slope of an earth-rock dam under construction after uniform illumination processing. The pixel coordinate system and geographic coordinate system are marked in the image.

[0106] Figure 10 This is a partial schematic diagram of the rockfill image recognition and tracking method of the present invention for rockfill instance segmentation, target matching and centroid tracking;

[0107] Figure 11 This is a displacement cloud map of a section of the slope of an earth-rock dam under construction;

[0108] Figure 12 These are error analysis data charts of the calculated displacement results of the rockfill mass;

[0109] Figure 13 This is a chart comparing the current method for monitoring surface displacement of earth-rock dams with existing methods. Detailed Implementation

[0110] Example 1:

[0111] like Figures 2 to 6 This invention discloses a method for recognizing and tracking rockfill images. The rockfill image in this embodiment is an image of the rockfill on the surface of an earth-rock dam, referred to as an earth-rock dam rockfill image. The rockfill of an earth-rock dam is composed of multiple stones, each stone being referred to as a rockfill individual. Each frame of an earth-rock dam rockfill image includes multiple rockfill individuals.

[0112] This embodiment illustrates a simulated example of an image recognition and tracking method for rockfill formations.

[0113] S1. Take two frames of images of the earth-rock dam rockfill body acquired at different times. In this embodiment, multiple frames of earth-rock dam rockfill body images are obtained using UAV oblique photogrammetry technology. Two frames of earth-rock dam rockfill body images acquired at different times are selected. The earlier earth-rock dam rockfill body image is marked as the base frame, and the later earth-rock dam rockfill body image is marked as the comparison frame. The base frame and the comparison frame are images with the same resolution and scale. Figure 2 and Figure 3 This example demonstrates how to obtain images of the rockfill body of an earth-rock dam.

[0114] In this step, a unified pixel coordinate system is established in both the base frame and the comparison frame. In earth-rock dam engineering, observation houses have regular shapes, especially in orthophotos, where most observation house projections are rectangular, which is helpful for finding feature points. Therefore, in this embodiment, the centroid of the orthophoto projection area of ​​the observation house is used as the reference point o.

[0115] In this step, deep learning model algorithms are used to identify the base frame and the comparison frame, and to obtain the bounding box of the outer rectangle of each individual rock pile in the base frame and the comparison frame. The deep learning model algorithms include Mask R-CNN, YOLO, and SegNet.

[0116] like Figure 4 As shown, in this embodiment, the base frame identifies four individual rock piles A, B, C, and D, while the comparison frame identifies six individual rock piles a, b, c, d, e, and f. Based on the pixel coordinate system ξoη, and according to the bounding rectangle of each individual rock pile, the position vector T represents the graphic features of each individual rock pile in the base frame and the comparison frame. A rectangular bounding box can be described by various mathematical models. In this embodiment, a mathematical model is established based on the four vertices of the rectangular bounding box, using... This represents the circumscribed rectangle of the rockfill with the number k:

[0117]

[0118] In the formula, This represents the position vector of the bounding rectangle of the rubble pile individual numbered k; This represents the coordinates of the top-left corner of the bounding rectangle of the rubble pile individual numbered k; This represents the coordinates of the lower left corner of the bounding rectangle of the rubble pile individual numbered k; This represents the coordinates of the lower right corner of the bounding rectangle of the rubble pile individual numbered k; This represents the coordinates of the upper right corner of the bounding rectangle of the rockfill individual with the number k.

[0119] Based on The definition is that the position vector of the bounding rectangle of each individual rock pile in the cell coordinate system ξoη can be calculated through the instance segmentation algorithm. In this embodiment:

[0120] Base frame image:

[0121] Rockfill A:

[0122] Rockfill B:

[0123] Rockfill C:

[0124] Rockfill D:

[0125] Compare frame images:

[0126] Rockfill body a:

[0127] Rockfill body b:

[0128] Rockfill body c:

[0129] Rockfill d:

[0130] Rockfill body e:

[0131] Rockfill f:

[0132] S2. Using a target matching strategy, based on the feature vector F of each individual rock pile, associate the images of the individual rock pile appearing in the base frame and the comparison frame. This step includes:

[0133] S21. The matching score of individual rock piles is calculated using the Intersection over Union (IOU) and the cosine similarity C of the eigenvectors. Let the matching score of the i-th rock pile individual in the base frame image and the j-th rock pile individual in the comparison frame image be... ,

[0134]

[0135] In equation (24), This represents the intersection-union ratio (IoU) between the bounding rectangle of the i-th boulders in the base frame image and the bounding rectangle of the j-th boulders in the comparison frame image, with a value range of [0, 1]. The cosine similarity of the feature vectors of the i-th boulders in the base frame image and the j-th boulders in the comparison frame image is used. This represents the preset positional weight coefficients between the i-th individual rock pile in the base frame image and the j-th individual rock pile in the comparison frame image. The appearance feature weight coefficients of the i-th boulders in the base frame image and the j-th boulders in the comparison frame image satisfy the constraints. ;

[0136] In equation (24):

[0137]

[0138] In equation (25), This represents the area of ​​the bounding rectangle of the i-th individual rock pile in the base frame image; This represents the area of ​​the bounding rectangle of the j-th individual rock pile in the comparison frame image; Let represent the intersection area of ​​the bounding rectangle of the i-th pile of stones in the base frame image and the bounding rectangle of the j-th pile of stones in the comparison frame image. In this embodiment, for the four piles of stones A, B, C, and D in the base frame and the six piles of stones a, b, c, d, e, and f in the comparison frame, relative to pile of stones A in the base frame and pile of stones b in the comparison frame, we have:

[0139]

[0140] Based on this, the total intersection-union ratio is calculated. as follows:

[0141] , , , , , ,

[0142] , , , , , ,

[0143] , , , , , ,

[0144] , , , , , .

[0145] To calculate the cosine similarity C of the feature vectors, it is necessary to construct the feature vector F of each individual rock pile. The feature vector F can contain multiple feature values. Based on engineering experience, in this embodiment, a preferred approach is to select six feature values ​​for the feature vector F. Based on the position vector T of the bounding rectangle of each rock pile in the cell coordinate system ξoη, the feature vector of each individual rock pile can be calculated. :

[0146]

[0147] In the formula, This represents the feature vector of the rockfill body numbered k; and These represent the coordinate values ​​of the centroid of the individual rock pile numbered k on the ξ-axis and η-axis of the pixel coordinate system ξoη, respectively. The centroid of the individual rock pile is the centroid of the circumscribed rectangular bounding box. The width of the boulders numbered k is indicated by its cell length along the ξ-axis. The height of the individual rock pile numbered k is indicated by its cell length along the η-axis. The aspect ratio of the rubble pile individual with the number k is defined as follows: ; The pixel area of ​​the boulders with the number k is the area enclosed by the circumscribed rectangular bounding box.

[0148] To simplify and visually demonstrate the calculation process, the centroid, width, height, aspect ratio, and area of ​​the circumscribed rectangle of each rock pile are used to approximate the centroid, width, height, aspect ratio, and area of ​​the rock pile, and these are used as feature vector elements. Therefore, based on the position vector of each circumscribed rectangle in the pixel coordinate system ξoη, the feature vector of each individual rock pile can be calculated as follows:

[0149] Base frame image:

[0150] Rockfill A:

[0151] Rockfill B:

[0152] Rockfill C:

[0153] Rockfill D:

[0154] Compare frame images:

[0155] Rockfill body a:

[0156] Rockfill body b:

[0157] Rockfill body c:

[0158] Rockfill d:

[0159] Rockfill body e:

[0160] Rockfill f:

[0161] To eliminate the differences in numerical magnitude among the features of the feature vector F, and to ensure that different features participate in the calculation on the same scale using the same standard, the feature vector needs to be standardized.

[0162]

[0163] In the formula, The normalized feature vector represents the individual rockfill with the number k; P and Q represent the number of rockfills segmented from the base frame image and the comparison frame image, respectively. This represents the feature vector of the individual rockfill with ID k. The nth feature element; This represents the standardized feature vector of the k-th individual rockfill. The nth feature element; The standardization coefficients representing feature elements are used to ensure that all... Not less than 1.0 × 10 -6 This ensures numerical accuracy and allows for initialization. Then according to By gradually adjusting and increasing the minimum value, we can ultimately ensure that all... Not less than 1.0 × 10 -6 .

[0164] In this example, let According to equation (27), the standardized feature vectors of each individual rock pile in this embodiment can be calculated as follows:

[0165]

[0166]

[0167]

[0168]

[0169]

[0170]

[0171] Take individual A of the rubble mound as an example:

[0172]

[0173] Therefore, we have the base frame image:

[0174] Stone pile individual A:

[0175] Stone pile individual B:

[0176] Stone pile individual C:

[0177] Stone pile individual D:

[0178] Compare frame images:

[0179] Stone pile individual a:

[0180] Stone pile individual b:

[0181] Individual c of the rubble mound:

[0182] Stone pile individual d:

[0183] Stone pile individual e:

[0184] Stone pile individual f:

[0185] All feature elements are greater than 1.0 × 10 -6 ,so The value is appropriate.

[0186] Cosine similarity of feature vectors between the i-th riprap individual in the base frame image and the j-th riprap individual in the comparison frame image The calculation formula is:

[0187]

[0188] The numerical range of is [0, 1].

[0189] Based on the standardized feature vectors of each individual rock pile, the cosine similarity of the feature vectors of the rock pile individual with base frame image number i and the rock pile individual with comparison frame image number j can be calculated according to equation (28). ,For example,

[0190]

[0191] Based on this, the cosine similarity of each feature vector as follows:

[0192] , , , , ,

[0193] , , , , ,

[0194] , , , , ,

[0195] , , , , ,

[0196] Based on the intersection-union ratio and the cosine similarity of eigenvectors, this embodiment uses a preset coefficient for the positional weight of individual boulders. Appearance feature weighting coefficient According to equation (24), the matching score S between the rubble pile individual with base frame image number A and the rubble pile individual with comparison frame image number a can be calculated. Aa for:

[0197]

[0198] Based on this, the matching score for each individual boulders is... for

[0199] , , , , ,

[0200] , , , , ,

[0201] , , , , ,

[0202] , , , , ,

[0203] S22. Establish the matching cost matrix between each individual rock pile in the base frame and each individual rock pile in the comparison frame. Let P be the number of individual rock piles after instance segmentation in the base frame and Q be the number of individual rock piles after instance segmentation in the comparison frame. Construct a P×Q dimensional matching cost matrix B. If P and Q are not equal, expand matrix B to form an R-order square matrix as the matching cost matrix M, R = max(P, Q). Assign a large number to each element of the expanded row or column. The formula for calculating the matching cost matrix M of order R is as follows:

[0204]

[0205] In the formula, This represents the matching cost between the i-th boulders in the base frame image and the j-th boulders in the comparison frame image. The calculation formula is:

[0206]

[0207] In the formula, The score represents the matching score between the i-th boulders in the base frame image and the j-th boulders in the comparison frame image. The calculated value is obtained according to equation (24). Large numbers .

[0208] In this embodiment, the score is obtained based on the matching. According to equations (29) and (30), the matching cost matrix M can be calculated, R = max(P, Q) = max(4, 6) = 6. Rows need to be expanded, and larger numbers should be used. The calculation result is

[0209]

[0210] .

[0211] S23. Match and associate each individual rock pile in the base frame and the comparison frame. The matching relationship is represented by a 0-1 assignment matrix E, which is an R-order square matrix corresponding to the matching matrix M.

[0212]

[0213] in:

[0214]

[0215] A single rock pile in the base frame can be matched with at most one rock pile in the comparison frame, and the matching matrix E satisfies the condition.

[0216]

[0217] A stone pile individual in the comparison frame can be matched and associated with at most one stone pile individual in the base frame, and the matching matrix E satisfies the condition.

[0218]

[0219] Construct the overall matching cost function H as follows:

[0220]

[0221] The one-to-one global optimal pairing of individual rock piles is transformed into finding the minimum value of the overall matching cost function H, min H. In this example, the number of instance segments P = 4 in the baseline frame image is less than the number of instance segments Q = 6 in the comparison frame image, indicating that new rock pile individuals have appeared in the comparison frame image. Generally speaking, after the slope construction is completed, the relative positions of the rock pile individuals are basically fixed, and most of the rock pile individuals are stable in the long term, with only a very small number of stones rolling or being misaligned. Therefore, it can be assumed that all rock pile individuals in the baseline frame image can be matched in the comparison frame image (for the very few stones that may be missing in the comparison frame image due to rolling, misalignment, etc., it is easy to process them in the later displacement data through outlier filtering and other methods). Based on the above analysis, the constraint conditions (34) and (35) can be further restricted to

[0222]

[0223] and

[0224]

[0225] It should be noted that this embodiment uses the Hungarian algorithm to solve for the matching matrix. This minimizes the overall matching cost function H. To accommodate the Hungarian algorithm, this embodiment also ensures that the sum of the elements in the expanded rows equals 1, meaning that the expanded rows also have matching relationships. However, since the element values ​​in the expanded rows are all greater than the maximum true matching cost, the matching process prioritizes matching between the actual rock piles. The existence of the expanded rows does not affect the final result, and the result of the expanded rows is merely a virtual match used to coordinate the solution of mathematical relationships. Furthermore, since the rock pile individuals with the corresponding numbers in the expanded rows do not exist in the base frame image, it is easy to filter the matching results of the expanded rows through the program.

[0226] Through programming calculations, the minimum value of the overall matching cost function H is min H = 4.676, at which point the corresponding matching matrix E is...

[0227]

[0228] Ultimately, the matching result between the base frame and the comparison frame is as follows:

[0229] Individual A of the pile in the base frame image is matched with individual b of the pile in the comparison frame image;

[0230] Individual B of the base frame image is matched with individual c of the comparison frame image.

[0231] The rock pile individual C in the base frame image is matched with the rock pile individual d in the comparison frame image;

[0232] The rock pile individual D in the base frame image is matched with the rock pile individual f in the comparison frame image;

[0233] Matching the rock pile individuals a and e in the expanded row of the comparison frame image indicates that they are newly added rock pile individuals.

[0234] Based on the matching results, each successfully matched rockfill structure is assigned a unique ID identifier, namely:

[0235] Individual A of the boulder in the base frame image and individual b of the boulder in the comparison frame image: ID 1;

[0236] Individual B of the base frame image and individual c of the comparison frame image: ID 2;

[0237] Individual C of the base frame image and individual d of the comparison frame image: ID 3;

[0238] Individual D of the base frame image and individual f of the comparison frame image: ID 4.

[0239] like Figure 5 As shown.

[0240] S3. For the interconnected rock piles in the base frame and the comparison frame, calculate the displacement value for each rock pile individual. This step includes:

[0241] Establish a unified pixel coordinate system for the base frame and the comparison frames. Based on the interconnected boulders obtained in step S2, the pixel coordinates of the centroid of the boulders in the base frame are... The centroid coordinates of the individual boulders in the comparison frame are: Then the relative pixel displacement of the rubble pile individual k relative to the reference point o is:

[0242]

[0243] In the formula, and These represent the individual k in the pile of stones on the pixel coordinate axis. and pixel coordinate axes The relative pixel displacement in the direction relative to the reference point o; K represents the number of boulders that match each other in the base frame and the comparison frame;

[0244] In this embodiment, based on the four matching identifiers ID1, ID2, ID3, and ID4 obtained in step S2, the centroid position of the individual rock piles between the base frame and the comparison frame image can be tracked, and the position of the individual rock piles in the pixel coordinate system can be calculated using equation (36). The relative pixel displacement relative to the reference point o, i.e.:

[0245] The relative cell displacement of the rockfill ID 1 relative to the reference point o:

[0246]

[0247]

[0248] The relative cell displacement of the rockfill ID 2 relative to the reference point o:

[0249]

[0250]

[0251] The relative cell displacement of the rockfill ID 3 relative to the reference point o:

[0252]

[0253]

[0254] The relative cell displacement of the rockfill ID 4 relative to the reference point o:

[0255]

[0256]

[0257] Relative pixel displacement is the difference in image pixels between the base frame and the comparison frame, which can be further converted into relative geographic displacement. Let D be the ratio of the distance to the geographic plane to the number of pixels in the image of the rockfill dam, then the rockfill body in the pixel coordinate system... The relative geographical displacement of the center relative to the reference point o is

[0258]

[0259] In the formula, and These represent the individual k in the pile of stones on the pixel coordinate axis. and pixel coordinate axes The relative geographical displacement in the direction of the reference point o.

[0260] In this embodiment, the distance between the geographic plane and the pixel ratio of the rockfill image of the earth-rock dam is set to D, which is 0.1 mm / pixel. The relative geographic displacement of the rockfill in the pixel coordinate system ξoη relative to the reference point o can be calculated according to equation (37).

[0261] The relative geographic displacement of the rockfill ID 1 relative to the reference point o in the cell coordinate system ξoη:

[0262]

[0263]

[0264] The relative geographic displacement of the rockfill ID 2 relative to the reference point o in the cell coordinate system ξoη:

[0265]

[0266]

[0267] The relative geographic displacement of the rockfill ID 3 relative to the reference point o in the cell coordinate system ξoη:

[0268]

[0269]

[0270] The relative geographic displacement of the rockfill ID 4 relative to the reference point o in the cell coordinate system ξoη:

[0271]

[0272]

[0273] In this example, the cell coordinate axis ξ is in the same direction as the geographic coordinate axis x, and the cell coordinate axis η is in the same direction as the geographic coordinate axis y. Therefore, the geographic displacement of the rockfill relative to the reference point o in the geographic plane coordinate system xOy is:

[0274] The relative geographic displacement of the rockfill ID 1 relative to the reference point o in the geographic coordinate system xOy:

[0275]

[0276]

[0277] The relative geographic displacement of the rockfill ID 2 relative to the reference point o in the geographic coordinate system xOy:

[0278]

[0279]

[0280] The relative geographic displacement of the rockfill ID 3 relative to the reference point o in the geographic coordinate system xOy:

[0281]

[0282]

[0283] The relative geographic displacement of the rockfill ID 4 relative to the reference point o in the geographic coordinate system xOy:

[0284]

[0285] .

[0286] Example 2:

[0287] A method for monitoring surface displacement of an earth-rock dam. This method employs the rockfill image recognition and tracking method described in Example 1, based on GNSS data and UAV oblique photogrammetry image processing methods for monitoring the surface displacement of the earth-rock dam. This example uses an actual earth-rock dam under construction as an example to apply a method for monitoring the surface displacement of an earth-rock dam based on GNSS data and UAV oblique photogrammetry image processing methods. In this example:

[0288] I. Unmanned aerial vehicle (UAV) oblique photogrammetry technology was used to acquire images of the rockfill body of the earth-rock dam. During the image acquisition process, UAV flight paths were planned, and data collection was conducted periodically. Based on the earth-rock dam construction process, standardized flight and camera parameters were set, including the same UAV, flight speed, flight path, camera, camera focal length and aperture, and sensor specifications, to acquire identical images of the target area of ​​the earth-rock dam. (Refer to the above for reference.) Figure 7 Images of the rockfill body of an earth-rock dam acquired using UAV oblique photogrammetry have the same resolution and scale, and a uniform sampling resolution is set during subsequent aerial triangulation and modeling of the target area of ​​the earth-rock dam. An image of the rockfill body of an earth-rock dam contains multiple individual rockfill units within the target area of ​​the earth-rock dam. The images of the rockfill body of an earth-rock dam acquired using UAV oblique photogrammetry include multiple frames of images of the rockfill body acquired at different times.

[0289] II. To verify the surface displacement monitoring of the earth-rock dam, no fewer than four GNSS monitoring points are deployed within the target area of ​​the earth-rock dam, including no fewer than three non-collinear GNSS calibration points and no fewer than one GNSS verification point. The long-term GNSS monitoring equipment is a pole-mounted device installed on the rockfill body of the earth-rock dam slope. Generally, at the dam scale, the centroid of the nearest rockfill individual to the GNSS monitoring equipment is close enough to the GNSS monitoring point that the displacement deformation of the rockfill individual is considered to be consistent with that of the GNSS monitoring point. In this example, the centroids of both the GNSS monitoring point and a nearby rockfill individual are less than 1 meter. Therefore, the GNSS monitoring point and the centroid of a nearby rockfill individual can be approximated as common monitoring points for both monitoring technologies, providing high-precision GNSS displacement monitoring data for verification and validation of UAV rockfill displacement calculations. In this embodiment, GNSS technology, along with the described rockfill image recognition and tracking and earth-rock dam surface displacement monitoring methods, establish a total of eight common monitoring points, designated G1 to G8. A 3D model of the earth-rock dam was created using unmanned aerial vehicles (UAVs), establishing a geographic coordinate system xOy, with the positive x-axis pointing due east and the positive y-axis pointing due north. Eight BeiDou GNSS monitoring devices (G1 to G8) were deployed on the dam slope, with six monitoring points serving as calibration points and two as verification points. Figure 8 As shown, G1, G2, G4, G5, G6, and G8 serve as check points, while G3 and G7 serve as verification points. G7 is located within the control range of the check points, that is, within the area enclosed by G1, G2, G4, G5, G6, and G8, and G7 can verify the check accuracy. G3 is located outside the control range of the check points, that is, outside the area enclosed by G1, G2, G4, G5, G6, and G8, and G3 can verify the numerical accuracy of the extended range of the check points. The combination of G3 and G7 can serve as a practical test of the effectiveness of this method.

[0290] Third, identify and segment the riprap. For example... Figure 9 As shown, two images of the earth-rock dam at different times were obtained using UAV oblique photogrammetry. The earlier image of the earth-rock dam is marked as the base frame, and the later image of the earth-rock dam is marked as the comparison frame. In the base frame and the comparison frame, the centroid of the projected area of ​​the same observation room (OB) is selected as the reference point to establish a pixel coordinate system ξoη.

[0291] IV. Global matching of the rockfill. Using the rockfill image recognition and tracking method described in Example 1, the rockfill in the base frame image and the comparison frame image are identified and segmented. For example... Figure 9 The bounding boxes (OS) in the base frame and the comparison frame, after being processed by the rockfill image recognition and tracking method, produce image data as follows: Figure 10As shown. Then, the change in the centroid position of the boulders is tracked to calculate the relative geographic displacement of each boulder relative to the reference point o. In this example, the cell coordinate axis ξ is in the same direction as the geographic coordinate axis x, and the cell coordinate axis η is in the same direction as the geographic coordinate axis y. Therefore, the geographic displacement of the boulders relative to the reference point o in the cell coordinate system ξoη and the geographic plane coordinate system xOy is consistent. The boulders in the cell coordinate system obtained according to Example 1 are... Relative geographical displacement of the center relative to the reference point o and The relative geographic displacement of the individual rock piles relative to the reference point o in the geographic plane coordinate system xOy is calculated.

[0292] V. Calculating the absolute geographic displacement of the rockfill body based on GNSS data. Sampling calculations are performed on the actual earth-rock dam project of this embodiment. Taking G1 as an example, the centroid geographic coordinates of the individual rockfill unit in the base frame are calculated. Referring to the method in Embodiment 1, a deep learning model is used to output the position vector of the bounding rectangle of the individual rockfill unit G1 in the base frame in the cell coordinate system ξoη.

[0293]

[0294] The corner points of the bounding rectangle of the output rockfill individual are continuous floating-point coordinates without pixel rounding. In this example, the ratio of the distance to the geographic plane to the number of pixels in the rockfill image of the earth-rock dam is D = 0.05m / pixel. To ensure that the displacement calculation (in mm) is retained to two decimal places, the pixel coordinates of the corner points of the bounding rectangle of the rockfill individual are retained to three decimal places. Therefore, the pixel coordinates of the rockfill individual G1 in the base frame can be calculated. Centroid pixel coordinates for

[0295]

[0296]

[0297] Therefore, G1 is in the pixel coordinate system The coordinates of the centroid pixel in the reference frame are UAV oblique photogrammetry can obtain point cloud coordinate data. Through manual marking, the geographic coordinates of the reference point o in the base frame can be obtained. Therefore, the centroid geographic coordinates of the rubble pile individual G1 are:

[0298]

[0299]

[0300] Therefore, the centroid geographic coordinates of the individual rockfill sites at locations G1 to G8 in the base frame are:

[0301]

[0302]

[0303]

[0304]

[0305]

[0306]

[0307]

[0308]

[0309] In this embodiment, referring to the method of Embodiment 1, sampling calculations are performed on the actual earth-rock dam project of this embodiment. The centroid displacement of the individual rockfill units at locations G1 to G8 relative to the reference point o in the geographic plane coordinate system xOy is calculated as follows:

[0310]

[0311]

[0312]

[0313]

[0314]

[0315]

[0316]

[0317]

[0318] The GNSS displacement measurements of G1 to G8 in the geographic plane coordinate system xOy are as follows:

[0319]

[0320]

[0321]

[0322]

[0323]

[0324]

[0325]

[0326]

[0327] Calculate the offset displacement of the check points, i.e., the offset displacement of individual rockfill units ID1 to ID8:

[0328]

[0329]

[0330] Based on this, the following calculations can be made:

[0331]

[0332]

[0333]

[0334]

[0335]

[0336]

[0337] A first-order polynomial is used to fit the bias displacement field, i.e.:

[0338]

[0339] In this example, six GNSS checkpoints are used. Since the number of equations exceeds the number of polynomial coefficients, the least squares method can be used to calculate the polynomial coefficients. The calculation formula is as follows:

[0340]

[0341] In the formula, This indicates the number of checkpoints. Therefore, we have

[0342]

[0343] Substitute the data to obtain

[0344]

[0345] Therefore, for all individual rockfill units, the formula for calculating their absolute displacement is:

[0346]

[0347]

[0348] In the formula, k represents the number of the individual rock pile.

[0349] For example, calculating the absolute geographic displacement of individual rockfill sites from G1 to G8 in the geographic plane coordinate system xOy based on GNSS data and UAV image processing methods:

[0350] G1:

[0351]

[0352]

[0353] G2:

[0354]

[0355]

[0356] G3:

[0357]

[0358]

[0359] G4:

[0360]

[0361]

[0362] G5:

[0363]

[0364]

[0365] G6:

[0366]

[0367]

[0368] G7:

[0369]

[0370]

[0371] G8:

[0372]

[0373]

[0374] The displacement results for other individual rockfills are calculated according to the above steps, as follows: Figure 7 The displacement contour plot is shown below.

[0375] 6. Compare the calculated displacement of the rockfill with GNSS monitoring values ​​to analyze the accuracy of the results. The calculation results are shown in Table 1. The specific calculation process is as follows:

[0376] Calculate the residuals at the checkpoints:

[0377] G1:

[0378]

[0379]

[0380] G2:

[0381]

[0382]

[0383] G4:

[0384]

[0385]

[0386] G5:

[0387]

[0388]

[0389] G6:

[0390]

[0391]

[0392] G8:

[0393]

[0394]

[0395] The calculation results show that the residual level at the calibration point is within ±1cm.

[0396] Calculate the error at the verification point:

[0397] G3:

[0398]

[0399]

[0400] G7:

[0401]

[0402]

[0403] The calculation results show that the error level of the verification points is within ±1cm. This is a relatively good level of accuracy for estimating the displacement level of a large area of ​​an earth-rock dam and understanding the displacement and deformation patterns of such a large area. Figure 12 Table 1 records the error analysis data of the displacement calculation results of the rockfill body.

[0404] To better demonstrate the beneficial effects of this invention, the surface displacement monitoring method for earth-rock dams of this invention is compared with existing UAV oblique photogrammetry technology. UAV oblique photogrammetry technology can perform 3D modeling of the dam and acquire 3D point cloud data. After two or more phases of modeling, the spatial coordinates of the feature points can be subtracted by manually identifying feature points to obtain the displacement of the feature points. Using existing UAV oblique photogrammetry technology and manually identifying feature points, the displacements of verification points G3 and G7 were obtained, and the results are shown in Table 2. The specific calculation process is as follows:

[0405] The coordinates of feature points near G3 and G7 are manually labeled in the base frame. for

[0406]

[0407]

[0408] The coordinates of feature points near G3 and G7 are manually marked in the comparison frames. for

[0409]

[0410]

[0411] The x-direction displacement of feature points near G3 and G7 is manually identified. and displacement in the y direction for

[0412]

[0413]

[0414]

[0415]

[0416] Calculate the error values ​​in the x and y directions of existing UAV oblique photogrammetry technology, i.e. and ,for

[0417]

[0418]

[0419]

[0420]

[0421] Compared with existing UAV oblique photogrammetry technology, it can be found that:

[0422] (1) Existing UAV oblique photogrammetry technology is greatly affected by subjective factors because it uses manual identification of feature points. It is difficult to be absolutely accurate when identifying feature points, which can easily lead to large errors, and even the calculated displacement direction may be opposite to that of GNSS data. On the other hand, the wind speed, light and other environmental factors between two UAV flights can cause inconsistent systematic errors, which can also affect the calculation accuracy. In contrast, the surface displacement monitoring method of earth-rock dam based on GNSS data and UAV oblique photogrammetry image processing method uses registration reference points to reduce the difference in systematic errors between two flights, and performs adjustment through some GNSS measurement points. As a result, the displacement result calculated is more consistent with the deformation law of GNSS data and has higher accuracy.

[0423] (2) Existing UAV oblique photogrammetry technology, due to its reliance on manually identifying feature points, struggles to identify the centroids of all individual rockfill units over a large area. Therefore, it typically only selects a series of representative or key monitoring areas as the research object, making it difficult to cover the entire target area. In contrast, the earth-rock dam surface displacement monitoring method based on GNSS data and UAV oblique photogrammetry rockfill image processing utilizes deep learning algorithms to rapidly and extensively locate the centroids of individual rockfill units, acquiring a large amount of monitoring data and achieving wide coverage of the target area. Figure 11 The area displacement cloud map is shown. In this example, the surface displacement monitoring method for earth-rock dams, based on GNSS data and UAV oblique photogrammetry image processing, effectively tracked 17,350 individual rockfill structures, providing data support for understanding the deformation patterns of earth-rock dam slopes on a large scale. Figure 13 Table 2 shows the comparison results between the surface displacement monitoring method of the earth-rock dam of the present invention and existing methods.

[0424] In this invention, the use of UAV oblique photogrammetry technology can acquire point cloud data of the target area in a reference coordinate system (such as the engineering coordinate system, the geodetic 2000 coordinate system, etc.), which is naturally adapted to displacement monitoring in the axial direction of traditional coordinate axes. As long as the corresponding points in the image can be identified and tracked (referring to the position of the same object in different images), displacement monitoring data based on "mass points" can be calculated. Considering the characteristics of earth-rock dams, their slopes contain a large amount of rockfill. Within the scale of earth-rock dams, a large number of individual rockfills can be regarded as a large number of "mass points" on the surface of earth-rock dams. Except for abnormal working conditions, the relative positions of these rockfills will not change drastically after completion (such as large-area misalignment, rolling, etc.). Their displacement deformation is mainly caused by the deformation of the dam's rockfill during the construction period and the long-term deformation during the operation period. They are natural and good monitoring objects, and their monitoring data can reflect the displacement distribution pattern of the dam body over a long period of time. Therefore, this invention focuses on the identification and tracking of rockfill bodies in UAV images, aiming to achieve surface displacement monitoring based on multiple "mass points." Simultaneously, it leverages the millimeter-level high precision of GNSS to improve the numerical accuracy of UAV image calculations, thereby meeting the data requirements for dam construction and operation. The features of this invention are:

[0425] (1) Use UAV oblique photogrammetry to acquire at least two UAV images of the target area of ​​the earth-rock dam containing multiple rockfill entities;

[0426] (2) Each individual rock pile is identified and tracked using image recognition and tracking algorithms, and its position in each frame of the image is marked;

[0427] (3) The displacement of the rockfill is characterized by calculating the centroid displacement of the rockfill;

[0428] (4) Combining the high precision advantage of GNSS, adjustment is performed by establishing common measurement points for GNSS and UAV images to improve the numerical accuracy of UAV image displacement calculation results;

[0429] (5) Collect the displacement of the rockfill in batches to form monitoring data, such as outputting displacement cloud maps.

[0430] Based on the aforementioned features of this invention, its innovation lies in acquiring high spatial resolution surface displacement monitoring data based on multiple "mass points" within the target area of ​​an earth-rock dam through UAV oblique photogrammetry technology and image recognition and tracking algorithms. This solves the bottleneck of current technologies, which cannot simultaneously address multiple monitoring needs such as "surface area," "multiple mass points," and "reference coordinate system axial direction." Furthermore, relying on millimeter-level high-precision GNSS monitoring data as prior constraints improves the numerical accuracy of UAV image calculation results, providing monitoring data reflecting the spatial distribution of displacement for construction quality control and numerical simulation. Compared with existing UAV oblique photogrammetry technology, this invention not only improves displacement monitoring accuracy but also enables batch acquisition of measuring points. Moreover, this invention is a non-contact monitoring method, avoiding interference with on-site construction or operation.

Claims

1. A method for image recognition and tracking of riprap, characterized in that, The method includes: S1. Take two frames of images of the earth-rock dam rockfill body acquired at different times. The earlier image of the earth-rock dam rockfill body is marked as the base frame, and the later image of the earth-rock dam rockfill body is marked as the comparison frame. Identify the individual rockfill bodies in the base frame and the comparison frame, and obtain the graphic features of each individual rockfill body in the base frame and the comparison frame. S2. Using a target matching strategy, the same physical feature point is selected as the reference point in the base frame and the comparison frame. Then, based on the graphic features of each individual stone pile, the images of the individual stone pile appearing in the base frame and the comparison frame are associated. S3. For the interrelated rock pile individuals in the base frame and the comparison frame, calculate the displacement value of each rock pile individual relative to the reference point.

2. The image recognition and tracking method for riprap bodies according to claim 1, characterized in that, In step S1, a unified pixel coordinate system is established in the base frame and the comparison frame with the reference point as the origin o. ; A deep learning model algorithm is used to identify the base frame and the comparison frame, and to obtain the bounding rectangle bounding box of each of the stone piles in the base frame and the comparison frame. The graphic features of the stone piles are the graphic features of the bounding rectangle bounding box of the stone piles. Step S2 includes: S21. Construct the position vector T of the individual rock pile based on the circumscribed rectangular bounding box. In the formula, Let k represent the position vector of the bounding rectangle of the rubble pile individual numbered k. The feature value representing the bounding rectangle of an individual rock pile; Construct the N-dimensional feature vector F of individual rubble piles. In the formula, This represents the feature vector of the rockfill body numbered k; Representing the eigenvector N-dimensional elements, where N is greater than or equal to 6; The intersection-union ratio (IOU) and the cosine similarity C of the eigenvectors were used to calculate the matching score of individual rocks. S22. Establish the matching cost matrix between each individual rock pile in the base frame and each individual rock pile in the comparison frame. S23. Match and associate each individual rock pile in the base frame and the comparison frame.

3. The image recognition and tracking method for riprap bodies according to claim 2, characterized in that, The deep learning model algorithms include Mask R-CNN, YOLO, and SegNet.

4. The image recognition and tracking method for riprap bodies according to claim 2, characterized in that, The graphic features are represented by position vectors. for: In the formula, This represents the coordinates of the top-left corner of the bounding rectangle of the rubble pile individual numbered k; This represents the coordinates of the lower left corner of the bounding rectangle of the rubble pile individual numbered k; This represents the coordinates of the lower right corner of the bounding rectangle of the rubble pile individual numbered k; This represents the coordinates of the upper right corner of the bounding rectangle of the rubble pile individual numbered k; Let the matching score between the i-th boulders in the base frame image and the j-th boulders in the comparison frame image be . , In the formula, This represents the intersection-union ratio (IoU) between the bounding rectangle of the i-th boulders in the base frame image and the bounding rectangle of the j-th boulders in the comparison frame image, with a value range of [0, 1]. The cosine similarity of the feature vectors of the i-th boulders in the base frame image and the j-th boulders in the comparison frame image is used. This represents the preset positional weight coefficients between the i-th individual rock pile in the base frame image and the j-th individual rock pile in the comparison frame image. The appearance feature weight coefficients of the i-th boulders in the base frame image and the j-th boulders in the comparison frame image satisfy the constraints. ; In equation (4): In equation (5), This represents the area of ​​the bounding rectangle of the i-th individual rock pile in the base frame image; This represents the area of ​​the bounding rectangle of the j-th individual rock pile in the comparison frame image; The area of ​​the intersection of the bounding rectangle of the i-th boulders in the base frame image and the bounding rectangle of the j-th boulders in the comparison frame image is represented. Based on the position vector of the outer rectangle of each individual rock pile in the cell coordinate system ξoη The characteristic vector F of each individual rock pile is calculated as follows: In equation (6), This represents the feature vector of the individual rock pile with the number k. They represent the eigenvectors respectively. The six-dimensional elements: and These represent the coordinate values ​​of the centroid of the individual rock pile numbered k on the ξ-axis and η-axis in the pixel coordinate system ξoη, respectively. The centroid of the individual rock pile is the centroid of the circumscribed rectangular bounding box. The width of the boulders numbered k is indicated by its cell length along the ξ-axis. The height of the individual rock pile numbered k is indicated by its cell length along the η-axis. The aspect ratio of the rubble pile individual with the number k is defined as follows: ; The pixel area of ​​the boulders numbered k is the area enclosed by the bounding rectangle. After calculating the feature vectors of all rockfill bodies, the feature vectors are then... The elements are standardized, that is... In equation (7), Let k represent the standardized feature vector of the individual rockfill unit numbered k. Let P and Q represent the standardized vector elements, respectively. Then, the specific formula for calculating the standardized vector elements is: In equation (8), Represents the standardized coefficient of the feature element, and Value makes all Not less than 1.0 × 10 -6 ; Cosine similarity of feature vectors between the i-th riprap individual in the base frame image and the j-th riprap individual in the comparison frame image The calculation formula is: In equation (9), The numerical range of is [0, 1]. In S22, a matching cost matrix is ​​established between each individual stone pile in the base frame and each individual stone pile in the comparison frame. Let P be the number of individual stone piles after instance segmentation in the base frame and Q be the number of individual stone piles after instance segmentation in the comparison frame. A P×Q dimensional matching cost matrix B is constructed. If P and Q are not equal, matrix B is expanded to form an R-order square matrix as the matching cost matrix M, where R = max(P, Q). The elements of the expanded rows or columns are all assigned a large number. The formula for calculating the matching cost matrix M of order R is as follows: In the formula, This represents the matching cost between the i-th boulders in the base frame image and the j-th boulders in the comparison frame image. The calculation formula is: In the formula, The score represents the matching score between the i-th boulders in the base frame image and the j-th boulders in the comparison frame image. The calculated value is obtained according to equation (4). Large numbers ; In S23, each individual rock pile in the base frame and the comparison frame is matched and associated. The matching relationship is represented by a 0-1 allocation matrix E, which is an R-order square matrix corresponding to the matching matrix M. in: A single rock pile in the base frame can be matched with at most one rock pile in the comparison frame, and the matching matrix E satisfies the condition. A stone pile individual in the comparison frame can be matched and associated with at most one stone pile individual in the base frame, and the matching matrix E satisfies the condition. Construct the overall matching cost function H as follows: The one-to-one global optimal pairing of individual rocks is transformed into finding the minimum value of the overall matching cost function H, min H. After solving for the matching matrix E, assign a unified identifier ID to the individual rocks that have been matched and associated in the base frame and comparison frame images.

5. The image recognition and tracking method for riprap bodies according to claim 1, characterized in that, In step S3, the method for calculating the displacement value of each individual rockfill unit includes: The same physical feature point is selected as the reference point in both the base frame and the comparison frame, and then a unified pixel coordinate system is established with the reference point as the origin. Based on the interconnected boulders obtained in step S2, the centroid pixel coordinates of the boulders in the base frame are... The centroid pixel coordinates of the individual boulders in the comparison frame are: Then, the relative pixel displacement of the rubble pile individual k relative to the reference point o is: In the formula, and These represent the individual k in the pile of stones on the pixel coordinate axis. and pixel coordinate axes The relative pixel displacement in the direction relative to the reference point o; K represents the number of boulders that match each other in the base frame and the comparison frame; Let D be the ratio of the distance on the geographic plane to the number of pixels in the image of the rockfill body of the earth-rock dam. Then, in the pixel coordinate system... The relative geographical displacement of the center relative to the reference point o is: In the formula, and These represent the individual k in the pile of stones on the pixel coordinate axis. and pixel coordinate axes The relative geographical displacement in the direction of the reference point o; Transform the pixel coordinate system using coordinate transformation methods. Relative geographical displacement and Transformed into relative geographic displacement in geographic coordinate system xOy and ,Right now 。 6. A method for monitoring the surface displacement of an earth-rock dam, wherein the method employs the image recognition and tracking method for rockfill bodies as described in claims 1 to 5, and performs surface displacement monitoring of the earth-rock dam based on GNSS data and UAV oblique photogrammetry image processing methods for rockfill bodies, characterized in that... The method includes: I. Image processing method for rockfill bodies of earth-rock dams obtained by UAV oblique photogrammetry, wherein the rockfill body image of the earth-rock dam is an image of multiple rockfill individuals within the target area of ​​the earth-rock dam acquired using UAV oblique photogrammetry technology, including at least two frames of rockfill body images of the earth-rock dam acquired at different times.

2. Deploy GNSS measuring points within the target area of ​​the earth-rock dam and establish a geographic coordinate system xOy; 3. Take two frames of the earth-rock dam rockfill body image acquired at different times. The earlier earth-rock dam rockfill body image is marked as the base frame, and the later earth-rock dam rockfill body image is marked as the comparison frame. Establish the pixel coordinate system ξoη. IV. Using the image recognition and tracking method for riprap bodies as described in claims 1 to 5, the riprap bodies are globally matched, the centroid position of the riprap bodies is tracked, and the relative geographical displacement of individual riprap bodies is calculated. V. Calculate the absolute geographic displacement of the rockfill body based on GNSS data; VI. Compare the results with GNSS monitoring values ​​to analyze the accuracy of the displacement calculation results of the rockfill.

7. The method for monitoring surface displacement of an earth-rock dam according to claim 6, characterized in that, The images of the earth-rock dam rockfill obtained by UAV oblique photogrammetry are acquired through multiple UAV aerial photography sessions using the same UAV, flight speed, flight path, camera, camera focal length and aperture, and sensor of the same specifications. The images of the earth-rock dam rockfill obtained by UAV oblique photogrammetry have the same scale, and a uniform sampling resolution is set when performing aerial triangulation and modeling on the target area of ​​the earth-rock dam.

8. The method for monitoring surface displacement of an earth-rock dam according to claim 6, characterized in that, No fewer than four GNSS measuring points shall be set up within the target area of ​​the earth-rock dam, including no fewer than three non-collinear GNSS calibration points and no fewer than one GNSS verification point, and the displacement of the GNSS measuring points shall be converted into geographic displacement in the geographic coordinate system xOy.

9. The method for monitoring surface displacement of an earth-rock dam according to claim 8, characterized in that, The offset displacement of the t-th GNSS calibration point is: In the formula, and These represent the offset displacements at the t-th GNSS calibration point along the x and y axes of geographic coordinates, respectively. and These represent the GNSS displacement measurements of the t-th calibration point along the x and y axes of geographic coordinates, respectively. and The relative geographic displacement of the earth-rock dam surface at the t-th GNSS calibration point in the x and y directions is represented by the UAV oblique photogrammetry image processing method for rockfill bodies. and These represent the additional displacements caused by systematic errors in the x and y directions of the geographic coordinate axis, respectively; and These represent the absolute geographic displacement of the reference point o in the x and y directions of the geographic coordinate axes, respectively, in the UAV oblique photogrammetry image processing method for rockfill.

10. The method for monitoring surface displacement of an earth-rock dam according to claim 9, characterized in that, The offset displacement function is fitted based on the offset displacement of the GNSS calibration point. The offset displacement function is fitted using a polynomial of at least first order, and its expression is: In the formula, Represents the geographic coordinates of the base frame; and These represent the offset displacement functions in the x and y directions of the geographic coordinate axis, respectively; The coefficients of the fitting polynomial represent the offset displacement function along the x-axis of the geographic coordinate system. This represents the coefficients of the fitting polynomial for the offset displacement function along the y-axis of the geographic coordinate system. The number of fitting polynomials for the offset displacement function is determined by the number of GNSS measurement points deployed in the actual project, but at least a first-order polynomial must be used for fitting. The polynomial coefficients... and The calculations and results are obtained based on a data fitting algorithm.

11. The method for monitoring surface displacement of an earth-rock dam according to claim 10, characterized in that, Calculate the absolute geographic displacement of each individual rockfill unit in the xOy geographic coordinate system: In the formula, and These represent the absolute displacements of individual rockfill k along the x and y axes of the geographic coordinate system, respectively. and Let x and y represent the offset displacement functions in the directions of geographic coordinate axis x and y, respectively, and calculate them according to equation (21).

Images