Intelligent identification and analysis method of construction progress of long linear projects based on UAV aerial photography

Abstract

The present invention relates to a method for intelligently identifying the construction progress of a long linear project based on drone aerial photography, comprising six steps: drone inspection aerial photography, construction completion area target detection, construction node pixel coordinate positioning, construction node space coordinate conversion, construction progress identification, and construction progress analysis. The method of the present invention is scientifically and reasonably designed, and combined with drone inspection, it realizes rapid full coverage detection and intelligent identification of long linear construction areas of project progress, overcoming the drawbacks of low efficiency and untimely information feedback of manual inspection in traditional progress; by using image recognition technology, it realizes automatic processing, identification and analysis of drone inspection image data, avoiding the problems of low processing efficiency, susceptibility to interference by human factors, and delayed feedback of processing results when manually processing image data.

Description

Technical Field

[0001] The present invention belongs to the technical field of engineering construction, and relates to a construction progress intelligent recognition technology, in particular to an intelligent recognition and analysis method for the construction progress of a long linear engineering project based on drone aerial photography. Background Art

[0002] Project progress control is one of the important links in the whole process of project management. The construction progress directly affects the economic and social benefits of the project. Long linear water conservancy projects such as river lining and ecological revetments in small watershed management projects in mountainous areas are long, wide, and scattered. In addition, transportation in mountainous areas is inconvenient. Traditional manual inspections are time-consuming and laborious, and it is difficult to grasp the overall construction progress of the project in a timely manner, and it is difficult to remotely control the construction and make real-time management decisions.

[0003] At present, researchers have proposed a variety of automated construction progress detection methods, such as collecting time-lapse images from a fixed perspective of the construction site, obtaining mask images of the same perspective in the 4D BIM (3D BIM + planned progress) model, and finally comparing the two sets of images to obtain the on-site progress; in recent years, with the rapid development of 3D reconstruction technologies such as SFM (structure from motion), progress monitoring methods based on 3D point clouds have become a mainstream research trend, such as using SFM technology to generate on-site building point clouds, restoring the scale and reference plane of the on-site point clouds through control points arranged on-site, and using point cloud software to generate CAD models from the on-site point clouds, which are compared with the planned BIM model to determine the on-site construction progress; SFM-MVS (multi viewstero) technology can generate dense on-site building point clouds, and then realize the alignment of the on-site building point clouds and the planned BIM point clouds through manual registration methods, and perform geometric occupancy judgment on the two point clouds to obtain the on-site construction progress.

[0004] The above-mentioned public technical content mainly focuses on the construction progress identification of buildings and vertical infrastructure. However, since the construction area of ​​long linear projects such as river lining and ecological revetment is much larger than that of conventional construction projects, the existing research methods have certain limitations in the progress identification of long linear projects.

[0005] As a new type of aerial inspection platform that has attracted widespread attention in recent years, drones are flexible and have a wide field of vision, and can be used to overcome the limitations of traditional fields such as engineering inspections. However, the efficiency of manual processing and analysis of the massive image data collected by drones is low; if image recognition technology can be used to intelligently identify construction progress information from image data, it will effectively solve the problems of existing methods. Summary of the invention

[0006] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a method for intelligent identification and analysis of the construction progress of long linear projects based on drone aerial photography. The method can intelligently obtain on-site construction progress information from aerial images and quickly compare and analyze it with the planned construction progress, which is helpful to remotely grasp and judge the on-site construction progress and effectively solve the current problems of low efficiency and high cost of manual inspection and difficulty in real-time and rapid progress identification.

[0007] The present invention solves the technical problem by adopting the following technical solutions:

[0008] An intelligent identification method for the construction progress of long linear projects based on drone aerial photography includes six steps: drone inspection and aerial photography, target detection in the completed construction area, pixel coordinate positioning of construction nodes, spatial coordinate conversion of construction nodes, construction progress identification, and construction progress analysis. The specific method is described as follows:

[0009] Step 1: Drone inspection and aerial photography

[0010] Use drones to inspect long linear projects and collect aerial video images of long linear projects. The drones need to be equipped with sufficiently high-resolution aerial cameras and high-precision real-time differential positioning (RTK) systems, which can collect drone coordinates and images in real time, and can synchronously store and transmit the collected image data with the corresponding drone coordinate information.

[0011] Step 2: Target detection in the construction completion area

[0012] Target detection is performed on the construction completed area in the aerial images collected by the drone, which specifically includes the following steps:

[0013] 2-1. Dataset collection and annotation

[0014] Using drone aerial images, we pre-collect and construct sample datasets corresponding to different types of long linear projects, and annotate the target areas.

[0015] 2-2. Image data preprocessing

[0016] The image data collected by the drone is enhanced to improve the accuracy and robustness of the target detection model. The data enhancement methods include changes in the brightness, contrast, hue, saturation and Gaussian noise of the image data, as well as cropping, flipping, rotating, random scaling, etc.

[0017] 2-3. Constructing a construction completion area target detection network based on deep learning algorithm

[0018] 2-4. Train the target detection network to obtain the target detection model for the completed construction area

[0019] 2-5. Based on the trained target detection model, target detection in the completed construction area is realized, and the coordinate values ​​of the corner points of the target detection box are output

[0020] Step 3: Pixel coordinate positioning of construction nodes

[0021] The pixel coordinates of construction nodes in aerial images can be located in the following two steps:

[0022] 3-1. Determine the location of the construction node based on the image recognition results

[0023] Use the method in step 2 to identify the drone inspection aerial video, select the last frame to identify the target project, and take the center point of the target detection frame in the image as the construction progress node. For the last frame of the target project image A, use the pixel coordinates of the upper left corner of the detection frame output in step 2 (u min ,v min ) and the pixel coordinates of the upper right corner (u max ,v max ) calculates the center coordinates (x, y) of the target detection box, where u is u min and u max The average value of v min and v max The pixel coordinates (u, v) of the center point of the target detection frame are taken as the pixel coordinates of the construction node.

[0024] 3-2. Position matching and pixel coordinate positioning of construction nodes in different images

[0025] For other images containing the construction nodes in image A, the SIFT (scale-invariant feature transform) algorithm is used to match and locate the construction nodes. The specific steps are as follows:

[0026] 1. Feature point extraction and description generation

[0027] First, use the SIFT built-in function sift.detectAndCompute() to extract the feature points in the image B to be detected and the image A of the target project in the last frame and generate a SIFT descriptor. The descriptor is a 128-dimensional vector containing scale, position and direction information. Then, use the pt method of the KeyPoint class in OpenCV to obtain the pixel coordinates of each feature point in the image. The feature point sets extracted in image A and image B are (A 1 ,A2 ,…,A n )、(B 1 ,B 2 ,…,B m ); the description vectors corresponding to each feature point in image A and image B are A i =(a i1 ,a i2 ,…a i128 ), B i =(b i1 ,b i2 ,…b i128 ).

[0028] 2. Feature point matching

[0029] Match the feature points of the image A with the target project in the last frame with those of the image B to be detected. The matching process uses the Kd-tree algorithm to traverse the feature points in image A (A 1 ,A 2 ,…,A n ), for each feature point A in image A i (i=1,2,…,n), in the feature point set of image B (B 1 ,B 2 ,…,B m ) to find the i The vector is closest to point B S1 Point B, which is the second closest to the vector S2 , calculate the feature point A respectively i , B S1 With A i , B S2 The vector distance between them is used to determine whether the feature point in the reference image and the point in the image to be registered that is closest to them are a matching pair.

[0030] The calculation of vector distance is as follows:

[0031]

[0032] Where n is the vector dimension, n = 128; L(A i ,B j ) is the feature point A i The corresponding description vector and feature point B j The distance of the corresponding description vector.

[0033] Determine feature point A i With B S1 The formula for a pair of matching points is:

[0034]

[0035] If the distance ratio of the vector satisfies the formula, that is, it is not greater than the set threshold (generally 0.8), then point B in the image B to be detected is considered S1 With point A in image A i For matching pairs.

[0036] 3. Feature point pixel coordinate positioning

[0037] Based on the feature point matching step, the matching point pair set (M) of the last frame image A that identifies the target project and the image B to be detected is obtained. 1 ,M 2 ,…,M t ). Among them, M k (A i ,B j )(k=1,2,…,t) is a matching point pair, which indicates the kth pair of matching points in image A and image B, and i and j are the indexes of the feature points corresponding to the matching point pair in image A and image B respectively. The index of the feature point in each matching point pair realizes the matching relationship between each feature point in the two images. The queryIdx and trainIdx methods of the DMatch class in OpenCV can return the index of each feature point pair in the matching pair of the two images. Among them, queryIdx returns the index of each feature point in image A in the matching pair, and trainIdx returns the index of each feature point in image B in the matching pair.

[0038] Calculate the distance between the pixel coordinates of each feature point in image A and the center coordinates of the target detection frame, and take the feature point A closest to the center of the target detection frame. p The coordinates of the construction node are taken as the coordinates of the construction node. p The index p and the matching point pair set are used to establish the correspondence between the feature points in image A and image B, and obtain the index q of the corresponding feature point in image B. The pt method of the KeyPoint class is used to calculate the feature point B q The coordinates in image B are the pixel coordinates of the construction node in image B to be detected.

[0039] Step 4: Construction node space coordinate conversion

[0040] The pixel coordinates of the construction nodes are converted into the actual work area coordinates, which can be divided into the following two steps:

[0041] 4-1. UAV camera calibration

[0042] Before the drone inspection, the drone camera is calibrated using Zhang Zhengyou's plane calibration method to obtain the camera's intrinsic parameter matrix and distortion parameters.

[0043] The longitude and latitude coordinates of the camera are obtained by using the RTK positioning of the drone, and the longitude and latitude coordinates are converted into the actual work area coordinates of the camera in the local EastNorth Up (ENU) coordinate system through the coordinate conversion method (the origin and coordinate axis direction of the selected local ENU coordinate system are consistent with the actual work area coordinate system), thereby obtaining the camera's translation vector; the camera's attitude (yaw angle, pitch angle and roll angle) is obtained by using the drone's three-axis accelerometer and three-axis gyroscope, thereby obtaining the camera's rotation matrix. The translation vector and rotation matrix constitute the camera's external parameter matrix.

[0044] 4-2. Spatial positioning based on motion parallax

[0045] First, the pixel coordinates of the construction nodes are located using the methods of steps 2 and 3. Two images containing the construction nodes and shot at different times are selected, and the pixel coordinates of the construction nodes in the two images are calculated respectively (u t ,v t )、(u t+Δt ,v t+Δt ). Then, the motion parallax method in computer vision is used to restore the depth information z of the construction node C , solve the actual work area coordinates (x W ,y W ,z W ), see formula:

[0046]

[0047] Where M t 、M t+Δt They are the camera parameters corresponding to time t and time t+Δt, that is, the product of the intrinsic parameter matrix and the extrinsic parameter matrix.

[0048] Step 5: Construction Progress Identification

[0049] The actual work area coordinates of the current construction node obtained in step 4 and the information of the plan layout of the long linear project (such as CAD electronic drawing) are used to identify the actual construction progress of the project. First, the project axis in the plan layout is divided into small segments with equal distances, and the endpoints of the line segments with equal distances are obtained, and the coordinate data of the endpoints are extracted into Excel; then, the distance between the actual work area coordinates of the construction node and the coordinates of each endpoint in the design drawing is calculated and compared, and the coordinates of the endpoint closest to the spatial coordinates of the construction node are taken as the coordinate point representing the construction progress. Finally, the serial number of the equally divided endpoint corresponding to the point is multiplied by the length of the equally divided line segment, which is the constructed length, which can be used as an estimate of the actual construction progress.

[0050] Step 6: Construction Progress Analysis

[0051] Compare the actual construction progress obtained in step 5 with the planned progress to analyze whether the actual progress is ahead or behind schedule. The content of the construction progress analysis includes: the cumulative completion percentage of the project, the length and percentage of construction ahead / behind, the construction ahead / behind time, and the estimated completion time, and the drawings are presented.

[0052] The above construction progress analysis results can intuitively reflect the progress of construction on the project site, assist project management personnel to remotely grasp and judge the actual construction progress on site, and provide a basis for project decision-making.

[0053] The advantages and positive effects of the present invention are:

[0054] The method of the present invention is scientifically and reasonably designed. Combined with drone inspection, it can realize rapid full coverage detection and intelligent identification of long linear work areas in the project progress, overcoming the disadvantages of low efficiency and untimely information feedback of manual inspection in traditional progress. By using image recognition technology, it realizes automatic processing, recognition and analysis of drone inspection image data, avoiding the problems of low processing efficiency, susceptibility to interference from human factors, and delayed feedback of processing results when manually processing image data. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 This is a flow chart of the method for identifying and analyzing the construction progress of a long linear project of the present invention;

[0056] Figure 2 This is a flowchart of pixel coordinate positioning of construction nodes of the present invention;

[0057] Figure 3 This is a schematic diagram of target detection results in the construction completion area of ​​the present invention;

[0058] Figure 4 This is a schematic diagram of position matching of construction nodes in different images based on the SIFT algorithm of the present invention. DETAILED DESCRIPTION

[0059] The present invention is further described in detail below through specific examples. The following examples are only illustrative and not restrictive, and the protection scope of the present invention cannot be limited thereto.

[0060] The embodiment of the present invention provides a method for intelligently identifying the construction progress of a long linear project based on drone aerial photography, which includes five steps: drone inspection and aerial photography, target detection in the completed construction area, pixel coordinate positioning of construction nodes, spatial coordinate conversion of construction nodes, and identification and analysis of construction progress. Figure 1 This example uses the lining project in a long linear river regulation project to explain the specific implementation method in detail.

[0061] Step 1: Drone inspection and aerial photography

[0062] UAVs are used to inspect the long linear river regulation project and collect aerial video images of the long linear project. In order to adapt to the long distance and complex meteorological and geographical conditions of the long linear project, UAVs with long-distance airworthiness (wind resistance, antifreeze, waterproof, long flight distance) and real-time differential positioning (RTK) systems are selected, carrying aerial cameras with sufficient high resolution and storage capacity, to inspect along the long linear project and synchronously store the collected aerial videos and corresponding UAV data information.

[0063] Step 2: Target detection in the construction completion area

[0064] Target detection is performed on the construction completed area in the aerial images collected by the drone, which specifically includes the following steps:

[0065] 2-1. Dataset collection and annotation

[0066] Using drone aerial images, a sample data set of the river lining construction status is collected in advance. The sample data set is selected from the images collected by drones along different river channels. The selected aerial images cover different perspectives, engineering conditions and lighting conditions as much as possible to improve the diversity of image data samples.

[0067] Use labelImg annotation tool software to mark the lining area in the image with a rectangular frame, and generate data including image name, lining pixel coordinates and lining type.

[0068] 2-2. Image data preprocessing

[0069] Preprocess the image data collected by the drone. First, adjust the image to a uniform size (such as 640×640 pixels). Then, by changing the brightness, contrast, hue, saturation, and Gaussian noise of the drone-collected image, or cropping, flipping, rotating, and randomly scaling the drone-collected image, the image after data augmentation is obtained as the data set for the input object detection network.

[0070] 2-3. Constructing a construction completion area target detection network based on deep learning algorithm

[0071] The deep learning algorithm uses the Yolov5 algorithm. The target detection network includes an input layer, a sequentially connected backbone network, a multi-scale feature fusion network, and a prediction layer.

[0072] The input layer includes Mosaic data enhancement, adaptive anchor box calculation and adaptive image scaling modules. The Mosaic data enhancement module randomly uses 4 images in the data set and splices them by random scaling, random cropping and random arrangement, which can enrich the sample data set and enhance the robustness of the network; the adaptive anchor box calculation and adaptive image scaling modules reduce the amount of inference calculation and improve the speed of target detection.

[0073] The backbone network uses the CSPDarknet network with SiLU as the activation function, which is used to extract features from images. The multi-scale feature fusion network consists of FPN+PAN modules, which is used to fuse the features output by the backbone network to obtain feature maps with higher semantic information and improve the diversity and robustness of features. The loss function used in the prediction layer is GIoU. Loss , used to regress higher semantic features and obtain prediction results

[0074] SiLU activation function is shown in the formula:

[0075]

[0076] Loss function GIoU Loss See formula:

[0077]

[0078] Among them, A is the predicted box, B is the actual box, and C is the minimum bounding rectangle containing A and B.

[0079] 2-4. Train the target detection network to obtain the target detection model for the completed construction area

[0080] The preprocessed drone aerial images are used as sample data sets to train the target detection network for completed construction areas and obtain the target detection model for completed construction areas. If new categories of long linear projects are added later, the corresponding data sets of the new types will be supplemented and retrained.

[0081] 2-5. Based on the trained target detection model, target detection in the construction completed area is realized

[0082] The images taken by the drone inspection are input into the target detection model of the completed construction area to realize the identification of the completed construction area, that is, the lining area. Figure 3(a)-(c) are shown. The threshold of the intersection over union (IoU) (which measures the overlap between the predicted box and the actual box) is set to 0.5. When IoU>0.5, the target is considered to be detected. The types of predicted boxes are TP (True Positives), FP (False Positives), TN (True Negatives), and FN (False Negatives). The model performance evaluation indicators P (precision), R (recall), AP (mean precision), and mAP (mean average precision, that is, the average AP value of all classes).

[0083] The calculation of accuracy, recall rate and AP value is shown in formula -:

[0084]

[0085]

[0086]

[0087] In this example, 351 aerial images of rivers collected by drones along different rivers are selected as sample data sets. The original image size is uniformly adjusted to 640×640 pixels, and data expansion is performed. The training set and validation set are divided into a ratio of 9:1. According to the lining type, the target detection categories are divided into concrete, mortar masonry, and gabion mesh. The training is frozen for 50 epochs and unfrozen for 200 epochs. The mAP value of the trained target detection model is 90.11%. Figure 3 As shown in (d), the reliability of the target detection model is demonstrated.

[0088] Step 3: Pixel coordinate positioning of construction nodes

[0089] Locate the pixel coordinates of construction nodes in aerial images. The basic process is shown in Figure 2 . It can be divided into the following two steps:

[0090] 3-1. Determine the location of the construction node based on the image recognition results

[0091] Use drones to inspect the long linear river regulation project to be inspected to obtain aerial video, and use the construction completion area target detection model described in step 2 to identify the drone inspection aerial video frame by frame. Select the first or last frame to identify the image of the target project (lining), and take the center point of the target detection frame in the image as the construction progress node. Specifically, if there is no lining detection result output from the 1st frame to the (k-1)th frame, and the lining detection result output appears in the kth frame, then the center point of the target detection frame in the kth frame image is selected as the construction progress node; if there is a lining detection result output from the 1st frame to the (t-1)th frame, and there is no lining detection result output in the tth frame, then the center point of the target detection frame in the tth frame image is selected as the construction progress node.

[0092] For the last frame of image A that identifies the target project, use the detection frame corner coordinates output in step 2 to calculate the center coordinates of the target detection frame, and take the pixel coordinates of this point as the pixel coordinates of the construction node. The calculation of the pixel coordinates (u, v) of the center point of the target detection frame is shown in the formula:

[0093]

[0094] In the formula, (u min ,v min )、(u max ,v max ) correspond to the pixel coordinates of the upper left corner and lower right corner of the target detection box respectively.

[0095] 3-2. Position matching and pixel coordinate positioning of construction nodes in different images

[0096] For other images containing the construction nodes in image A, the SIFT (scale-invariant feature transform) algorithm is used to match and locate the construction nodes. The specific steps are as follows:

[0097] 1. Feature point extraction and description generation

[0098] Use the built-in SIFT function sift.detectAndCompute() in the opencv-contrib-python library to extract the feature points in the image B to be detected and the image A of the target project in the last frame and generate a SIFT descriptor. The descriptor is a 128-dimensional vector containing scale, position and direction information. Then, use the pt method of the KeyPoint class in OpenCV to obtain the pixel coordinates of each feature point in the image. The feature point sets extracted in image A and image B are (A 1 ,A 2 ,…,A n )、(B1 ,B 2 ,…,B m ), the feature point extraction results of the two images are Figure 4 (a); the description vectors corresponding to each feature point in image A and image B are A i =(a i1 ,a i2 ,…a i128 ), B i =(b i1 ,b i2 ,…b i128 ).

[0099] 2. Feature point matching

[0100] Match the feature points of image A and image B. During the matching process, the SIFT algorithm uses the Kd-tree algorithm to traverse the feature points in image A (A 1 ,A 2 ,…,A n ), for each feature point A in image A i (i=1,2,…,n), in the feature point set of image B (B 1 ,B 2 ,…,B m ) to find the i The vector is closest to point B S1 Point B, which is the second closest to the vector S2 ,like Figure 4 As shown in (b), the feature points A are calculated respectively. i , B S1 With A i , B S2 The vector distance between them is used to determine whether the feature point in the reference image and the point in the image to be registered that is closest to them are a matching pair.

[0101] The calculation of vector distance is as follows:

[0102]

[0103] Where n is the vector dimension, n = 128; L(A i ,B j ) is the feature point A i The corresponding description vector and feature point B j The distance of the corresponding description vector.

[0104] Determine feature point A i With B S1 The formula for a pair of matching points is:

[0105]

[0106] If the distance ratio of the vectors satisfies the formula, that is, is not greater than the set threshold (the threshold is 0.8 in this embodiment), then it is considered that point B in the image B to be detected S1 With point A in image A i is a matching pair. The matching results of feature point pairs in different images are as follows Figure 4 As shown in (c), all matching lines in the figure constitute a matching point pair set (M 1 ,M 2 ,…,M t ), each matching line corresponds to a matching point pair M k (A i ,B j ), A i , B j They are the two points connected by the matching line.

[0107] 3. Feature point pixel coordinate positioning

[0108] Based on the feature point matching step, the matching point pair set (M) of the last frame image A that identifies the target project and the image B to be detected is obtained. 1 ,M 2 ,…,M t ), where M k (A i ,B j )(k=1,2,…,t) is a matching point pair, which indicates the kth matching point pair in image A and image B. i and j are the indexes of the corresponding feature points of the matching point pair in image A and image B respectively. Use the queryIdx and trainIdx methods of the DMatch class in OpenCV to return the indexes of the feature point pairs matched in the two images in the matching pair. Among them, queryIdx returns the index of each feature point in image A in the matching pair, and trainIdx returns the index of each feature point in image B in the matching pair.

[0109] Then, calculate the distance between the pixel coordinates of each feature point in image A and the center coordinates (u, v) of the target detection frame, and take the feature point A closest to the center point of the target detection frame. p The coordinates (u A ,v A ) as the construction node coordinates. According to the feature point A i The index p and the matching point pair set are used to establish the correspondence between the feature points in image A and image B, and the index q of the corresponding feature point in image B is obtained. Then the pt method of the KeyPoint class is used to calculate the feature point B q The coordinates (u B ,v B ), which is the pixel coordinate of the construction node of the image B to be detected.

[0110] The mapping relationship of construction nodes in different images obtained based on the center coordinates (u, v) of the target detection frame is as follows: Figure 4 (d) as shown.

[0111] Step 4: Construction node space coordinate conversion

[0112] The pixel coordinates of the construction nodes are converted into the actual work area coordinates, which can be divided into the following two steps:

[0113] 4-1. UAV camera calibration

[0114] The mapping relationship between the pixel coordinate system and the world coordinate system is shown in the formula:

[0115]

[0116] In the formula, is the affine transformation matrix, is the perspective projection matrix, the product of the two is the intrinsic parameter matrix of the drone camera; is the external parameter matrix of the drone camera, where R is the rotation matrix and T is the translation vector; z C is the depth, that is, the value of the target point in the Z direction of the camera coordinate system. The specific calibration method of the intrinsic parameter matrix and the extrinsic parameter matrix is ​​as follows:

[0117] 1. Calibration of the internal parameter matrix

[0118] Before the drone inspection, the drone camera is calibrated using Zhang Zhengyou's plane calibration method to obtain the camera's intrinsic parameter matrix and distortion parameters. First, the Zhang Zhengyou calibration method chessboard is attached to a flat plate to make a calibration plate. The drone parameters are set to be consistent with the actual application parameters, and multiple calibration plate images are taken from different angles. For the captured calibration plate image, the function cv2.findChessboardCorners() in OpenCV is used to detect the chessboard corners and obtain the pixel coordinates of the corners. Define the plane where the chessboard is located as the XY plane of the world coordinate (Z=0), and the origin is located at the upper left corner of the chessboard. The world coordinate values ​​of each corner of the chessboard can be calculated in combination with the known chessboard size. Based on the above-obtained chessboard corner pixel coordinates and world coordinates, the cv2.calibrateCamera() function is used for calibration, and the intrinsic parameter matrix and distortion parameters of the drone camera are returned.

[0119] 2. External parameter matrix calibration

[0120] (1) Translation vector solution. The longitude and latitude coordinates of the camera are obtained by using the RTK positioning of the drone. The longitude and latitude coordinates are converted into the actual work area coordinates of the camera in the local EastNorth Up (ENU) coordinate system through the coordinate conversion method, thereby obtaining the camera's translation vector. The origin and coordinate axis directions of the selected local ENU coordinate system are consistent with the actual work area coordinate system.

[0121] The longitude and latitude coordinates of the drone camera must first be converted to the earth-centered, earth-fixed (ECEF) coordinate system, and then converted to the ENU coordinate system. The position coordinates of the drone camera in the ENU coordinate system are the camera's translation vector. The conversion of the longitude and latitude coordinates (lat, lon, alt) in the longitude and latitude coordinate system to the point (x, y, z) in the ECEF coordinate system is shown in the formula:

[0122]

[0123] Where, e is the eccentricity of the reference ellipsoid, e 2 =(a 2 -b 2 ) / a 2 , N is the radius of curvature of the reference ellipsoid, a and b are the major and minor axes of the reference ellipsoid respectively. For the WGS84 coordinate system, the ellipsoid parameters are a=6378137 and b=6356752.3142.

[0124] The latitude and longitude coordinates of the origin of the ENU coordinate system are (lon 0 ,lat 0 ,alt 0 ), the ECEF coordinates are (x 0 ,y 0 ,z 0 ), then the transformation of any point (x, y, z) in the ECEF coordinate system to the ENU coordinate system is as follows:

[0125]

[0126] The coordinates of the drone camera in the local ENU coordinate system (e, n, u) are calculated by formula and the camera translation vector T = [enu] T .

[0127] (2) Solving the rotation matrix. The three-axis accelerometer and three-axis gyroscope of the drone are used to obtain the camera attitude (yaw angle ψ, pitch angle θ and roll angle φ), thereby obtaining the rotation matrix of the camera. The calculation of the rotation matrix from the reference coordinate system (ENU coordinate system) to the body coordinate system is shown in the formula:

[0128]

[0129] The camera's external parameter matrix can be obtained through the translation vector T and rotation matrix R of the drone camera.

[0130] 4-2. Spatial positioning based on motion parallax

[0131] Using motion parallax method in computer vision to solve the depth value z C First, according to the methods of step 2 and step 3, the construction nodes are identified and their pixel coordinates are located. Two images containing construction nodes and shot at different times are selected, and the pixel coordinates of the construction nodes in the two images are calculated respectively (u t ,v t )、(u t+Δt ,v t+Δt Then, the depth information of the construction node is restored by using the triangular relationship to solve the actual work area coordinates (x W ,y W ,z W ), see formula:

[0132]

[0133] Where M t 、M t+Δt are the camera parameters corresponding to time t and time t+Δt, that is, the product of the internal parameter matrix and the external parameter matrix,

[0134] Step 5: Construction Progress Identification

[0135] The actual construction progress of the project is identified by using the construction progress coordinates obtained in step 4 and the plan layout diagram of the long linear project (the present embodiment uses the CAD plan layout electronic diagram).

[0136] First, in AutoCAD software, the axis of the long linear project in the plan layout is divided into small segments at equal distances (equally divided at 0.1m intervals), and the endpoints of the equally divided line segments are obtained. The position X and position Y coordinate data of the equally divided points are imported into Excel using the data extraction command. Then, the distance between the world coordinates of the construction node obtained in step 4 and the coordinates of all endpoints in the design drawing is calculated, and the obtained distances are compared, and the coordinates of the endpoint closest to the spatial coordinates of the construction node are taken as the coordinate points representing the construction progress.

[0137] Multiply the endpoint number of the equally divided point corresponding to the point by the length of the equally divided line segment (0.1m) to get the constructed length, which can be used as an estimate of the actual construction progress.

[0138] Step 6: Construction Progress Analysis

[0139] Compare the actual construction progress obtained in step 5 with the planned progress to analyze whether the actual progress is ahead of schedule or behind schedule. The construction progress analysis results are updated every time the drone inspects. The content of the construction progress analysis includes the cumulative completion percentage of the project, the length and percentage of construction ahead / behind schedule, the construction ahead / behind schedule, and the estimated completion time, and is represented by a drawing.

[0140] The specific calculation and analysis method of construction progress is as follows:

[0141] 1. Construction progress percentage (completion percentage) is calculated according to the formula:

[0142]

[0143] Where, L is the actual construction length identified, L 总 It is the total length of design construction.

[0144] 2. The length and percentage of construction advance / delay shall be calculated according to the following formula:

[0145] Construction lead / lag length = LL 0 (2.15)

[0146]

[0147] Where, L 0 is the planned construction length corresponding to the drone inspection image acquisition time T. The inspection image acquisition time is the time corresponding to the first frame or the last frame of the image of the target project identified in step 3. If the actual construction length is greater than the planned construction length, that is, L>L 0 , the current construction progress is ahead of schedule; otherwise it is behind schedule.

[0148] In the planned construction information, the relationship between construction length and time is discrete, and the interpolation method is used to approximate it, as shown in the formula:

[0149]

[0150] Where, T 1 , L 1 are the previous planned construction time and length corresponding to the drone inspection image acquisition time T in the planned construction information; T 2 , L 2 They are respectively the next planned construction time and length corresponding to the drone inspection image acquisition time T in the planned construction information.

[0151] 3. Construction lead / lag time is calculated according to the formula:

[0152] Construction lead / lag time = T 0 -T (2.18)

[0153] Where T is the UAV inspection image acquisition time, T 0 is the planned construction time corresponding to the actual construction length L. If the drone inspection image acquisition time is earlier than the planned construction time, that is, T<T 0 , the current construction progress is ahead of schedule; otherwise it is behind schedule.

[0154] As mentioned above, T 0 The interpolation method is used for calculation, as shown in the formula:

[0155]

[0156] Where, L 1 , T 1 are the previous planned construction length and time corresponding to the actual construction length L in the planned construction information; L 2 , T 2 They are respectively the next planned construction length and time corresponding to the actual construction length L in the planned construction information.

[0157] 4. Estimated completion time, calculated according to the formula:

[0158]

[0159] Where V is the construction speed, V = L / t, which is the ratio of the identified actual construction length L to the construction time t.

[0160] The above construction progress analysis results can intuitively reflect the construction progress of the project site, assist project management personnel to remotely grasp and judge the actual construction progress on site, and provide a basis for project decision-making.

[0161] Although the embodiments of the present invention are disclosed for illustrative purposes, those skilled in the art will appreciate that various substitutions, changes and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the scope of the present invention is not limited to the contents disclosed in the embodiments.

Claims

1. A method for intelligently identifying the construction progress of a long linear project based on drone aerial photography, comprising the following steps: Step 1: Drone inspection and aerial photography Use drones to inspect long linear projects and collect aerial video images of long linear projects. The drones are equipped with high-resolution aerial cameras and high-precision real-time differential positioning systems to collect drone coordinates and take images in real time, and synchronize the collected image data with the corresponding drone coordinate information for storage and transmission; Step 2: Target detection in the construction completion area Performing target detection on the completed construction area in the aerial image collected by the drone in step 1 specifically includes the following steps: (1) Dataset collection and annotation: Using drone aerial images, we pre-collect and construct sample datasets corresponding to different types of long linear projects, and annotate the target areas; (2) Image data preprocessing: image data collected by the drone are enhanced to improve the accuracy and robustness of the target detection model. Data enhancement methods include changes in image data brightness, contrast, hue, saturation, and Gaussian noise, as well as cropping, flipping, rotating, and randomly scaling images. (3) Construct a construction completion area target detection network based on deep learning algorithm; (4) Train the target detection network to obtain the target detection model of the construction completion area; (5) Based on the trained target detection model, target detection is realized in the construction completed area, and the coordinate values ​​of the corner points of the target detection box are output; Step 3: Pixel coordinate positioning of construction nodes The pixel coordinates of construction nodes in aerial images can be located in the following two steps: (1) Determine the location of the construction node based on the image recognition results Use the method in step 2 to identify the drone inspection aerial video, select the last frame to identify the target project image, take the center point of the target detection frame in the image as the construction progress node, and for the last frame to identify the target project image A, use the pixel coordinates of the upper left corner of the detection frame output in step 2 (u min ,v min ) and the pixel coordinates of the lower right corner (u max ,v max ) calculates the center coordinates (x, y) of the target detection box, where u is u min and u max The average value of v min and v max The pixel coordinates (u, v) of the center point of the target detection frame are taken as the pixel coordinates of the construction node; (2) Position matching and pixel coordinate positioning of construction nodes in different images For other images containing the construction nodes in image A, the SIFT scale-invariant feature transformation algorithm is used to match and locate the construction nodes; Step 4: Construction node space coordinate conversion The pixel coordinates of the construction nodes are converted into the actual work area coordinates, which can be divided into the following two steps: (1) UAV camera calibration Before the drone inspection, the drone camera is calibrated using Zhang Zhengyou's plane calibration method to obtain the camera's internal parameter matrix and distortion parameters. The drone's RTK positioning is used to obtain the camera's longitude and latitude coordinates. The longitude and latitude coordinates are converted into the camera's actual work area coordinates in the local Northeast Sky East North Up (ENU) coordinate system through the coordinate conversion method. The origin and coordinate axis directions of the selected local ENU coordinate system are consistent with the actual work area coordinate system, thereby obtaining the camera's translation vector. The drone's three-axis accelerometer and three-axis gyroscope are used to obtain the camera's attitude yaw angle, pitch angle, and roll angle, thereby obtaining the camera's rotation matrix. The translation vector and rotation matrix constitute the camera's external parameter matrix. (2) Spatial positioning based on motion parallax First, the pixel coordinates of the construction nodes are located using the methods of steps 2 and 3. Two images containing the construction nodes and shot at different times are selected, and the pixel coordinates of the construction nodes in the two images are calculated respectively (u t ,v t )、(u t+Δt ,v t+Δt ), then the motion parallax method in computer vision is used to restore the depth information z of the construction node C , solve the actual work area coordinates (x W ,y W ,z W ), see the following formula: Where M t 、M t+Δt are the camera parameters corresponding to time t and time t+Δt, i.e., the product of the intrinsic parameter matrix and the extrinsic parameter matrix; Step 5: Construction Progress Identification The actual work area coordinates of the current construction node obtained in step 4 and the information of the plane layout CAD drawing of the long linear project are used to identify the actual construction progress of the project. First, the project axis in the plane layout drawing is divided into small segments with equal distances to obtain the endpoints of the line segments with equal distances, and the coordinate data of the endpoints are extracted into Excel; then, the distance between the actual work area coordinates of the construction node and the coordinates of each endpoint in the design drawing is calculated and compared, and the coordinates of the endpoint closest to the spatial coordinates of the construction node are taken as the coordinate point representing the construction progress; finally, the sequence number of the equally divided endpoint corresponding to the point is multiplied by the length of the equally divided line segment, which is the constructed length, which can be used as an estimate of the actual construction progress; Step 6: Construction Progress Analysis Compare the actual construction progress obtained in step five with the planned progress, analyze whether the actual progress is ahead of schedule or behind schedule, and draw a diagram to represent it.

2. According to claim 1, a method for intelligently identifying the construction progress of a long linear project based on drone aerial photography is characterized by: The specific method and steps for matching and locating construction nodes using the SIFT scale-invariant feature transformation algorithm are as follows: (1) Feature point extraction and description generation First, use the SIFT built-in function sift.detectAndCompute() to extract the feature points in the image B to be detected and the image A of the target project in the last frame and generate a SIFT descriptor. The descriptor is a 128-dimensional vector containing scale, position and direction information. Then, use the pt method of the KeyPoint class in OpenCV to obtain the pixel coordinates of each feature point in the image. The feature point sets extracted in image A and image B are (A1, A2, …, A n )、(B1,B2,…,B m ), the description vectors corresponding to each feature point in image A and image B are A i =(a i1 ,a i2 ,…a i128 ), B i =(b i1 ,b i2 ,…b i128 ); (2) Feature point matching The image A with the target project identified in the last frame is matched with the feature points of the image B to be detected. The matching process uses the Kd-tree algorithm to traverse the feature points in image A (A1, A2, ..., A n ), for each feature point A in image A i (i=1,2,…,n), in the feature point set of image B (B1,B2,…,B m ) to find the i The vector is closest to point B S1 Point B, which is the second closest to the vector S2 , calculate the feature point A respectively i , B S1 With A i , B S2 The vector distance between them is used to determine whether the feature point in the reference image and the point in the image to be registered that is closest to each other are a matching pair. The calculation of vector distance is as follows: Where n is the vector dimension, n = 128; L(A i ,B j ) is the feature point A i The corresponding description vector and feature point B j The distance of the corresponding description vector, Determine feature point A i With B S1 The formula for a pair of matching points is: If the distance ratio of the vector satisfies the above formula, that is, it is not greater than the set threshold, then point B in the image B to be detected is considered S1 With point A in image A i For matching pairs; (3) Feature point pixel coordinate positioning Based on the feature point matching step, the matching point pair set (M1, M2, ..., M2) of the last frame image A and the image to be detected B is obtained. t ), Among them, M k (A i ,B j )(k=1,2,…,t) is a matching point pair, which represents the kth pair of matching points in image A and image B. i and j are the indexes of the feature points corresponding to the matching point pair in image A and image B respectively. The index of the feature point in each matching point pair realizes the matching relationship between the feature points in the two images. The queryIdx and trainIdx methods of the DMatch class in OpenCV can return the index of the feature point pairs matched in the two images in the matching pair. Among them, queryIdx returns the index of each feature point in image A in the matching pair, and trainIdx returns the index of each feature point in image B in the matching pair. Calculate the distance between the pixel coordinates of each feature point in image A and the center coordinates of the target detection frame, and take the feature point A closest to the center of the target detection frame. p The coordinates of the construction node are taken as the coordinates of the construction node. p The index p and matching point pair set are used to establish the correspondence between the feature points in image A and image B, and the index q of the corresponding feature point in image B is obtained. The pt method of the KeyPoint class is used to calculate the feature point B. q The coordinates in image B are the pixel coordinates of the construction node in image B to be detected.

3. The method for intelligently identifying the construction progress of a long linear project based on drone aerial photography according to claim 1 is characterized by: The contents of construction progress analysis include: the cumulative completion percentage of the project, the length and percentage of construction advance / delay, the construction advance / delay time and the estimated completion time.

Images