A method for inspection and distance measurement based on registration of point cloud and image collected by a UAV

Abstract

The application relates to a kind of inspection and ranging methods based on unmanned aerial vehicle acquisition point cloud and image registration, and relates to the technical field of laser radar point cloud data application.The application realizes the registration of point cloud and shooting image obtained by unmanned aerial vehicle through the proposed camera internal parameter acquisition and pose solution;The registration effect is visualized through the proposed registration visualization module, and the identification, positioning and protection zone and hidden danger actual distance measurement of hidden danger in the monitoring and shooting image are realized through the proposed hidden danger identification and positioning, registration transformation and distance calculation.

Description

TECHNICAL FIELD

[0001] The application discloses a method for inspection and distance measurement based on registration of point cloud and image collected by a UAV, and relates to the technical field of laser radar point cloud data application. BACKGROUND

[0002] To improve the efficiency of target scene inspection and reduce the cost of inspection, the UAV has broad development prospects in the field of inspection. However, due to the complexity of UAV pose solving, current integrated schemes for hazard identification and distance measurement are still based on fixed monitoring equipment. The present patent solves the key technology of UAV and camera pose solving, and applies it to the registration of three-dimensional point cloud and image to establish a mapping relationship from three-dimensional to two-dimensional. By using this relationship and combining with a general target detection algorithm, the positioning of any scene hazard and the distance measurement from the protection zone are realized.

[0003] In the prior art, the following method for distance measurement using a UAV is disclosed:

[0004] Chinese patent document CN115170745A discloses a UAV distance measurement method based on stereovision. However, this document does not involve the use of pose solving method to directly realize registration. Instead, it indirectly establishes a mapping model between point cloud and image using block matching. The technical solution involved in this document has poor interpretability and requires a large amount of computing resources, which is not suitable for large and complex application scenarios.

[0005] In addition, Chinese patent document CN114140659A discloses a social distance monitoring method based on human body detection from the perspective of a UAV. This scheme uses a pose solving method to realize point cloud and image registration. Specifically, it requires the UAV to carry a gimbal that only changes the pitch angle. This implementation has low difficulty. However, this scheme cannot achieve satisfactory registration results for complex application scenarios due to the lack of transformation of roll angle and yaw angle.

[0006] In summary, the development of current UAV inspection is restricted by the complexity of aerial pose solving. Existing solving techniques often limit the flight freedom of the UAV or gimbal in a certain direction to solve this problem. However, such restrictions are unreasonable and fatal in some complex and confidential scenarios. The present application solves the problem of UAV pose solving by recording the initial state of the UAV before takeoff and using three-dimensional rotation technology to analyze the conversion relationship between point cloud and UAV inertial navigation. Furthermore, the present application combines recognition technology to achieve further analysis of the inspection scene. SUMMARY

[0007] To address the shortcomings of existing technologies, this invention discloses an inspection method based on point cloud data collected by UAVs and image registration.

[0008] The present invention also discloses a ranging method based on point cloud data collected by UAV and image registration. Invention Overview:

[0010] This invention achieves registration of point clouds acquired by UAVs with captured images through proposed camera intrinsic parameter acquisition and pose calculation; visualizes the registration effect through the proposed registration visualization module; and identifies, locates, and measures the actual distance between protected areas and hazards in monitored images through proposed hazard identification and positioning, registration transformation, and distance calculation.

[0011] Detailed technical solution of this invention:

[0012] An inspection method based on point cloud data collected by UAV and image registration, characterized by comprising:

[0013] Step S0: Acquire point cloud and image data of the target scene. Use a drone equipped with a lidar to scan the target scene and acquire the required point cloud and image data.

[0014] Step S1: Preprocess the point cloud and image data to remove noise points from the point cloud data and blurry images from the image data;

[0015] Step S2: Calculate the camera intrinsic parameter matrix M. The camera intrinsic parameter matrix M is calculated using the camera focal length, the optical center position in the pixel coordinate system, the camera film size, and the camera image size. The calculation process is as follows:

[0016]

[0017] In formula (1), f x = (f×U) / w, f y = (f×V) / h; camera focal length f, in mm; optical center position in pixel coordinate system (c x ,c y The camera film size is w*h (width*height), in mm; the camera image size is U*V (width*height), in pixels. All of the above parameters are constants.

[0018] Step S3: Facilitates pose calculation. The pose calculation process involves obtaining the transformation relationships between several coordinate systems. The point cloud data is processed based on the obtained transformation relationships to obtain point cloud data based on the camera coordinate system. This step directly uses the inertial navigation parameters recorded by the UAV in real time for pose calculation without any other preprocessing steps. This step discloses all possible transformation situations between the UAV inertial navigation and the gimbal inertial navigation, and is suitable for pose calculation with infinite changes in any direction and angle.

[0019] Compared with existing technologies, this invention does more than just address the issue of two additional angle conversions. First, angle conversion is the final and simplest step in pose calculation, considered only after the relationship between the gimbal's inertial navigation and the UAV's inertial navigation has been clearly defined. However, existing technologies do not disclose this inertial navigation relationship, while this invention lists all possible transformations between the UAV's inertial navigation and the gimbal's inertial navigation. It can be said that the pose calculation steps disclosed in this patent are quite comprehensive. Second, existing technologies do not explain how they read the pitch angle, and the pitch angle only changes once. In this invention, any angle can be changed an unlimited number of times. Furthermore, this invention directly uses the inertial navigation data recorded by the UAV in real time for calculation, without intermediate processing steps. It can then be directly embedded into the chip carried by the UAV, making its applications more extensive.

[0020] Step S4: Registration and visualization module: Based on the camera coordinate system, the point cloud data is processed according to the known camera intrinsic parameter matrix M to obtain the depth map.

[0021] Define the coordinate system in advance:

[0022] WGS84 coordinate system: This is an existing publicly available coordinate system. The horizontal axis is longitude B, the vertical axis is latitude L, and the vertical axis is altitude H. The coordinates are represented by (B, L, H).

[0023] Earth-centered and Earth-fixed coordinate system: This is an existing and publicly available coordinate system, with coordinates represented by (X,Y,Z);

[0024] Station-centered coordinate system: The horizontal coordinate X″ axis points east, the vertical coordinate Y″ axis points north, and the vertical coordinate Z″ axis points to the sky. The coordinates are represented by (X″,″,″).

[0025] Northeast coordinate system: the horizontal coordinate X″″ points north, the vertical coordinate Y″″ points east, and the vertical coordinate Z″″ points to the ground. The coordinates are represented by (X″″, Y″″, Z″″).

[0026] Camera coordinate system: The horizontal coordinate X″″ points in the direction of the normal vector of the camera plane, the vertical coordinate Y″″ points in the direction to the right parallel to the long side of the camera plane, and the vertical coordinate Z″″ points in the direction downward parallel to the short side of the camera plane. The coordinates are represented by (X″″, Y″″, Z″″).

[0027] Let the point cloud data obtained in step 3 based on the camera coordinate system be x, y, z, where x, y, and z are the coordinate values ​​of a spatial point on the X″″ axis, Y″″ axis, and Z″″ axis in the camera coordinate system, and z is also called the depth value. The pixel coordinates (u, v) of the depth map are calculated based on x, y, and z.

[0028] u=x*f x / z+c x(2)

[0029] v = y * f y / z+c y (3);

[0030] The camera image size is U*V. For the calculated u, the portion between 0 and U is retained; for the calculated v, only the portion between 0 and V is retained. Then, the retained u and v are rounded down. The obtained depth map and the corresponding visible light image are then pixel-weightedly fused to obtain a point cloud-image mapping model. The fused image can clearly show the registration effect.

[0031] A precise mapping model between point cloud and image is established through steps S0-S4, namely, a mapping model between two-dimensional pixels and three-dimensional real points in the image. This mapping model can be used to convert any two-dimensional pixel in the image to three-dimensional real position coordinates. The pixel weighted fusion described is achievable by those skilled in the art and is not the content to be protected by this invention.

[0032] Step S5: Hazard identification and localization. A pre-trained detection algorithm, such as the YOLO target detection algorithm, is called to identify the image data captured by the UAV during inspection, obtain hazard detection boxes, record the bottom center coordinates of the detection boxes, and process the bottom center coordinates of the detection boxes according to the point cloud-image mapping model to obtain the three-dimensional real position coordinates of the hazard, so as to realize the localization of the hazard location.

[0033] A ranging method based on point cloud acquisition and image registration using UAV, characterized by comprising:

[0034] Distance measurement is performed to delineate the protected area. Visualization tools (e.g., drawing tools, Cloudcompare), automated tools (e.g., Labelimg), or segmentation algorithms (e.g., Mask R-CNN) are used to divide the protected area, obtaining the two-dimensional or three-dimensional coordinates of the boundary points.

[0035] If the two-dimensional coordinates of the boundary point of the protected area are obtained, the point cloud-image mapping model is used to convert them into the corresponding three-dimensional coordinates, and then the distance between the boundary point of the protected area and the location of the hidden danger is calculated.

[0036] If the three-dimensional coordinates of the boundary point of the protected area are obtained, the distance to the location of the hidden danger can be directly calculated.

[0037] According to a preferred embodiment of the present invention, in step S1, the method for removing noise points from the point cloud data is as follows: using a point cloud discrete point removal algorithm, preferably a radius filtering algorithm, to preprocess the point cloud data to remove noise points;

[0038] The method for removing blurry images from image data is as follows: using an image blur discrimination algorithm, preferably the Tenengrad gradient method, to remove blurry images.

[0039] According to a preferred embodiment of the present invention, the pose calculation process in step S3 is as follows:

[0040] (3-1) Determine the initial pose of the UAV: ​​Before the UAV takes off, set the gimbal to center and reset to ensure that the initial pose of the UAV is the same as the initial pose of the gimbal; take pictures to save the yaw, pitch and roll of the UAV at this moment, which are recorded as y0, p0 and r0 respectively.

[0041] (3-2) Correction of pose transformation angle: Images taken after the UAV takes off to record:

[0042] The drone's yaw, pitch, and roll angles are respectively y t ,p t ,r t ;

[0043] The gimbal's yaw, pitch, and roll angles are y′, ... t ,p′ t ,r′ t ;

[0044] The correction of the pose transformation angle is determined based on the specific relationship between the gimbal inertial navigation and the UAV inertial navigation:

[0045] When the gimbal's pose does not change with the drone's pose, but is achieved through coordinated changes in the angles of the gimbal and the drone, the corrected gimbal's pose transformation angle relative to the northeast coordinate system is: y′ t -y t +y0,p′ t -p t +p0,r′ t -r t +r0;

[0046] When the gimbal pose does not change with the drone pose, and is not achieved through coordinated changes in the gimbal and drone angles, the corrected gimbal pose transformation angle relative to the northeast coordinate system is: y′ t +y0,p′ t +p0,r′ t +r0;

[0047] When the gimbal's pose changes only with the UAV's pose, the corrected gimbal's pose transformation angle relative to the northeast coordinate system is: y t ,p t,r t ;

[0048] The above three are the most typical relationships between gimbal inertial navigation and UAV inertial navigation. The above relationships can be applied according to the specific situation of each direction and angle.

[0049] (3-3) Solve for the pose transformation matrix, and solve for the rotation matrix based on the angle of the gimbal relative to the northeast coordinate system:

[0050] If the rotations around the three axes North-X″′, East-Y″′, and Earth-Z″′ are r, p, and y respectively, then the rotation matrix R is calculated as follows:

[0051]

[0052]

[0053]

[0054] R = R x (r)*R y (p)*R z (y) (8);

[0055] In formulas (5)-(8), R x (r), R y (p), R z (y) represents the rotation matrix for transforming the northeast coordinate system around the coordinate axes to the gimbal pose:

[0056] (3-4) Transform the coordinate system of the point cloud data to realize the transformation from WGS84 coordinates to the geocentric coordinate system, the station-centered coordinate system, the northeast-east coordinate system and the camera coordinate system in sequence;

[0057] a. Transformation from the WGS84 coordinate system to the Earth-centered Earth-fixed coordinate system:

[0058] The WGS84 coordinate system uses longitude B, latitude L, and altitude H to record the coordinates (B, L, H) of a three-dimensional point. Formulas (9)-(15) are used to convert the coordinates (B, L, H) of the three-dimensional point into the coordinates (X, Y, Z) of a three-dimensional point based on the geocentric coordinate system:

[0059] F = 1 / 298.257223563 (9)

[0060] a = 6378137 (10)

[0061]

[0062]

[0063]

[0064] In formulas (9)-(13), F is the flattening of the Earth's ellipticity; a is the length of the semi-major axis of the ellipse; B is the longitude of the point; L is the latitude of the point; and H is the altitude of the point.

[0065] b. Then, the transformation from the geocentric coordinate system to the station-centered coordinate system.

[0066] Using the formulas (16)-(18) below, the three-dimensional point coordinates (X,Y,Z) in the geocentric coordinate system are transformed to the station-centered coordinate system. The camera optical center is selected as the origin of the station-centered coordinate system, and its geocentric coordinates are (X0,Y0,Z0). Finally, the coordinate data (X″,Y″,Z″) of the point cloud data in the station-centered coordinate system are obtained:

[0067]

[0068]

[0069]

[0070] In formulas (16)-(18), (X′,Y′,Z′) are the coordinate values ​​of the point cloud data after translation; C is the rotation matrix parameter from the geocentric coordinate system to the stationary coordinate system;

[0071] c. Then, transform from the station-centered coordinate system (X″ axis points east, Y″ axis points north, Z″ axis points to the sky) to the northeast-ground coordinate system (X″′ axis points north, Y″′ axis points east, Z″′ axis points to the ground).

[0072] Using formula (19), the three-dimensional point coordinates (X″, Y″, Z″) in the station-centered coordinate system are transformed to the northeast-northeast coordinate system, and the coordinates are marked as (X″′, Y″′, Z″′):

[0073]

[0074] d. Then, the transformation from the northeast coordinate system to the camera coordinate system.

[0075] The following formula (20) is used to transform the three-dimensional point coordinates (X″′, Y″′, Z″′) in the northeast coordinate system to the camera coordinate system, where R is the pose transformation matrix obtained in step (3-3):

[0076]

[0077] According to a preferred embodiment of the present invention, a preset condition is further included before the pose calculation in step S3:

[0078] The yaw, pitch, and roll angles of the UAV's inertial navigation system, as recorded in the images captured by the UAV, are relative to the northeast coordinate system; the positive and negative values ​​of the yaw, pitch, and roll angles correspond exactly to the clockwise and counterclockwise rotations of the XOY, ZOX, and YOZ planes, respectively.

[0079] The yaw, pitch, and roll angles of the gimbal inertial navigation system recorded in images captured by the drone are relative to the initial position of the gimbal, and their positive and negative definitions are consistent with the angle parameters of the drone.

[0080] Technical advantages of the present invention:

[0081] This invention determines the initial state of the UAV, utilizes 3D rotation technology to clarify the relationship between the known public coordinate system and the UAV's inertial navigation coordinate system, and directly performs point cloud and image registration based on the recorded parameters. It exhibits strong interpretability, versatility, and generalization. This invention goes beyond simply solving two additional angle conversion problems; it also includes:

[0082] (1) In view of the fact that the inertial navigation relationship is not disclosed in the existing technology, this invention lists all possible transformations between UAV inertial navigation and gimbal inertial navigation, and comprehensively records the pose calculation steps;

[0083] (2) In this invention, any angle can be changed an unlimited number of times, and this invention directly uses the inertial navigation data recorded in real time by the UAV for calculation without intermediate processing steps. It can be directly embedded into the chip carried by the UAV, making it more widely used. Attached Figure Description

[0084] Figure 1 This is a flowchart of the inspection and ranging method described in this invention;

[0085] Figure 2 This is a schematic diagram illustrating an example of point cloud data for a target scene in this invention;

[0086] Figure 3 This is an example of target scene image data in this invention;

[0087] Figure 4 This is the registration depth map in this invention. Detailed Implementation

[0088] The present invention will now be described in detail with reference to the embodiments and accompanying drawings, but is not limited thereto.

[0089] Example 1

[0090] An inspection method based on point cloud data collected by UAV and image registration includes:

[0091] Step S0: Acquire point cloud and image data of the target scene. Use a drone equipped with a LiDAR to scan the target scene and acquire the required point cloud and image data. The point cloud data is as follows: Figure 2 As shown, the image data is as follows Figure 3 As shown, there are a total of 69 images;

[0092] Step S1: Preprocess the point cloud and image data to remove noise points from the point cloud data and blurry images from the image data;

[0093] In step S1, the method for removing noise points from the point cloud data is as follows: using a point cloud discrete point removal algorithm, here a radius filtering algorithm is used, the radius is set to 0.2, the point cloud number threshold is 50, and the point cloud data is preprocessed to remove noise points.

[0094] The method for removing blurry images from the image data is as follows: using an image blur discrimination algorithm, preferably the Tenengrad gradient method, setting the gradient threshold to 3 to remove blurry images, leaving 67 images in the end;

[0095] Step S2: Calculate the camera intrinsic parameter matrix M. The camera intrinsic parameter matrix M is calculated using the camera focal length (8mm for the lens mounted on the drone), the optical center position in the pixel coordinate system (optical center position is (2736, 1824)), the camera film size (13.2mm × 8.8mm (width × height)), and the camera image size (5472 × 3648 (width × height)). The calculation process is as follows:

[0096]

[0097] In formula (1), f x = (f×u) / w, f y = (f×v) / h; camera focal length f, in mm; optical center position in pixel coordinate system (c x ,c y The camera film size is w*h (width*height), in mm; the camera image size is u*v (width*height), in pixels.

[0098] The calculation formula (1) yields: M = [[3648,0,2736],[0,3648,1824],[0,0,1]];

[0099] Step S3: Facilitates pose calculation. The pose calculation process involves obtaining the transformation relationships between several coordinate systems. The point cloud data is processed based on the obtained transformation relationships to obtain point cloud data based on the camera coordinate system. This step directly uses the inertial navigation parameters recorded by the UAV in real time for pose calculation without any other preprocessing steps. This step discloses all possible transformation situations between the UAV inertial navigation and the gimbal inertial navigation, and is suitable for pose calculation with infinite changes in any direction and angle.

[0100] Step S4: Registration and visualization module: Based on the camera coordinate system, the point cloud data is processed according to the known camera intrinsic parameter matrix M to obtain the depth map.

[0101] Define the coordinate system in advance:

[0102] WGS84 coordinate system: This is an existing publicly available coordinate system. The horizontal axis is longitude B, the vertical axis is latitude L, and the vertical axis is altitude H. The coordinates are represented by (B, L, H).

[0103] Earth-centered and Earth-fixed coordinate system: This is an existing and publicly available coordinate system, with coordinates represented by (X,Y,Z);

[0104] Station-centered coordinate system: The horizontal coordinate X″ axis points east, the vertical coordinate Y″ axis points north, and the vertical coordinate Z″ axis points to the sky. The coordinates are represented by (X″,″,″).

[0105] Northeast coordinate system: the horizontal coordinate X″″ points north, the vertical coordinate Y″″ points east, and the vertical coordinate Z″″ points to the ground. The coordinates are represented by (X″″, Y″″, Z″″).

[0106] Camera coordinate system: The horizontal coordinate X″″ points in the direction of the normal vector of the camera plane, the vertical coordinate Y″″ points in the direction to the right parallel to the long side of the camera plane, and the vertical coordinate Z″″ points in the direction downward parallel to the short side of the camera plane. The coordinates are represented by (X″″, Y″″, Z″″).

[0107] Let the point cloud data obtained in step 3 based on the camera coordinate system be x, y, z, where x, y, z are the coordinate values ​​corresponding to the x-axis, y-axis, and z-axis in the camera coordinate system, and z is also called the depth value. Calculate the pixel coordinates (u, v) of the depth map based on x, y, z:

[0108] u=x*f x / z+c x (3)

[0109] v = y * f y / z+c y (4);

[0110] The camera image size is U*V. Values ​​of u (0-U) and v (0-V) are taken, and both u and v are rounded down. The resulting depth map is then pixel-weightedly fused with the corresponding visible light image to obtain a point cloud-image mapping model. The fused image clearly shows the registration effect.

[0111] A precise mapping model between point cloud and image is established through steps S0-S4, namely, a mapping model between two-dimensional pixels and three-dimensional real points in the image. This mapping model can be used to convert any two-dimensional pixel in the image to three-dimensional real position coordinates. The pixel weighted fusion described is achievable by those skilled in the art and is not the content to be protected by this invention.

[0112] The point cloud data based on the camera coordinate system mentioned above is denoted as:

[0113] x=[[5.7321655,6.86145656,8.18334788...-54.55782049,-62.28270174,-69.43021049];

[0114] y=[36.49734829,38.91588856,41.20076682...9.16944682,4.60647502,1.16719196];

[0115] z=[4.98927533,2.89691339,0.95749379...65.33247165,85.66599562,100.90904878]];

[0116] Based on the known camera intrinsic parameter matrix M and the calculation formula given in the invention, the specific data of the pixel coordinates u and v corresponding to z are as follows:

[0117] u=[[2506,1843 1590...446 84 226]

[0118] v=[1855 2186 2338...2350 2020 1866]]

[0119] The obtained depth map is then pixel-weightedly fused with the corresponding 0018 image. The resulting image clearly shows the registration effect. The registration depth map of the 0018 image is shown below. Figure 4 As shown;

[0120] Step S5: Hazard identification and localization. A pre-trained detection algorithm is invoked, such as the YOLO target detection algorithm. This algorithm can identify hazard categories such as excavators, tower cranes, and trucks, and identify image data captured during drone inspections to obtain hazard detection boxes. The bottom center coordinates of the detection boxes are recorded. The coordinates of the bottom center point of the detection boxes are processed according to the point cloud-image mapping model to obtain the three-dimensional true location coordinates of the hazard, thereby achieving hazard localization.

[0121] Specifically, image 0018 is identified to obtain a hazard detection box. The bottom center coordinates of the detection box are recorded as [1843, 2186]. The bottom center coordinates of the detection box are processed according to the point cloud-image mapping model to obtain the three-dimensional true position coordinates of the hazard based on the geocentric coordinate system as [-2406434.36592097, 4509951.63989265, 3801799.00041652], thus realizing the location of the hazard.

[0122] Example 2

[0123] In Example 1, a preset condition is included before the pose calculation in step S3:

[0124] The yaw, pitch, and roll angles of the UAV's inertial navigation system, as recorded in the images captured by the UAV, are relative to the northeast coordinate system; the positive and negative values ​​of the yaw, pitch, and roll angles correspond exactly to the clockwise and counterclockwise rotations of the XOY, ZOX, and YOZ planes, respectively.

[0125] The yaw, pitch, and roll angles of the gimbal inertial navigation system recorded in images captured by the drone are relative to the initial position of the gimbal, and their positive and negative definitions are consistent with the angle parameters of the drone.

[0126] Example 3

[0127] In Examples 1 and 2, the pose calculation process in step S3 is illustrated by taking the pose calculation of the captured image named 0018 as an example:

[0128] (3-1) Determine the initial attitude of the UAV: ​​Before the UAV takes off, set the gimbal to center and reset to ensure that the initial attitude of the UAV is the same as the initial attitude of the gimbal. At the initial moment, the yaw, pitch and roll of the gimbal are 0°, 0° and 0° respectively. Take pictures to save the yaw, pitch and roll of the UAV at this moment, and record them as: y0 = 0.7°, p0 = 0.04° and r0 = 0.46° respectively.

[0129] (3-2) Correcting pose transformation angle: Read the EXIF ​​file storage information of the image, the picture taken after the UAV took off, and obtain the time of image capture 0018:

[0130] The drone's yaw, pitch, and roll angles are respectively y t =-18.4,p t =2.63,r t =2.4;

[0131] The gimbal's yaw, pitch, and roll angles are y′, ... t =21.98,p′ t =-11.4,r′ t =0;

[0132] The correction of the pose transformation angle is determined based on the specific relationship between the gimbal inertial navigation and the UAV inertial navigation:

[0133] When the gimbal's pose does not change with the drone's pose, but is achieved through coordinated changes in the angles of the gimbal and the drone, the corrected gimbal's pose transformation angle relative to the northeast coordinate system is: y′ t -y t +y0,p′ t -p t +p0,r′ t -r t +r0;

[0134] When the gimbal pose does not change with the drone pose, and is not achieved through coordinated changes in the gimbal and drone angles, the corrected gimbal pose transformation angle relative to the northeast coordinate system is: y′ t +y0,p′ t +p0,r′ t +r0; The pitch and roll poses of a certain brand of drone are not changed by the drone's pose because the angles of the gimbal and the drone are linked. However, the yaw pose of the gimbal is not changed by the drone's pose because it is achieved independently by the inertial navigation system. Therefore, the corrected pose transformation angle of the gimbal relative to the NE-G coordinate system is: y′ t +y0=22.68,p′ t -p t +p0=13.99,r′ t -r t +r0 = -1.94;

[0135] When the gimbal's pose changes only with the UAV's pose, the corrected gimbal's pose transformation angle relative to the northeast coordinate system is: y t ,p t ,rt ;

[0136] (3-3) Solve for the pose transformation matrix, and solve for the rotation matrix based on the angle of the gimbal relative to the northeast coordinate system:

[0137] The rotation angles around the three axes North-X″′, East-Y″′, and Earth-Z″′ are r = -1.94, p = 13.99, and y = 22.68, respectively. The rotation matrix R is calculated as follows:

[0138]

[0139]

[0140]

[0141] R = R x (r)*R y (p)*R z (y) (8);

[0142] In formulas (5)-(8), R x (r), R y (p), R z (y) represents the rotation matrix of the northeast coordinate system around the coordinate axis to the gimbal pose;

[0143] Specifically, we obtained:

[0144] (3-4) Transform the coordinate system of the point cloud data to realize the transformation from WGS84 coordinates to the geocentric coordinate system, the station-centered coordinate system, the northeast-east coordinate system and the camera coordinate system in sequence;

[0145] a. Transformation from the WGS84 coordinate system to the Earth-centered Earth-fixed coordinate system:

[0146] The WGS84 coordinate system uses longitude B, latitude L, and altitude H to record the coordinates (B, L, H) of a three-dimensional point. Formulas (9)-(15) are used to convert the coordinates (B, L, H) of the three-dimensional point into the coordinates (X, Y, Z) of a three-dimensional point based on the geocentric coordinate system:

[0147] F = 1 / 298.257223563 (9)

[0148] a = 6378137 (10)

[0149]

[0150]

[0151]

[0152] In formulas (9)-(13), F is the flattening of the Earth's ellipticity; a is the length of the semi-major axis of the ellipse; B is the longitude of the point; L is the latitude of the point; and H is the altitude of the point.

[0153] The point cloud data collected by the drone is stored in .las format. The point cloud data is based on the WGS84 coordinate system, and the specific data is as follows:

[0154] [[118.08360885,118.08363108,118.08365456...118.0827095,118.0825398,118.08239854];

[0155] [36.8237208,36.82370295,36.82368727...36.82405434,36.82420095,36.8243062];

[0156] [83.70712474,81.79792474,79.97772474...98.36152474,98.38632474,98.46552474]];

[0157] The point cloud data based on the geocentric-geostatic coordinate system is obtained by processing the data using the formula mentioned in the invention:

[0158] [[-2406432.7766533,-2406434.36592097,-2406436.01917056...-2406357.06389637,-2406339.1249067,-2406324.74009165];

[0159] [4509952.8744169,4509951.63989265,4509950.28838031...4509981.42296348,4509979.96401064,4509979.77613317];

[0160] [3801801.73039674,3801799.00041652,3801796.5165498...3801840.14337471,3801853.18219388,3801862.57942842]];

[0161] b. Then, the transformation from the geocentric coordinate system to the station-centered coordinate system.

[0162] Using the formulas (16)-(18) below, the three-dimensional point coordinates (X,Y,Z) in the geocentric coordinate system are transformed to the station-centered coordinate system. The camera optical center is selected as the origin of the station-centered coordinate system, and its geocentric coordinates are (X0,Y0,Z0). Finally, the coordinate data (X″,Y″,Z″) of the point cloud data in the station-centered coordinate system are obtained:

[0163]

[0164]

[0165]

[0166] In formulas (16)-(18), (X ′ ,Y ′ Z ′ The coordinates of the point cloud data after translation; C is the rotation matrix parameter from the geocentric coordinate system to the stationary coordinate system;

[0167] The camera's optical center is selected as the origin of the station-centered coordinate system. Using the formula mentioned in the invention, the three-dimensional point coordinates (X, Y, Z) in the geocentric coordinate system are transformed to the station-centered coordinate system, resulting in the following coordinate data of the point cloud data in the station-centered coordinate system:

[0168] [[5.57812432,7.56143777,9.65627318...-74.65984302,-89.79991904,-102.40265111]

[0169] [-1.47580032,-3.45671583,-5.19681228...35.53947672,51.8098239,63.49019792]

[0170] [-36.83087786,-38.74008067,-40.56028468...-22.17701101,-22.15251773,-22.07361329]];

[0171] c. Then, transform from the station-centered coordinate system to the northeast-oriented coordinate system.

[0172] Using formula (19), the three-dimensional point coordinates (X″, Y″, Z″) in the station-centered coordinate system are transformed to the northeast-northeast coordinate system, and the coordinates are marked as (X″′, Y″′, Z″′):

[0173]

[0174] Using the formula mentioned in the invention, the three-dimensional point coordinates in the station-centered coordinate system are transformed to the northeast-northeast coordinate system. The transformed three-dimensional coordinate data are as follows:

[0175] [[-1.47580032,-3.45671583,-5.19681228...35.53947672,51.8098239,63.49019792];

[0176] [5.57812432,7.56143777,9.65627318...-74.65984302,-89.79991904,-102.40265111];

[0177] [36.83087786,38.74008067,40.56028468...22.17701101,22.15251773,22.07361329]];

[0178] d. Then, the transformation from the northeast coordinate system to the camera coordinate system.

[0179] The following formula (20) is used to transform the three-dimensional point coordinates (X″′, Y″′, Z″′) in the northeast coordinate system to the camera coordinate system, where R is the pose transformation matrix obtained in step (3-3):

[0180]

[0181] Using the formula mentioned in the invention, the three-dimensional point coordinates in the northeast coordinate system are transformed to the camera coordinate system. The transformed three-dimensional coordinate data is as follows:

[0182] [[5.7321655,6.86145656,8.18334788...-54.55782049,-62.28270174,-69.43021049];

[0183] [36.49734829,38.91588856,41.20076682...9.16944682,4.60647502,1.16719196];

[0184] [4.98927533,2.89691339,0.95749379...65.33247165,85.66599562,100.90904878]].

[0185] Example 4

[0186] A ranging method based on point cloud data collected by UAV and image registration includes:

[0187] Distance measurement is performed to delineate the protected area. Visualization tools (e.g., drawing tools, Cloudcompare), automated tools (e.g., Labelimg), or segmentation algorithms (e.g., Mask R-CNN) are used to divide the protected area, obtaining the two-dimensional or three-dimensional coordinates of the boundary points.

[0188] If the two-dimensional coordinates of the boundary point of the protected area are obtained, the point cloud-image mapping model is used to convert them into the corresponding three-dimensional coordinates, and then the distance between the boundary point of the protected area and the location of the hidden danger is calculated.

[0189] If the three-dimensional coordinates of the boundary point of the protected area are obtained, the distance to the location of the hidden danger can be directly calculated.

[0190] Specifically: The location of the protected area is delineated. Using drawing software, the protected area is divided on the 0018 image, and the coordinates of the boundary pixels of the protected area are recorded as [[2515, 1877], [1856, 2215], [1604, 2370]...[463, 2393], [99, 2067][240, 1912]]. Based on the point cloud-image mapping model, these coordinates are converted into three-dimensional coordinates based on the geocentric coordinate system: [[-2406432.7766533, 4509952.8744169, 3801801.73039674], [-2406434.36592097, 4509951.63989265, 3801799.00041652]. The distance formula is used to calculate the shortest true distance between the three-dimensional coordinates of the boundary of the protected area and the three-dimensional coordinates of the hidden danger, which is 51.2m.

Claims

1. A method for inspection based on registration of point cloud and image collected by a UAV, characterized in that, The method comprises the following steps: Step S0: obtaining target scene point cloud and image data; Step S1: preprocessing the point cloud and image data to remove noise points in the point cloud data and to remove blurred images in the image data; Step S2: Calculate the camera intrinsic matrix The calculation process is as follows: （1） In formula (1), , ; focal length of camera , unit is ; position of optical center in pixel coordinate system ; size of camera film , units are ; imaging size of camera , in pixels, and the above parameters are constants; Step S3: using pose solving to obtain point cloud data based on a camera coordinate system; Step S4: The registration visualization module registers the point cloud data based on the camera coordinate system according to the known camera intrinsic matrix Processing to obtain a depth map: Defining coordinate systems in advance: WGS84 coordinate system: this is a publicly known coordinate system, the horizontal coordinate is longitude B, the vertical coordinate is latitude L, and the vertical coordinate is altitude H, and the coordinates are represented by (B, L, H); Earth-centered Earth-fixed coordinate system: this is a publicly known coordinate system, and the coordinates are represented by (X, Y, Z); ECEF: east coordinate N: north coordinate Z: vertical coordinate T: time coordinate T: time coordinate North-East Geodetic Coordinate System: abscissa North, ordinate East, ordinate Ground, coordinate , , indicated; Camera coordinate system: horizontal coordinate Normal vector direction of the camera plane, vertical coordinate Direction pointing to the right parallel to the long side of the camera plane, vertical coordinate Direction pointing down parallel to the short side of the camera plane, coordinate using ( , , ) indicates; Let the point cloud data based on the camera coordinate system obtained in the step S3 be x, y, z, wherein x, y, z are coordinates of a spatial point in the camera coordinate system axis, axis, axis, the coordinate value z, also called a depth value, is calculated according to x, y, M, and a pixel point coordinate (u, v) of a depth map: （2）； （3）； The camera imaging size is U*V, for the obtained u, only the part between 0 and U is retained, for the obtained v, only the part between 0 and V is retained, then, the retained u and v are respectively subjected to down rounding operation; the obtained depth map and the corresponding visible light image are pixel weighted and fused to obtain a point cloud-image mapping model; Step S5: hazard identification and positioning, calling a pre-trained detection algorithm to identify the image data taken by the unmanned aerial vehicle during inspection to obtain a hazard class detection box, recording the bottom center coordinates of the detection box, and processing the bottom center point coordinates of the detection box according to the point cloud-image mapping model to obtain three-dimensional real position coordinates of the hazard. 2.The inspection method based on registration of point cloud and image collected by UAV according to claim 1, wherein, In step S1, the method for removing noise points in the point cloud data is: using a point cloud discrete point removal algorithm, including a radius filtering algorithm, to preprocess the point cloud data to remove noise points; The method for removing blurred images in the image data is: using an image blur discrimination algorithm, including a Tenengrad gradient method, to remove blurred images. 3.The inspection method based on registration of point cloud and image collected by UAV according to claim 1, wherein, The process of pose solving in step S3 is as follows: (3-1) Determine the initial pose of the unmanned aerial vehicle: set the gimbal to the center reset before the unmanned aerial vehicle takes off, so that the initial pose of the unmanned aerial vehicle is the initial pose of the gimbal; take a picture to record the yaw angle yaw, pitch angle pitch, and roll angle roll of the unmanned aerial vehicle at this moment, respectively denoted as: ; (3-2) correcting the pose transformation angle: taking a picture after the unmanned aerial vehicle takes off to record: The yaw angle yaw, the pitch angle pitch, and the roll angle roll of the UAV are respectively ; The yaw angle, the pitch angle, and the roll angle of the gimbal are yaw, pitch, and roll, respectively ; The correction of the pose transformation angle is determined according to the specific relationship between the gimbal inertial navigation and the unmanned aerial vehicle inertial navigation: When the pose of the holder does not change with the change of the pose of the unmanned aerial vehicle and is achieved through the angle linkage change of the holder and the unmanned aerial vehicle, the corrected pose transformation angle of the holder relative to the north-east terrestrial coordinate system is: ; When the gimbal pose does not change with the change of the UAV pose, and is not realized through the angle linkage change between the gimbal and the UAV, the corrected gimbal pose transformation angle relative to the north-east ground coordinate system is: ; When the pose of the gimbal only changes with the change of the pose of the unmanned aerial vehicle, the pose transformation angle of the corrected gimbal relative to the north-east-geodetic coordinate system is: ; (3-3) solving the pose transformation matrix: solving the rotation matrix according to the obtained angle of the gimbal relative to the north-east-ground coordinate system: Rotates around the north , east , and ground axes by angles r, p, y , respectively. The rotation matrix R is calculated as follows: （5） （6） （7） （8）； In the formula (5) - formula (8), , , Respectively, the rotation matrix of the north-east geodetic coordinate system around the coordinate axis to the rotation matrix of the PT pose: (3-4) transforming the point cloud data coordinate system to realize the conversion from the WGS84 coordinate to the earth-centered earth-fixed coordinate system, the station-centered coordinate system, the north-east-ground coordinate system and the camera coordinate system; a. conversion from the WGS84 coordinate system to the earth-centered earth-fixed coordinate system: The WGS84 coordinate system records the coordinates (B, L, H) of a three-dimensional point using longitude B, latitude L and altitude H, and uses formulas (9)-(15) to convert the coordinates (B, L, H) of the three-dimensional point to the three-dimensional point coordinates (X, Y, Z) based on the earth-centered earth-fixed coordinate system: （9） （10） （11） （12） （13） In formulas (9)-(13), F is the ellipticity of the earth's spheroid; a is the length of the major axis of the ellipse; B is the longitude of the point; L is the latitude of the point; H is the altitude of the point: b. conversion from the earth-centered earth-fixed coordinate system to the station-centered coordinate system The three-dimensional point coordinates (X, Y, Z) in the geocentric and fixed coordinate system are converted to the station-centered coordinate system by using the following formulas (16)-(18). The camera optical center point is selected as the origin of the station-centered coordinate system, and its geocentric and fixed coordinates are Finally, the coordinate data of the point cloud data in the station-centered coordinate system is obtained ): （16） （17） （18） In the formulas (16)-(18), (x, y, z) is the coordinate value of the point cloud data after translation; ) is the coordinate value of the point cloud data after translation; is the rotation matrix parameter from the geocentric coordinate system to the station-centered coordinate system; c. conversion from the station-centered coordinate system to the north-east-ground coordinate system Using formula (19), the coordinates of the three-dimensional point in the station center coordinate system are ( Transform to the northeast coordinate system, with coordinates marked as ( ): （19）； d. conversion from the north-east-ground coordinate system to the camera coordinate system Using the following formula (20), the coordinates of a three-dimensional point in the northeast coordinate system are ( Transform to the camera coordinate system, where R is the pose transformation matrix obtained in step (3-3): （20）。 4.The inspection method based on registration of point cloud and image collected by UAV according to claim 1, characterized in that, Before the pose solving in step S3, a preset condition is further included: The yaw angle, the pitch angle and the roll angle of the unmanned aerial vehicle body inertial navigation recorded by the unmanned aerial vehicle shooting image are relative to the north-east ground coordinate system; the positive and negative of the yaw angle, the pitch angle and the roll angle correspond to the clockwise and counterclockwise rotation of the XOY, ZOX and YOZ planes. The yaw angle, the pitch angle and the roll angle of the gimbal inertial navigation recorded by the unmanned aerial vehicle shooting image are relative to the initial position of the gimbal, and the definition of the positive and negative is consistent with the angle parameters of the unmanned aerial vehicle.

5. A ranging method based on unmanned aerial vehicle acquisition point cloud and image registration, characterized in that, The method comprises the following steps: Distance measurement, delimiting the position of the protection zone, obtaining the two-dimensional or three-dimensional coordinates of the boundary points of the protection zone: If the two-dimensional coordinates of the boundary points of the protection zone are obtained, the point cloud-image mapping model of any one of claims 1-4 is used to convert the corresponding three-dimensional coordinates, and then the distance measurement between the boundary points of the protection zone and the location of the hidden danger is performed; If the three-dimensional coordinates of the boundary points of the protection zone are obtained, the distance measurement between the boundary points of the protection zone and the location of the hidden danger is directly performed.

Images