A Method and System for Autonomous Inspection of Unmanned Aerial Vehicles Based on Prior Structural Knowledge

By using a UAV autonomous inspection method based on prior structural knowledge, inspection routes and orientations are generated using single tower images, solving the problem of multi-view image dependence in existing technologies and improving the efficiency and real-time performance of UAV inspections.

CN123113944APending Publication Date: 2026-11-17CENT SOUTH UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611603920.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-10-16
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing UAV autonomous inspection technology requires multi-view images to generate flight paths, which is inefficient and computationally complex.

Method used

By acquiring knowledge templates of pole and tower structures, two-dimensional key points are extracted from single images of pole and towers captured by drones. Guided by prior structural knowledge, rotation matrices and translation parameters are calculated to generate the inspection flight path and orientation of the drone. This allows the spatial transformation relationship between the real pole and tower and the template to be solved from a single image.

Benefits of technology

It significantly reduces the number of image acquisitions and the time required, lowers the complexity of coordinate calculations, improves the efficiency and real-time performance of inspection route generation, and enhances the overall efficiency of UAV autonomous inspection.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123113944A_ABST
    Figure CN123113944A_ABST
Patent Text Reader

Abstract

This application relates to a method and system for autonomous inspection of unmanned aerial vehicles (UAVs) based on prior structural knowledge. The method includes: acquiring a knowledge template of a tower structure and extracting two-dimensional key points from images of the tower acquired by the UAV; minimizing an objective function based on each three-dimensional key point and its corresponding two-dimensional key point to obtain a first rotation matrix and a first translation parameter; obtaining a second pose matrix of the tower's local coordinate system relative to the world coordinate system based on the first rotation matrix, the first translation parameter, and the first pose matrix of the camera coordinate system relative to the world coordinate system; determining the inspection coordinates of each inspection viewpoint in the world coordinate system and the observation coordinates of the observed target in the world coordinate system based on the second pose matrix; determining the UAV's inspection flight path based on the inspection coordinates, determining the UAV's orientation based on the observation coordinates, and controlling the UAV to perform autonomous inspection based on the inspection flight path and orientation. This method can improve efficiency.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of unmanned aerial vehicle (UAV) inspection technology, and in particular to an autonomous UAV inspection method and system guided by prior structural knowledge. Background Technology

[0002] With the development of drone platforms, airborne cameras, and intelligent vision algorithms, drones have been widely used for power transmission and distribution line inspection. Current drone inspection methods mainly include manual inspection and autonomous inspection, with autonomous inspection gradually becoming the mainstream due to its high efficiency and safety.

[0003] Existing autonomous inspection methods primarily involve first acquiring images of inspection points and camera position and attitude data, calculating the 3D coordinates of the inspection points based on camera imaging principles, then determining the cruise point information through preset positional constraints between the inspection points and the cruise points. Subsequently, image processing is used to adjust the position of the inspection points in the images and the camera pose to optimize the cruise points, ensuring they remain centered in the captured images, and finally generating an autonomous inspection route that includes all cruise points. However, current technologies require images from multiple perspectives to generate autonomous inspection routes, resulting in low inspection efficiency. Summary of the Invention

[0004] Therefore, it is necessary to provide a method and system for autonomous inspection of unmanned aerial vehicles (UAVs) based on prior structural knowledge to address the aforementioned technical problems.

[0005] An autonomous inspection method for unmanned aerial vehicles (UAVs) guided by prior structural knowledge, the method comprising:

[0006] S1. Obtain the knowledge template of the tower structure and extract the two-dimensional key points from the tower images collected by the UAV; the two-dimensional key points are associated with semantic identifiers, and the knowledge template of the tower structure includes the local coordinate system of the tower, the three-dimensional key points associated with semantic identifiers, the inspection viewpoints, and the observation targets of each inspection viewpoint;

[0007] S2. Based on each of the three-dimensional key points and the corresponding two-dimensional key points Minimize the objective function The first rotation matrix is ​​obtained. and the first translation parameter K represents the intrinsic parameter matrix of the camera mounted on the UAV. Let N be the coordinates of the i-th 3D keypoint, and N be the number of 3D keypoints.

[0008] S3. Based on the first rotation matrix, the first translation parameters, and the first pose matrix of the camera coordinate system relative to the world coordinate system, obtain the second pose matrix of the tower local coordinate system relative to the world coordinate system.

[0009] S4. Based on the second pose matrix, determine the inspection coordinates of each inspection viewpoint in the world coordinate system and the observation coordinates of the observed target in the world coordinate system;

[0010] S5. Determine the inspection route of the UAV based on the inspection coordinates, determine the orientation of the UAV based on the observation coordinates, and control the UAV to perform autonomous inspection through the inspection route and the orientation.

[0011] In this application, the spatial transformation relationship between the actual tower and the tower structure knowledge template, i.e., the second pose matrix, can be solved by acquiring only a single tower image, without relying on multi-view image reconstruction. This significantly reduces the number of image acquisitions and acquisition time, lowers the complexity of coordinate calculations, improves the efficiency and real-time performance of inspection route generation, and thus enhances the overall efficiency of UAV autonomous inspection.

[0012] In one embodiment, step S3 includes:

[0013] Based on the first rotation matrix and the first translation parameter ,pass The third pose matrix of the tower's local coordinate system relative to the camera coordinate system is obtained. ;

[0014] Based on the third pose matrix and the first pose matrix from the camera coordinate system to the world coordinate system ,pass The second pose matrix of the tower's local coordinate system relative to the world coordinate system is obtained. The second pose matrix , The second rotation matrix represents the local coordinate system of the tower relative to the world coordinate system. This indicates the position of the origin of the local coordinate system of the tower in the world coordinate system.

[0015] In this application, based on the first rotation matrix and the first translation parameter ,pass The third pose matrix of the tower's local coordinate system relative to the camera coordinate system is obtained. Based on the third pose matrix The first pose matrix from the camera coordinate system to the world coordinate system ,pass The second pose matrix of the tower's local coordinate system relative to the world coordinate system is obtained. This allows us to use the second pose matrix to convert the coordinates in the local coordinate system of the tower into coordinates in the world coordinate system.

[0016] In one embodiment, step S4 includes:

[0017] Based on the second pose matrix and the coordinates of each inspection viewpoint in the local coordinate system of the tower. ,pass The inspection coordinates of each inspection viewpoint in the world coordinate system are obtained. ;

[0018] Based on the second pose matrix and the coordinates of each of the observed targets in the local coordinate system of the tower. ,pass The observation coordinates of each of the observed targets in the world coordinate system are obtained. .

[0019] In this application, the second pose matrix and the coordinates of each inspection viewpoint in the local coordinate system of the tower are used. ,pass Obtain the inspection coordinates of each inspection viewpoint in the world coordinate system. Based on the second pose matrix and the coordinates of each observed target in the local coordinate system of the tower. ,pass Obtain the observation coordinates of each target in the world coordinate system. This allows for spatial alignment between the knowledge template of tower structures and the real-world scenario, and provides a unified spatial reference for subsequent autonomous inspections.

[0020] In one embodiment, the tower structure knowledge template further includes the inspection targets and shooting requirements corresponding to each inspection viewpoint, and the autonomous inspection of the UAV by controlling the inspection route and orientation includes:

[0021] The drone is controlled to photograph the target inspection point according to the shooting requirements and orientation corresponding to the target inspection point, thereby obtaining an inspection image; the target inspection point is one of the inspection points.

[0022] The inspection object in the inspection image is identified by the target detection model, and the detection box information of each inspection object is output; the detection box information includes the center coordinates of the detection box;

[0023] Based on the second pose matrix, through The coordinates of each inspection target in the local coordinate system of the tower are... Convert to the first inspection coordinates in the world coordinate system ;

[0024] The fourth pose matrix based on the world coordinate system to the target camera coordinate system ,pass The first inspection coordinates are converted into the second inspection coordinates in the target camera coordinate system. The target camera coordinate system is a three-dimensional coordinate system established with the camera optical center as the origin when acquiring the inspection images;

[0025] Based on the aforementioned intrinsic parameter matrix, through The theoretical two-dimensional projection position of the inspection target in the inspection image is obtained. ;

[0026] Calculate the first distance between the theoretical two-dimensional projection position of the inspection target and the center of the detection frame;

[0027] Based on the first distance, the target detection frame of the inspection target is determined from the detection frame.

[0028] In this application, the first distance between the theoretical two-dimensional projection position of the inspection target and the center of each detection box is calculated. Based on the first distance, the target detection box of the inspection target is determined from the detection box. This can complete the association between the inspection target and the detection box and solve the problem that it is difficult to distinguish multiple inspection targets with similar appearances by target detection alone.

[0029] In one embodiment, the number of inspection targets and detection boxes in the inspection image is multiple, and the detection box information further includes confidence levels. Determining the target detection box of the inspection target from the detection boxes based on the first distance includes:

[0030] Normalize each of the first distances to obtain the normalized distance between the theoretical two-dimensional projection position of each inspection target and the center coordinates of each detection frame. ;

[0031] pass Calculate the comprehensive cost between each of the theoretical two-dimensional projection positions and the center of each of the detection frames. ; This is the distance weighting coefficient. Let k be the area of ​​the k-th detection box. Let j be the theoretical area of ​​the j-th inspection target. These are the scale weighting coefficients. The confidence weighting coefficient is... Let k be the confidence level of the k-th detection box;

[0032] Based on the aforementioned comprehensive cost, minimize To obtain the related variables The specific value;

[0033] Based on the specific values ​​of the associated variables, the target detection box of each of the inspection targets is determined; This indicates that the k-th detection box is the target detection box for the j-th inspection target.

[0034] In this application, by minimizing based on comprehensive cost... To obtain the related variables Based on the specific values ​​of the associated variables, the target detection boxes of each inspection target are determined. This enables the association between inspection targets and detection boxes, solving the problem that multiple inspection targets with similar appearances are difficult to distinguish by target detection alone.

[0035] In one embodiment, the detection box information further includes the detection box width, detection box height, and confidence level, and the method further includes:

[0036] Calculate the second distance between the center of the target detection box and the center of the inspected image; through Calculate the area ratio of the inspected target to which the target detection box belongs in the inspected image. ;pass Calculate the height percentage of the main tower body in the inspection image. ;w n h is the width of the target detection box. n Let W be the height of the target detection box, W be the width of the inspection image, and H be the height of the inspection image. The height of the main tower body in the inspection image;

[0037] Based on the area proportion, confidence level, height proportion, and second distance corresponding to the target detection box, through The comprehensive score of the inspection image is obtained; As the center deviation weight, For scale weights, As the confidence level weight, Main body weight; It is the maximum value in the second distance;

[0038] When the overall score is less than a preset threshold, the coordinates of the target inspection viewpoint and / or the orientation of the UAV at the target inspection viewpoint are adjusted so that the overall score of the adjusted inspection image is greater than or equal to the preset threshold.

[0039] In this application, when the overall score is less than a preset threshold, the coordinates of the target inspection viewpoint and / or the orientation of the UAV at the target inspection viewpoint are adjusted so that the overall score of the adjusted inspection image is greater than or equal to the preset threshold. This avoids using invalid images for defect detection and reduces the error rate of the detection results.

[0040] In one embodiment, when the overall score is less than a preset threshold, the orientation of the UAV at the target inspection viewpoint is adjusted so that the overall score of the adjusted inspection image is greater than or equal to the preset threshold, including:

[0041] When the overall score is less than a preset threshold, the center coordinates of the target detection box are used as the basis for the determination. The height H and width W of the inspection image are determined by... Calculate width deviation ,pass Calculate height deviation ;

[0042] Based on the width deviation The width of the inspection image and the preset yaw angle adjustment coefficient. ,pass Determine the yaw angle adjustment amount Based on the height deviation The height of the inspection image and the preset pitch angle adjustment coefficient. ,pass Determine the pitch angle adjustment amount ;

[0043] Based on the yaw angle adjustment and the pitch angle adjustment, the adjusted orientation of the UAV after adjusting at the target inspection viewpoint is determined;

[0044] The drone is adjusted to the adjusted orientation at the target inspection viewpoint, and the drone is controlled to take pictures of the inspection target at the target inspection viewpoint according to the corresponding shooting requirements, so as to obtain an adjusted inspection image with a comprehensive score greater than or equal to the preset threshold.

[0045] In this application, the adjustment orientation of the UAV after adjustment at the target inspection viewpoint is determined based on the yaw angle adjustment amount and the pitch angle adjustment amount. The orientation of the UAV at the target inspection viewpoint is adjusted to the adjustment orientation, and the UAV is controlled to take pictures of the inspection target at the target inspection viewpoint according to the corresponding shooting requirements, so as to obtain the adjustment inspection image with a comprehensive score greater than or equal to the preset threshold. This can avoid using invalid images for defect detection and reduce the error rate of the detection results.

[0046] In one embodiment, when the overall score is less than a preset threshold, the coordinates of the target inspection viewpoint are adjusted so that the overall score of the adjusted inspection image is greater than or equal to the preset threshold, including:

[0047] When the overall score is less than a preset threshold, the inspection coordinates are based on the target inspection viewpoint. The observation coordinates of the observed target corresponding to the target inspection viewpoint ,pass Calculate the unit direction vector from the target inspection viewpoint to the observed target. ;

[0048] Based on the unit direction vector ,pass Obtain the adjusted coordinates of the target inspection viewpoint ; The signed step size is the distance moved along the direction of observation.

[0049] The drone is controlled to capture images of the corresponding inspection target from the adjusted target inspection viewpoint, so as to obtain an adjusted inspection image with a comprehensive score greater than or equal to the preset threshold.

[0050] In this application, based on the unit direction vector... ,pass Obtain the adjusted coordinates of the target inspection viewpoint The system controls the drone to take pictures of the corresponding inspection target from the adjusted target inspection viewpoint, so as to obtain an adjusted inspection image with a comprehensive score greater than or equal to a preset threshold. This can avoid using invalid inspection images for defect detection and reduce the error rate of the detection results.

[0051] In one embodiment, determining the orientation of the drone based on the observed coordinates includes:

[0052] Based on the observation coordinates of the observed target corresponding to the inspection viewpoint and the inspection coordinates of the inspection viewpoint, through The direction vector pointing from the inspection viewpoint to the corresponding observed target is obtained;

[0053] Based on the components of the direction vector on the X-axis of the world coordinate system Components along the Y-axis ,pass Calculate the yaw angle of the inspection viewpoint ;

[0054] Based on the components of the direction vector on the X-axis of the world coordinate system Y-axis components and the components of Z ,pass Calculate the pitch angle ;

[0055] Based on the yaw angle and pitch angle corresponding to each of the inspection viewpoints, the orientation of the UAV at each of the inspection viewpoints is determined.

[0056] In this application, based on the observation coordinates of the target corresponding to the inspection viewpoint and the inspection coordinates of the inspection viewpoint, through... Obtain the direction vector pointing from the inspection viewpoint to the corresponding observed target, based on the components of the direction vector along the X-axis of the world coordinate system. Components along the Y-axis ,pass Calculate the yaw angle of the inspection viewpoint Based on the components of the direction vector along the X-axis in the world coordinate system Y-axis components and the components of Z ,pass Calculate pitch angle Based on the yaw and pitch angles corresponding to each inspection viewpoint, the orientation of the UAV at each inspection viewpoint is determined. In this way, the UAV does not need to directly understand semantic information such as "left crossarm insulator" and "tower head". Instead, it can directly calculate the direction vector based on the geometric position of the observation target in the world coordinate system predefined in the tower structure knowledge template. This provides attitude control parameters (pitch angle and yaw angle) for subsequent autonomous inspection flight and image acquisition, and automatically completes the orientation control of the nose and gimbal.

[0057] An autonomous inspection system for unmanned aerial vehicles (UAVs) guided by prior structural knowledge, used to execute the above method, the system comprising:

[0058] The template acquisition module is used to acquire knowledge templates of tower structures and extract two-dimensional key points from tower images collected by UAVs. The two-dimensional key points are associated with semantic identifiers. The knowledge template of tower structure includes a local coordinate system of the tower, three-dimensional key points associated with semantic identifiers, inspection viewpoints, and observation targets of each inspection viewpoint.

[0059] The optimization module is used to optimize each of the three-dimensional key points and the corresponding two-dimensional key points. Minimize the objective function The first rotation matrix is ​​obtained. and the first translation parameter K represents the intrinsic parameter matrix of the camera mounted on the UAV. Let N be the coordinates of the i-th 3D keypoint, and N be the number of 3D keypoints.

[0060] The pose matrix determination module is used to obtain the second pose matrix of the tower local coordinate system relative to the world coordinate system based on the first rotation matrix, the first translation parameters and the first pose matrix of the camera coordinate system relative to the world coordinate system.

[0061] The coordinate transformation module is used to determine the inspection coordinates of each inspection viewpoint in the world coordinate system and the observation coordinates of the observed target in the world coordinate system based on the second pose matrix.

[0062] The inspection module is used to determine the inspection route of the UAV based on the inspection coordinates, determine the orientation of the UAV based on the observation coordinates, and control the UAV to perform autonomous inspections through the inspection route and the orientation.

[0063] The beneficial effect of the aforementioned UAV autonomous inspection method guided by prior structural knowledge is that it can solve the spatial transformation relationship between the real tower and the tower structure knowledge template, i.e., the second pose matrix, by acquiring only a single tower image, without relying on multi-view image reconstruction. This significantly reduces the number of image acquisitions and acquisition time, lowers the complexity of coordinate calculations, improves the efficiency and real-time performance of inspection route generation, and thus enhances the overall efficiency of UAV autonomous inspection. Attached Figure Description

[0064] Figure 1 This is an application environment diagram of an UAV autonomous inspection method guided by prior structural knowledge in one embodiment;

[0065] Figure 2 This is a flowchart illustrating an autonomous inspection method for unmanned aerial vehicles (UAVs) guided by prior structural knowledge in one embodiment.

[0066] Figure 3 This is a schematic diagram of a knowledge template for a tower structure in one embodiment;

[0067] Figure 4 This is a schematic diagram illustrating the estimation of the second pose matrix in one embodiment;

[0068] Figure 5 This is a schematic diagram illustrating the alignment of a knowledge template of a tower structure with a real-world scene in one embodiment.

[0069] Figure 6 This is a schematic diagram of the autonomous inspection state machine of a UAV in one embodiment;

[0070] Figure 7 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0071] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0072] The UAV autonomous inspection method based on prior structural knowledge provided in this application can be applied to, for example... Figure 1 In the application environment shown, terminal 102 interacts with server 104 via a wired / wireless channel. The data storage system can store the data that server 104 needs to process. The server acquires a knowledge template of the tower structure and extracts two-dimensional key points from the tower images collected by the drone; the two-dimensional key points are associated with semantic identifiers. The knowledge template of the tower structure includes the tower's local coordinate system, three-dimensional key points associated with semantic identifiers, inspection viewpoints, and the observation targets of each inspection viewpoint; the server, based on each three-dimensional key point and its corresponding two-dimensional key point... Minimize the objective function The first rotation matrix is ​​obtained. and the first translation parameter K represents the intrinsic parameter matrix of the camera mounted on the drone. Let N be the coordinates of the i-th 3D keypoint and N be the number of 3D keypoints. Based on the first rotation matrix, the first translation parameters, and the first pose matrix of the camera coordinate system relative to the world coordinate system, the server obtains the second pose matrix of the tower's local coordinate system relative to the world coordinate system. Based on the second pose matrix, the server determines the inspection coordinates of each inspection viewpoint in the world coordinate system and the observation coordinates of the observed target in the world coordinate system. The server determines the UAV's inspection route based on the inspection coordinates, determines the UAV's orientation based on the observation coordinates, and controls the UAV to perform autonomous inspections through the inspection route and orientation. The terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, etc. The server 104 can be a single server, a server cluster consisting of multiple servers, or a cloud computing center consisting of multiple servers.

[0073] In one embodiment, such as Figure 2 As shown, a method for autonomous inspection of unmanned aerial vehicles (UAVs) based on prior structural knowledge is presented, and this method is applied to... Figure 1 Taking server 104 as an example, the following steps are included:

[0074] S1. Obtain the knowledge template of the tower structure and extract the two-dimensional key points from the tower images collected by the UAV; the two-dimensional key points are associated with semantic tags, and the knowledge template of the tower structure includes the local coordinate system of the tower, the three-dimensional key points associated with semantic tags, the inspection viewpoints, and the observation targets of each inspection viewpoint;

[0075] The tower structure knowledge template is a priori structural knowledge. It is derived from tower structure information extracted from historical inspection data, old scene 3D point clouds, or other structured spatial data. A schematic diagram of the tower structure knowledge template is shown below. Figure 3 As shown.

[0076] Tower structure knowledge templates can be stored in JSON, XML, or database tables. Each inspection viewpoint in the tower structure knowledge template is not a simple waypoint, but an inspection task with execution semantics and quality constraints. This design allows the same tower structure knowledge template to be migrated to similar towers in different locations, simply by completing coordinate alignment through pose estimation.

[0077] The pole image is an image of the pole to be inspected. The drone flies to a location where the pole can be fully observed, and the image acquired at that location is the pole image. This location can be determined manually, by coarse GPS (Global Positioning System) positioning, existing flight paths, or other navigation methods. The pole image acquired by the drone at this location includes the overall structure of the pole, especially key structural points used for pose estimation. Only one pole image is needed to establish a two-dimensional to three-dimensional correspondence.

[0078] Two-dimensional keypoints can be extracted directly using yolov8n-pose. Alternatively, you can first use yolov8n-seg to extract the key regions of the tower image, and then use yolov8n-pose to extract the two-dimensional keypoints within those regions. Specifically, the two-dimensional keypoints are the two-dimensional pixel coordinates of the tower's keypoints within the tower image.

[0079] The local coordinate system of the tower has the center point of the tower top as the origin, the rightward direction of the crossarm as the X-axis, the inspection direction or line direction as the Y-axis, and the vertically upward direction as the Z-axis. The expression is , The center point of the top of the tower, Indicates the right direction of the crossbar. Indicates the direction of inspection or route. This indicates the vertical upward direction. Using the local coordinate system of the tower, key points of the tower, insulator strings, crossarms, tower body, and inspection viewpoints can all be uniformly expressed within the same coordinate system.

[0080] Specifically, a 3D keypoint refers to the 3D coordinates of a tower's key points within the tower's local coordinate system. A set of multiple 3D keypoints is... , , Let i represent the set of key points, where i represents the index number of the key point, and N represents the number of 3D key points. This is the semantic identifier associated with the i-th 3D keypoint. The semantic identifier is a category label or attribute description of the 3D keypoint. For example, "insulator".

[0081] The inspection viewpoint is the location on the tower that the drone can photograph during the inspection process. The observation target determines where the camera on the drone is looking, and it is represented as a three-dimensional geometric point in the local coordinate system of the tower. The observation target can be used to adjust the direction of the drone's nose and the camera gimbal.

[0082] Furthermore, the inspection viewpoint is a viewpoint that allows for a complete observation of the main body of the tower and key areas.

[0083] Furthermore, when the number of 2D keypoints detected in the tower image is less than the number of 3D keypoints, the tower image is re-acquired or a downgrade mode with known tower top coordinates is enabled. When the number of 2D keypoints is greater than the number of 3D keypoints but outliers exist, outliers are removed using a reprojection error threshold and interior point filtering mechanism, and the first rotation matrix and first translation parameters are determined using the 2D keypoints after removing outliers.

[0084] S2, Based on each 3D keypoint and its corresponding 2D keypoint Minimize the objective function The first rotation matrix is ​​obtained. and the first translation parameter K represents the intrinsic parameter matrix of the camera mounted on the drone. Let N be the coordinates of the i-th 3D keypoint, and N be the number of 3D keypoints.

[0085] Here, the corresponding 2D keypoint refers to the 2D keypoint corresponding to the 3D keypoint. The 3D keypoint and its corresponding 2D keypoint have the same semantic identifier.

[0086] Intrinsic parameter matrix f x f represents the focal length in the horizontal direction. y The focal length in the vertical direction is represented by (c x ,c y () represents the coordinates of the principal point.

[0087] 2D key points It is the two-dimensional keypoint corresponding to the i-th three-dimensional keypoint. Let be the coordinates of the i-th 3D key point in the local coordinate system of the tower.

[0088] is the theoretical two-dimensional coordinate of the i-th 3D keypoint in the tower image after pose transformation, and π(∙) is the pinhole projection function.

[0089] It is a reprojection error.

[0090] The first rotation matrix and the first translation parameter constitute the pose transformation relationship from the tower's local coordinate system to the camera's coordinate system. Using the first rotation matrix and the first translation parameter, coordinates in the tower's local coordinate system can be converted to coordinates in the camera's coordinate system.

[0091] In practical applications, keypoint detection may suffer from false detections, missed detections, and occlusion. To improve robustness, a random sampling consensus algorithm can be used to eliminate false matching points, and nonlinear optimization can be performed on the inlier set. After eliminating false matching points using the random sampling consensus algorithm and further optimizing the pose parameters using a nonlinear optimization method, the optimal pose matrix of the tower's local coordinate system relative to the camera coordinate system can be obtained. The rotation matrix in the optimal pose matrix is ​​then used as the first rotation matrix, and the translation parameters are used as the first translation parameters.

[0092] S3. Based on the first rotation matrix, the first translation parameters, and the first pose matrix of the camera coordinate system relative to the world coordinate system, the second pose matrix of the tower local coordinate system relative to the world coordinate system is obtained.

[0093] Among them, the camera coordinate system is a three-dimensional coordinate system established with the camera optical center as the origin when acquiring tower images, and the pixel position in the image is represented by the two-dimensional image pixel coordinate system.

[0094] The first pose matrix of the camera coordinate system relative to the world coordinate system includes the rotation matrix and translation parameters. This first pose matrix can be determined based on the UAV's positioning and attitude information at the time of tower image acquisition, combined with camera mounting extrinsic parameters and gimbal attitude information. The first pose matrix is ​​used to transform coordinates in the camera coordinate system to coordinates in the world coordinate system.

[0095] The second pose matrix includes a rotation matrix and translation parameters. It is used to transform the coordinates in the tower's local coordinate system to coordinates in the world coordinate system.

[0096] S4. Based on the second pose matrix, determine the inspection coordinates of each inspection viewpoint in the world coordinate system and the observation coordinates of the observed target in the world coordinate system;

[0097] S5. Determine the inspection route of the UAV based on the inspection coordinates, determine the orientation of the UAV based on the observation coordinates, and control the UAV to perform autonomous inspections through the inspection route and orientation.

[0098] Multiple inspection coordinates are connected to form the inspection route of the drone inspection.

[0099] Each inspection viewpoint corresponds to a specific orientation. The orientation of an inspection viewpoint includes the pitch and yaw angles of the UAV at that viewpoint.

[0100] The aforementioned UAV autonomous inspection method guided by prior structural knowledge solves the spatial transformation relationship (i.e., the second pose matrix) between the real tower and the tower structure knowledge template by acquiring only a single tower image, without relying on multi-view image reconstruction. This significantly reduces the number of image acquisitions and acquisition time, lowers the complexity of coordinate calculations, improves the efficiency and real-time performance of inspection route generation, and ultimately enhances the overall efficiency of UAV autonomous inspection.

[0101] In one embodiment, step S3 includes:

[0102] Based on the first rotation matrix and the first translation parameter ,pass The third pose matrix of the tower's local coordinate system relative to the camera coordinate system is obtained. ;

[0103] Based on the third pose matrix The first pose matrix from the camera coordinate system to the world coordinate system ,pass The second pose matrix of the tower's local coordinate system relative to the world coordinate system is obtained. Second pose matrix , This represents the second rotation matrix of the tower's local coordinate system relative to the world coordinate system. This indicates the position of the origin of the local coordinate system of the tower in the world coordinate system.

[0104] The third pose matrix is ​​used to convert the coordinates in the local coordinate system of the tower to the coordinates in the camera coordinate system.

[0105] It is also the second translation parameter of the tower's local coordinate system relative to the world coordinate system.

[0106] A schematic diagram of the estimation of the second pose matrix is ​​shown below. Figure 4 As shown in the figure. Among them, (u1,v1), (u2,v2), (u3,v3), (u4,v4), and (u5,v5) are all coordinates of two-dimensional key points.

[0107] In this embodiment, based on the first rotation matrix and the first translation parameter ,pass The third pose matrix of the tower's local coordinate system relative to the camera coordinate system is obtained. Based on the third pose matrix The first pose matrix from the camera coordinate system to the world coordinate system ,pass The second pose matrix of the tower's local coordinate system relative to the world coordinate system is obtained. This allows us to use the second pose matrix to convert the coordinates in the local coordinate system of the tower into coordinates in the world coordinate system.

[0108] In one embodiment, step S4 includes:

[0109] Based on the second pose matrix and the coordinates of each inspection viewpoint in the local coordinate system of the tower. ,pass The inspection coordinates of each inspection viewpoint in the world coordinate system are obtained. ;

[0110] Based on the second pose matrix and the coordinates of each observed target in the local coordinate system of the tower. ,pass The observation coordinates of each target in the world coordinate system are obtained. .

[0111] in, Indicates the first One inspection viewpoint, =1,2,…,L, where L is the total number of inspection viewpoints.

[0112] A diagram illustrating the alignment of the knowledge template for tower structures with real-world scenarios is shown below. Figure 5 As shown.

[0113] Furthermore, the tower structure knowledge template also includes flight actions corresponding to each inspection viewpoint. Flight actions include taking photos or flying. Further, through... Only the inspection coordinates of the inspection viewpoint in the world coordinate system are determined when the flight action is to take pictures.

[0114] In this embodiment, the second pose matrix and the coordinates of each inspection viewpoint in the local coordinate system of the tower are used as the basis. ,pass Obtain the inspection coordinates of each inspection viewpoint in the world coordinate system. Based on the second pose matrix and the coordinates of each observed target in the local coordinate system of the tower. ,pass Obtain the observation coordinates of each target in the world coordinate system. This allows for spatial alignment between the knowledge template of tower structures and the real-world scenario, and provides a unified spatial reference for subsequent autonomous inspections.

[0115] In one embodiment, the tower structure knowledge template also includes the inspection targets and shooting requirements corresponding to each inspection viewpoint, and the autonomous inspection of the drone by controlling the inspection route and orientation includes:

[0116] Control the drone to inspect the target viewpoint, and take pictures of the target according to the shooting requirements and orientation corresponding to the target inspection viewpoint to obtain inspection images; the target inspection viewpoint is one of the inspection viewpoints;

[0117] The target detection model identifies the inspection objects in the inspection image and outputs the detection box information for each inspection object; the detection box information includes the center coordinates of the detection box.

[0118] Based on the second pose matrix, through The coordinates of each inspection target in the local coordinate system of the tower are... Convert to the first inspection coordinates in the world coordinate system ;

[0119] The fourth pose matrix based on the world coordinate system to the target camera coordinate system ,pass The first inspection coordinates are converted to the second inspection coordinates in the target camera coordinate system. The target camera coordinate system is a three-dimensional coordinate system established with the camera optical center as the origin when acquiring inspection images;

[0120] Based on the intrinsic parameter matrix, through Obtain the theoretical two-dimensional projection position of the inspection target in the inspection image. ;

[0121] Calculate the first distance between the theoretical two-dimensional projection position of the inspection target and the center of the detection frame;

[0122] Based on the first distance, the target detection box of the inspection target is determined from the detection box.

[0123] The shooting requirements and orientation correspond to the target inspection viewpoint. Each inspection viewpoint serves as the target inspection viewpoint, resulting in inspection images for each viewpoint, used to inspect key points of the pole to be inspected. The inspection images are taken from the pole to be inspected.

[0124] An inspection viewpoint may correspond to one or more inspection targets. Each inspection target also has a unique logical number and semantic name.

[0125] The inspection items include, but are not limited to, insulator strings, crossarms, tower bodies, hardware, and conductor connection components.

[0126] The detection frame information of each inspection object is represented as follows: B k This represents the information of the k-th detection box. It is the x-coordinate of the center of the k-th detection box. It is the ordinate of the center of the k-th detection box. It is the width of the k-th detection box. It is the height of the k-th detection box. is the confidence score of the k-th bounding box, k=1,2...Q; Q is the total number of bounding boxes in the inspected image.

[0127] Since multiple inspection targets with similar appearances, such as multiple insulator strings, often exist on the same tower, relying solely on the output of the target detection model is insufficient to determine which inspection target corresponds to a given detection box. Therefore, this application further projects each inspection target onto the inspection image plane and determines the target detection box of the inspection target based on a first distance between the theoretical two-dimensional projection position of the inspection target and the center coordinates of the detection box. The theoretical two-dimensional projection position reflects the position in the fourth pose matrix. Given that both the second pose matrix and the second pose matrix are correct, the theoretical position of the j-th inspection target in the inspection image.

[0128] coordinate and the first inspection coordinates All coordinates are three-dimensional. This is the homogeneous coordinate form of the first inspection coordinate.

[0129] Fourth pose matrix This includes the rotation matrix and translation parameters. The fourth pose matrix can be obtained through traditional calibration methods. The fourth pose matrix is ​​used to transform coordinates in the world coordinate system to coordinates in the target camera coordinate system.

[0130] The fourth pose matrix is ​​inconsistent for different inspection viewpoints.

[0131] The formula for calculating the first distance is: , Let the center coordinates of the k-th detection box be... Let be the first distance between the center of the j-th inspected target and the center of the k-th detection box. The smaller the first distance, the closer the detection box and the inspected target are in spatial position in the inspected image, and the greater the probability that they belong to the same target.

[0132] Based on the first distance, the target detection box of the inspected target is determined from the detection box by: selecting the detection box corresponding to the smallest first distance as a candidate detection box based on the first distance between the inspected target and the center of each detection box in the inspected image; when the first distance between the center of the inspected target and the candidate detection box is less than a preset distance threshold, the candidate detection box is determined as the target detection box of the inspected target. Further, when the first distance between the center of the inspected target and the candidate detection box is greater than or equal to the preset distance threshold, it is determined that the inspected target was not detected in the inspected image, or that the inspected target in the inspected image is occluded, missed, or has a pose deviation.

[0133] In this embodiment, by calculating the first distance between the theoretical two-dimensional projection position of the inspection target and the center of each detection frame, the target detection frame of the inspection target is determined from the detection frames based on the first distance. This can complete the association between the inspection target and the detection frame, and solve the problem that it is difficult to distinguish multiple inspection targets with similar appearances by target detection alone.

[0134] In one embodiment, the number of inspection targets and detection boxes in the inspection image is multiple, and the detection box information also includes confidence scores. Based on a first distance, the target detection boxes for the inspection targets are determined from the detection boxes as follows:

[0135] Normalize each first distance to obtain the normalized distance between the theoretical two-dimensional projection position of each inspection target and the center coordinates of each detection frame. ;

[0136] pass Calculate the comprehensive cost between each theoretical two-dimensional projection position and the center of each detection box. ; This is the distance weighting coefficient. Let k be the area of ​​the k-th detection box. Let j be the theoretical area of ​​the j-th inspection target. These are the scale weighting coefficients. The confidence weighting coefficient is... Let k be the confidence level of the k-th detection box;

[0137] Based on the overall cost, minimize To obtain the related variables The specific value;

[0138] Based on the specific values ​​of the associated variables, determine the target detection box for each inspection target; This indicates that the k-th detection box is the target detection box for the j-th inspection target.

[0139] Among them, normalized distance , The normalized distance is the distance between the center of the j-th inspection target and the center of the k-th detection frame.

[0140] Used to measure the spatial proximity between the inspection target and the target detection box of the inspection target; Used to measure the consistency between the detection frame size and the expected size at this inspection viewpoint; Used to suppress low-confidence detections. The distance weight coefficient, scale weight coefficient, and confidence weight coefficient are preset values.

[0141] The theoretical area of ​​an inspected target is the pixel area that the target "should" occupy in the inspected image. For example, theoretical area = (actual width of the inspected target × actual height of the inspected target × focal length of the camera) / squared depth distance from the inspected target to the camera.

[0142] Related variables In this context, j represents the j-th inspection target, and k represents the k-th detection box. The associated variables can form an association variable matrix, where rows represent inspection targets and columns represent detection boxes. The k-th detection box is the target detection box of the j-th inspection target; that is, the k-th detection box is associated with the j-th inspection target.

[0143] Based on the overall cost, minimize To obtain the related variables The specific values ​​include: constructing constraints, and minimizing the overall cost based on the constraints and the overall cost. To obtain the related variables The specific value; the constraints include that each row and each column of the correlation variable matrix contains at most one element that is 1.

[0144] In this embodiment, by minimizing based on comprehensive cost... To obtain the related variables Based on the specific values ​​of the associated variables, the target detection boxes for each inspection target are determined. This enables the association between the inspection target and the detection box, solving the problem that multiple inspection targets with similar appearances are difficult to distinguish by target detection alone.

[0145] In one embodiment, the detection box information further includes the detection box width and the detection box height, and the method further includes:

[0146] Calculate the second distance between the center of the target detection box and the center of the inspected image; through Calculate the area ratio of the inspected target to which the target detection box belongs in the inspected image. ;pass Calculate the height percentage of the main tower body in the inspection image. ;w n h is the width of the target detection box. n Let W be the height of the target detection bounding box, W be the width of the inspection image, and H be the height of the inspection image. The height of the main tower in the inspection image;

[0147] Based on the area proportion, confidence level, height proportion, and second distance of the target detection box, through... The overall score of the inspection images is obtained; As the center deviation weight, For scale weights, As the confidence level weight, Main body weight; It is the maximum value in the second distance;

[0148] When the overall score is less than the preset threshold, adjust the coordinates of the target inspection viewpoint and / or the orientation of the UAV at the target inspection viewpoint so that the overall score of the adjusted inspection image is greater than or equal to the preset threshold.

[0149] The coordinates of the center of the inspection image are: Second distance The calculation formula is . The coordinates are the center of the nth target detection box. The smaller the value, the closer the inspection target is to the center of the inspection image.

[0150] The main body of the pole / tower refers to the main structural part of the pole / tower to be inspected in the inspection image. It is the pixel height of the main body of the tower.

[0151] The center deviation weight, scale weight, confidence weight, and subject proportion weight are all preset data. The weights for center deviation, scale, confidence level, and subject proportion can vary across different inspection viewpoints. For example, the subject proportion weight can be appropriately increased in a panoramic inspection viewpoint, while the scale weight can be increased in a close-up insulator inspection viewpoint.

[0152] The calculation steps for the overall score of the adjusted inspection images are the same as those for the overall score of the inspection images themselves. Both the adjusted inspection images and the inspection images can be used for subsequent downstream tasks such as defect detection and condition analysis.

[0153] The overall score is essentially a quality evaluation result. Through this quality evaluation, this application can avoid low-quality inspection results caused by factors such as the target deviating from the image center, the target being too small, insufficient proportion of the main subject in the image, or low detection confidence, thereby ensuring the reliability of subsequent defect identification, state analysis, and inspection report generation.

[0154] Furthermore, if there are multiple inspection targets in the inspection image, the sub-scores of each inspection target in the inspection image are calculated based on the area ratio, confidence level, height ratio, and second distance corresponding to the target detection box of each inspection target, and the average of the sub-scores is used as the comprehensive score of the inspection image.

[0155] In this embodiment, when the overall score is less than a preset threshold, the coordinates of the target inspection viewpoint and / or the orientation of the UAV at the target inspection viewpoint are adjusted so that the overall score of the adjusted inspection image is greater than or equal to the preset threshold. This avoids using invalid images for defect detection and reduces the error rate of the detection results.

[0156] In one embodiment, when the overall score is greater than or equal to a preset threshold and the inspection image is an image captured according to the corresponding shooting requirements, the inspection image is determined to meet the inspection requirements. Simultaneously, the inspection image is saved, and the drone is controlled to fly to the next inspection viewpoint. The inspection requirements are that the overall score is greater than or equal to the preset threshold and the inspection image is an image captured according to the corresponding shooting requirements. Further, for a panoramic inspection viewpoint, the inspection requirements also include... For a top-down viewpoint, inspection requirements also include... .

[0157] In one embodiment, when the overall score is less than a preset threshold, the orientation of the drone at the target inspection viewpoint is adjusted so that the overall score of the adjusted inspection image is greater than or equal to the preset threshold, including:

[0158] When the overall score is less than a preset threshold, the center coordinates of the target detection box are used as the basis for determining the target detection score. The height H and width W of the inspection image are determined by... Calculate width deviation ,pass Calculate height deviation ;

[0159] Based on width deviation The width of the inspection image and the preset yaw angle adjustment coefficient. ,pass Determine the yaw angle adjustment amount Based on height deviation The height of the inspection image and the preset pitch angle adjustment coefficient ,pass Determine the pitch angle adjustment amount ;

[0160] Based on the yaw angle adjustment and pitch angle adjustment, determine the UAV's adjusted orientation after adjusting the target inspection viewpoint;

[0161] Adjust the drone's orientation at the target inspection viewpoint to an adjusted orientation, and control the drone to take pictures of the inspection target at the target inspection viewpoint according to the corresponding shooting requirements, so as to obtain an adjusted inspection image with a comprehensive score greater than or equal to the preset threshold.

[0162] The adjustment of orientation includes the adjusted pitch angle. and the adjusted yaw angle , The pitch angle before adjustment. This is the yaw angle before adjustment.

[0163] Furthermore, if the overall score of the adjusted inspection image is still less than the preset threshold, the coordinates of the target inspection viewpoint and / or the orientation of the UAV at the target inspection viewpoint are repeatedly adjusted until the overall score of the adjusted inspection image is greater than or equal to the preset threshold.

[0164] Furthermore, when | When the impact is large, prioritize adjusting the drone's nose direction or moving it left or right; when... When the angle is large, prioritize adjusting the gimbal's pitch angle or make minor adjustments to its vertical position.

[0165] Furthermore, if there are multiple inspection targets, the height deviation and width deviation corresponding to the target detection frame of each inspection target are determined, and the average of each height deviation is taken as the final height deviation. The pitch angle adjustment is determined by the final height deviation, and the yaw angle adjustment is determined by the average of each width deviation.

[0166] In this embodiment, based on the yaw angle adjustment and pitch angle adjustment, the adjustment orientation of the UAV after adjustment at the target inspection viewpoint is determined. The orientation of the UAV at the target inspection viewpoint is adjusted to the adjustment orientation, and the UAV is controlled to take pictures of the inspection target at the target inspection viewpoint according to the corresponding shooting requirements, so as to obtain the adjustment inspection image with a comprehensive score greater than or equal to the preset threshold. This can avoid using invalid images for defect detection and reduce the error rate of the detection results.

[0167] In one embodiment, when the overall score is less than a preset threshold, the coordinates of the target inspection viewpoint are adjusted so that the overall score of the adjusted inspection image is greater than or equal to the preset threshold, including:

[0168] When the overall score is less than a preset threshold, the inspection coordinates are based on the target inspection viewpoint. The observation coordinates of the target corresponding to the target inspection viewpoint ,pass Calculate the unit direction vector from the target inspection viewpoint to the observed target. ;

[0169] Based on unit direction vector ,pass Obtain the adjusted coordinates of the target inspection viewpoint ; The signed step size is the distance moved along the direction of observation.

[0170] Control the drone to take pictures of the corresponding inspection target from the adjusted target inspection viewpoint to obtain the adjusted inspection image, so as to obtain the adjusted inspection image with a comprehensive score greater than or equal to the preset threshold.

[0171] The corresponding inspection target refers to the inspection target corresponding to the original inspection viewpoint. When controlling the drone to capture and adjust the inspection images, the shooting requirements corresponding to the target inspection viewpoint will also be followed.

[0172] Inspection coordinates It is the inspection coordinate of the l-th inspection viewpoint. It is the observation coordinate of the target corresponding to the l-th inspection viewpoint.

[0173] Determine the drone's direction and distance of movement. Only the geometric direction is determined. For example, if the observed target remains consistently too small in the inspection image, then... This indicates that the drone travels along... Move closer to the target; if the target remains too large, take a step back. This is equivalent to the drone traveling along... Backwards. Dimensionally, and The unit is meters. The data is dimensionless.

[0174] When adjusting the coordinates of the target inspection viewpoint for the Pth time... P represents the coordinates of the target inspection viewpoint obtained after the previous adjustment, where P is an integer greater than or equal to 2.

[0175] After each fine-tuning (adjusting the coordinates of the target inspection viewpoint and / or the UAV's orientation at the target inspection viewpoint), the UAV re-acquires inspection images and performs target detection and image quality evaluation. If the overall score of the inspection image is greater than or equal to a preset threshold, and the inspection image was taken according to the corresponding shooting requirements, the inspection image is determined to meet the power inspection requirements, the image is saved, and the next inspection viewpoint is entered; otherwise, fine-tuning continues until the maximum number of fine-tuning attempts is reached, or the overall score of the adjusted inspection image is greater than or equal to a preset threshold, and the adjusted inspection image was taken according to the corresponding shooting requirements.

[0176] In this embodiment, based on the unit direction vector ,pass Obtain the adjusted coordinates of the target inspection viewpoint The system controls the drone to take pictures of the corresponding inspection target from the adjusted target inspection viewpoint, so as to obtain an adjusted inspection image with a comprehensive score greater than or equal to a preset threshold. This can avoid using invalid inspection images for defect detection and reduce the error rate of the detection results.

[0177] In one embodiment, determining the drone's orientation based on observed coordinates includes:

[0178] Based on the observation coordinates of the target corresponding to the inspection viewpoint and the inspection coordinates of the inspection viewpoint, through Obtain the direction vector from the inspection viewpoint to the corresponding observed target;

[0179] Based on the components of the direction vector along the X-axis of the world coordinate system Components along the Y-axis ,pass Calculate the yaw angle of the inspection viewpoint ;

[0180] Based on the components of the direction vector along the X-axis of the world coordinate system Y-axis components and the components of Z ,pass Calculate the pitch angle ;

[0181] Based on the yaw and pitch angles corresponding to each inspection viewpoint, the orientation of the UAV at each inspection viewpoint is determined.

[0182] The direction vector from the inspection viewpoint to the corresponding observation target refers to the direction vector from the inspection viewpoint to the observation target corresponding to that viewpoint. This direction vector contains both horizontal orientation information and height difference information.

[0183] Yaw angle It is the first Yaw angle and pitch angle of each inspection viewpoint It is the first Pitch angle of each inspection viewpoint. Yaw angle. and pitch angle This is also the first time the pitch and yaw angles have been adjusted. The yaw angle is used to control the direction of rotation of the drone's nose in the horizontal plane. The time indicates the positive direction of the UAV towards the X-axis of the world coordinate system. The pitch angle is used to control the rotation of the camera gimbal around the horizontal axis, so that the observed target is located near the center of the image.

[0184] In this embodiment, based on the observation coordinates of the target corresponding to the inspection viewpoint and the inspection coordinates of the inspection viewpoint, through... Obtain the direction vector pointing from the inspection viewpoint to the corresponding observed target, based on the components of the direction vector along the X-axis of the world coordinate system. Components along the Y-axis ,pass Calculate the yaw angle of the inspection viewpoint Based on the components of the direction vector along the X-axis in the world coordinate system Y-axis components and the components of Z ,pass Calculate pitch angle Based on the yaw and pitch angles corresponding to each inspection viewpoint, the orientation of the UAV at each inspection viewpoint is determined. In this way, the UAV does not need to directly understand semantic information such as "left crossarm insulator" and "tower head". Instead, it can directly calculate the direction vector based on the geometric position of the observation target in the world coordinate system predefined in the tower structure knowledge template. This provides attitude control parameters (pitch angle and yaw angle) for subsequent autonomous inspection flight and image acquisition, and automatically completes the orientation control of the nose and gimbal.

[0185] In one embodiment, to ensure the controllability and interpretability of complex inspection tasks, this application employs a state machine to manage the entire inspection process. The state machine clearly distinguishes between the initialization state, pose estimation state, template alignment state, inspection execution state, image quality evaluation state, visual fine-tuning state, image saving state, and inspection completion state, avoiding logical confusion between modules. WAIT represents the waiting-for-initialization state, PNP represents the pose estimation state, ALIGN represents the template alignment state, INSPECT represents the inspection execution state, VALIDATE represents the image quality evaluation state, REFINE represents the visual fine-tuning state, CAPTURE represents the image saving state, and FINISH represents the inspection completion state. Inspection State Set .

[0186] The overall state transition relationship is as follows: At each inspection viewpoint, the system first enters the INSPECT state, then the VALIDATE state. If the inspected image meets the inspection requirements, it enters the CAPTURE state; if the inspected image does not meet the inspection requirements, it enters the REFINE state and returns to the VALIDATE state after fine-tuning. After all inspection viewpoints are completed, the system enters the FINISH state. During operation, the server records the entry time, exit time, execution result, and cause of any exceptions for each state. Upon completion of the inspection, an inspection report is generated, containing the first rotation matrix, first translation parameters, inspection viewpoints, inspection images, detection box information, quality evaluation results, and the visual fine-tuning process. This report is used for subsequent review and archiving. A schematic diagram of the UAV autonomous inspection state machine is shown below. Figure 6 As shown.

[0187] This application also provides an application scenario in which the aforementioned UAV autonomous inspection method guided by prior structural knowledge is applied. Specifically, the application of the UAV autonomous inspection method guided by prior structural knowledge in this scenario is as follows:

[0188] This embodiment uses the inspection of power distribution towers as an example. The top of the tower, the ends of the upper and lower crossarms, the positions of the insulator strings, several inspection viewpoints, and the observation targets and shooting requirements for each viewpoint are pre-calibrated in a 3D modeling environment, generating a tower structure knowledge template. The tower structure knowledge template uses the top of the tower as the origin of the local coordinate system and includes several 3D key points and inspection viewpoints.

[0189] After the inspection mission begins, the drone first flies to a location where the tower to be inspected can be fully observed, either manually, via GPS coarse positioning, or by setting a preset initial waypoint, and then acquires images of the tower. The server detects the two-dimensional key points of the tower in these images and establishes a correspondence between them and the three-dimensional key points in the tower structure knowledge template. If the number of two-dimensional key points is insufficient, the system prompts the user to reacquire tower images; if the number of key points meets the requirements, pose estimation is performed to obtain the second pose matrix from the tower's local coordinate system to the world coordinate system.

[0190] Subsequently, the server maps the inspection viewpoints and observation targets in the tower structure knowledge template to the world coordinate system based on the second pose matrix, generating real-world inspection coordinates. The UAV executes the inspection tasks sequentially according to the order of the inspection viewpoints in the tower structure knowledge template. For inspection viewpoints whose flight action is to take pictures, the UAV flies to the inspection coordinates of that viewpoint, automatically faces the corresponding observation target, collects inspection images, and performs target detection, target association, and image quality evaluation. For inspection viewpoints whose flight action is to fly, the UAV only flies through as a transition waypoint and does not take pictures. For example, inspection viewpoint A is used to take pictures of the entire tower and obtain an inspection image; inspection viewpoint B is used to take pictures of the tower head and obtain another inspection image; inspection viewpoint C is used to take pictures of the left crossarm and its insulator; inspection viewpoint D is used to take pictures from above the top of the tower; inspection viewpoint E is used to take pictures of the right side or the large side crossarm insulator; and inspection viewpoint F is used to take pictures of the line passage conditions.

[0191] When an inspection image from a certain inspection viewpoint does not meet the inspection requirements (e.g., the inspection target is not centered, the target scale is too small, the confidence level is insufficient, or the subject proportion does not meet the specifications), the server triggers local visual fine-tuning. Fine-tuning is only performed near the current inspection viewpoint. After fine-tuning, the re-captured adjusted inspection image is re-inspected and evaluated until the adjusted inspection image meets the inspection requirements or the maximum number of fine-tuning times is reached.

[0192] To verify the feasibility of this application, a power distribution tower inspection scenario is constructed in a simulation environment or a real inspection environment. Specifically, the top of the tower, the left and right ends of the upper crossarm, the left and right ends of the lower crossarm, the center of the insulator string, and multiple inspection viewpoints are pre-marked in 3D modeling software, and a structural knowledge template is exported.

[0193] At the start of the inspection, the drone first flies to a position where it can observe the entire tower and acquire images of it. The server detects key points of the tower in the images, solves for the tower pose (first rotation matrix and first translation parameters), and maps the inspection viewpoints in the template to the simulated or real world coordinate system. Subsequently, the drone performs the inspection sequentially from the tower head viewpoint, left crossarm viewpoint, top top viewpoint, right crossarm viewpoint, and passageway viewpoint.

[0194] At each inspection viewpoint requiring photography, the service saves inspection images that meet the inspection requirements, images with detection box information, and each captured inspection image. It also records the number of inspection targets, second distance, area percentage, quality evaluation result, whether fine-tuning was triggered, and the number of fine-tuning attempts for that viewpoint. For inspection viewpoints where the flight action is "flying," the server only records the flight trajectory and passage time, without saving inspection images.

[0195] The feasibility study can use the following evaluation indicators: pose estimation reprojection error, inspection point arrival error after template alignment, target detection success rate, target identity association accuracy rate, image quality pass rate, average number of fine adjustments, single-base tower inspection time, and abnormal termination rate. By comparing with inspection using fixed KML (Keyhole Markup Language) files, inspection using only GPS translation templates, and inspection using pure vision target detection, the advantages of this application in terms of adaptability to unknown scenarios, consistency of inspected targets, and image quality can be verified.

[0196] In engineering deployment, the knowledge template for pole structure can be pre-configured according to different pole types. When the pole to be inspected is expanded from ordinary straight poles to special poles with transformers, switches, disconnectors, or surge arresters, only the corresponding inspection targets and inspection viewpoints need to be added, without rewriting the entire inspection control process. This feature is beneficial for the promotion and application of this application in various power equipment inspection scenarios.

[0197] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0198] Based on the same inventive concept, this application also provides a priori structural knowledge-guided UAV autonomous inspection system for implementing the aforementioned priori structural knowledge-guided UAV autonomous inspection method. The solution provided by this system is similar to the implementation scheme described in the above method. Therefore, the specific limitations of one or more priori structural knowledge-guided UAV autonomous inspection system embodiments provided below can be found in the limitations of the priori structural knowledge-guided UAV autonomous inspection method described above, and will not be repeated here.

[0199] In one embodiment, an autonomous inspection system for unmanned aerial vehicles (UAVs) guided by prior structural knowledge is provided, comprising:

[0200] The template acquisition module is used to acquire knowledge templates of tower structures and extract two-dimensional key points from tower images collected by UAVs. The two-dimensional key points are associated with semantic identifiers. The knowledge template of tower structure includes a local coordinate system of the tower, three-dimensional key points associated with semantic identifiers, inspection viewpoints, and observation targets of each inspection viewpoint.

[0201] The optimization module is used to optimize each of the three-dimensional key points and the corresponding two-dimensional key points. Minimize the objective function The first rotation matrix is ​​obtained. and the first translation parameter K represents the intrinsic parameter matrix of the camera mounted on the UAV. Let N be the coordinates of the i-th 3D keypoint, and N be the number of 3D keypoints.

[0202] The pose matrix determination module is used to obtain the second pose matrix of the tower local coordinate system relative to the world coordinate system based on the first rotation matrix, the first translation parameters and the first pose matrix of the camera coordinate system relative to the world coordinate system.

[0203] The coordinate transformation module is used to determine the inspection coordinates of each inspection viewpoint in the world coordinate system and the observation coordinates of the observed target in the world coordinate system based on the second pose matrix.

[0204] The inspection module is used to determine the inspection route of the UAV based on the inspection coordinates, determine the orientation of the UAV based on the observation coordinates, and control the UAV to perform autonomous inspections through the inspection route and the orientation.

[0205] The modules in the aforementioned UAV autonomous inspection system guided by prior structural knowledge can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the computer device's memory as software, so that the processor can call and execute the corresponding operations of each module.

[0206] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 7 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores various types of data. The network interface communicates with external terminals via a network connection. When executed by the processor, the computer program implements an autonomous inspection method for unmanned aerial vehicles (UAVs) guided by prior structural knowledge.

[0207] Those skilled in the art will understand that Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0208] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0209] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0210] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0211] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0212] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0213] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

Claims

1. A method for autonomous inspection of unmanned aerial vehicles (UAVs) based on prior structural knowledge, characterized in that, The method includes: S1. Obtain the knowledge template of the tower structure and extract the two-dimensional key points from the tower images collected by the UAV; the two-dimensional key points are associated with semantic identifiers, and the knowledge template of the tower structure includes the local coordinate system of the tower, the three-dimensional key points associated with semantic identifiers, the inspection viewpoints, and the observation targets of each inspection viewpoint; S2. Based on each of the three-dimensional key points and the corresponding two-dimensional key points Minimize the objective function The first rotation matrix is ​​obtained. and the first translation parameter K represents the intrinsic parameter matrix of the camera mounted on the UAV. Let N be the coordinates of the i-th 3D keypoint, and N be the number of 3D keypoints. S3. Based on the first rotation matrix, the first translation parameters, and the first pose matrix of the camera coordinate system relative to the world coordinate system, obtain the second pose matrix of the tower local coordinate system relative to the world coordinate system. S4. Based on the second pose matrix, determine the inspection coordinates of each inspection viewpoint in the world coordinate system and the observation coordinates of the observed target in the world coordinate system; S5. Determine the inspection route of the UAV based on the inspection coordinates, determine the orientation of the UAV based on the observation coordinates, and control the UAV to perform autonomous inspection through the inspection route and the orientation.

2. The method according to claim 1, characterized in that, Step S3 includes: Based on the first rotation matrix and the first translation parameter ,pass The third pose matrix of the tower's local coordinate system relative to the camera coordinate system is obtained. ; Based on the third pose matrix and the first pose matrix from the camera coordinate system to the world coordinate system ,pass The second pose matrix of the tower's local coordinate system relative to the world coordinate system is obtained. The second pose matrix , The second rotation matrix represents the local coordinate system of the tower relative to the world coordinate system. This indicates the position of the origin of the local coordinate system of the tower in the world coordinate system.

3. The method according to claim 2, characterized in that, Step S4 includes: Based on the second pose matrix and the coordinates of each inspection viewpoint in the local coordinate system of the tower. ,pass The inspection coordinates of each inspection viewpoint in the world coordinate system are obtained. ; Based on the second pose matrix and the coordinates of each of the observed targets in the local coordinate system of the tower. ,pass The observation coordinates of each of the observed targets in the world coordinate system are obtained. .

4. The method according to claim 2, characterized in that, The tower structure knowledge template also includes the inspection targets and shooting requirements corresponding to each inspection viewpoint, and the autonomous inspection of the UAV by controlling the inspection route and orientation includes: The drone is controlled to photograph the target inspection point according to the shooting requirements and orientation corresponding to the target inspection point, thereby obtaining an inspection image; the target inspection point is one of the inspection points. The inspection object in the inspection image is identified by the target detection model, and the detection box information of each inspection object is output; the detection box information includes the center coordinates of the detection box; Based on the second pose matrix, through The coordinates of each inspection target in the local coordinate system of the tower are... Convert to the first inspection coordinates in the world coordinate system ; The fourth pose matrix based on the world coordinate system to the target camera coordinate system ,pass The first inspection coordinates are converted into the second inspection coordinates in the target camera coordinate system. The target camera coordinate system is a three-dimensional coordinate system established with the camera optical center as the origin when acquiring the inspection images; Based on the aforementioned intrinsic parameter matrix, through The theoretical two-dimensional projection position of the inspection target in the inspection image is obtained. ; Calculate the first distance between the theoretical two-dimensional projection position of the inspection target and the center of the detection frame; Based on the first distance, the target detection frame of the inspection target is determined from the detection frame.

5. The method according to claim 4, characterized in that, The number of inspection targets and detection boxes in the inspection image is multiple, and the detection box information also includes confidence levels. Determining the target detection box of the inspection target from the detection boxes based on the first distance includes: Normalize each of the first distances to obtain the normalized distance between the theoretical two-dimensional projection position of each inspection target and the center coordinates of each detection frame. ; pass Calculate the comprehensive cost between each of the theoretical two-dimensional projection positions and the center of each of the detection frames. ; This is the distance weighting coefficient. Let k be the area of ​​the k-th detection box. Let j be the theoretical area of ​​the j-th inspection target. These are the scale weighting coefficients. The confidence level weighting coefficient is... Let k be the confidence level of the k-th detection box. Based on the aforementioned comprehensive cost, minimize To obtain the related variables The specific value; Based on the specific values ​​of the associated variables, the target detection box of each of the inspection targets is determined; This indicates that the k-th detection box is the target detection box for the j-th inspection target.

6. The method according to claim 4, characterized in that, The detection frame information also includes the detection frame width, detection frame height, and confidence level; the method further includes: Calculate the second distance between the center of the target detection box and the center of the inspected image; through Calculate the area ratio of the inspected target to which the target detection box belongs in the inspected image. ;pass Calculate the height percentage of the main tower body in the inspection image. ;w n h is the width of the target detection box. n Let W be the height of the target detection box, W be the width of the inspection image, and H be the height of the inspection image. The height of the main tower body in the inspection image; Based on the area proportion, confidence level, height proportion, and second distance corresponding to the target detection box, through The comprehensive score of the inspection image is obtained; As the center deviation weight, For scale weights, As the confidence level weight, Main body weight; It is the maximum value in the second distance; When the overall score is less than a preset threshold, the coordinates of the target inspection viewpoint and / or the orientation of the UAV at the target inspection viewpoint are adjusted so that the overall score of the adjusted inspection image is greater than or equal to the preset threshold.

7. The method according to claim 6, characterized in that, When the overall score is less than a preset threshold, the orientation of the UAV at the target inspection viewpoint is adjusted so that the overall score of the adjusted inspection image is greater than or equal to the preset threshold, including: When the overall score is less than a preset threshold, the center coordinates of the target detection box are used as the basis for the determination. The height H and width W of the inspection image are determined by... Calculate width deviation ,pass Calculate height deviation ; Based on the width deviation The width of the inspection image and the preset yaw angle adjustment coefficient. ,pass Determine the yaw angle adjustment amount Based on the height deviation The height of the inspection image and the preset pitch angle adjustment coefficient. ,pass Determine the pitch angle adjustment amount ; Based on the yaw angle adjustment and the pitch angle adjustment, the adjusted orientation of the UAV after adjusting at the target inspection viewpoint is determined; The drone is adjusted to the adjusted orientation at the target inspection viewpoint, and the drone is controlled to take pictures of the inspection target at the target inspection viewpoint according to the corresponding shooting requirements, so as to obtain an adjusted inspection image with a comprehensive score greater than or equal to the preset threshold.

8. The method according to claim 6, characterized in that, When the overall score is less than a preset threshold, the coordinates of the target inspection viewpoint are adjusted so that the overall score of the adjusted inspection image is greater than or equal to the preset threshold, including: When the overall score is less than a preset threshold, the inspection coordinates are based on the target inspection viewpoint. The observation coordinates of the observed target corresponding to the target inspection viewpoint ,pass Calculate the unit direction vector from the target inspection viewpoint to the observed target. ; Based on the unit direction vector ,pass Obtain the adjusted coordinates of the target inspection viewpoint ; The signed step size is the distance moved along the direction of observation. The drone is controlled to capture images of the corresponding inspection target from the adjusted target inspection viewpoint, so as to obtain an adjusted inspection image with a comprehensive score greater than or equal to the preset threshold.

9. The method according to claim 1, characterized in that, Determining the orientation of the UAV based on the observed coordinates includes: Based on the observation coordinates of the observed target corresponding to the inspection viewpoint and the inspection coordinates of the inspection viewpoint, through The direction vector pointing from the inspection viewpoint to the corresponding observed target is obtained; Based on the components of the direction vector on the X-axis of the world coordinate system Components along the Y-axis ,pass Calculate the yaw angle of the inspection viewpoint ; Based on the components of the direction vector on the X-axis of the world coordinate system Y-axis components and the components of Z ,pass Calculate the pitch angle ; Based on the yaw angle and pitch angle corresponding to each of the inspection viewpoints, the orientation of the UAV at each of the inspection viewpoints is determined.

10. An autonomous inspection system for unmanned aerial vehicles (UAVs) guided by prior structural knowledge, used to execute the method according to any one of claims 1-9, characterized in that, The system includes: The template acquisition module is used to acquire knowledge templates of tower structures and extract two-dimensional key points from tower images collected by UAVs. The two-dimensional key points are associated with semantic identifiers. The knowledge template of tower structure includes a local coordinate system of the tower, three-dimensional key points associated with semantic identifiers, inspection viewpoints, and observation targets of each inspection viewpoint. The optimization module is used to optimize each of the three-dimensional key points and the corresponding two-dimensional key points. Minimize the objective function The first rotation matrix is ​​obtained. and the first translation parameter K represents the intrinsic parameter matrix of the camera mounted on the UAV. Let N be the coordinates of the i-th 3D keypoint, and N be the number of 3D keypoints. The pose matrix determination module is used to obtain the second pose matrix of the tower local coordinate system relative to the world coordinate system based on the first rotation matrix, the first translation parameters and the first pose matrix of the camera coordinate system relative to the world coordinate system. The coordinate transformation module is used to determine the inspection coordinates of each inspection viewpoint in the world coordinate system and the observation coordinates of the observed target in the world coordinate system based on the second pose matrix. The inspection module is used to determine the inspection route of the UAV based on the inspection coordinates, determine the orientation of the UAV based on the observation coordinates, and control the UAV to perform autonomous inspections through the inspection route and the orientation.