Method and device for intelligent visual inspection of edge of unmanned aerial vehicle in remote area

Abstract

The invention discloses a method and a device for unmanned aerial vehicle edge intelligent visual inspection in a remote area, relates to the technical field of unmanned aerial vehicle inspection, is applied to areas with complex terrains, lack of RTK base stations or areas with weak network signals, and is used for inspecting a target power grid area comprising power grid facilities and power lines by an unmanned aerial vehicle. The method comprises the following steps of S1, performing three-dimensional reconstruction on a power tower and a power line of a target power grid region in real time based on monocular vision of airborne edge equipment; s2, the unmanned aerial vehicle autonomously and dynamically plans an inspection route based on machine vision; and carrying out real-time detection and analysis based on the key photographing part information of the airborne edge equipment. According to the method, the unmanned aerial vehicle inspection technology is not limited by the coverage range of the RTK base station any more, the dependence on external facilities is reduced, the flexibility and adaptability of inspection are improved, and the method has great research and application value for unmanned aerial vehicle autonomous inspection in remote areas of Xinjiang.

Description

Technical Field

[0001] The present invention relates to the field of drone inspection technology, and in particular to a method and device for drone edge intelligent visual inspection in remote areas. Background Art

[0002] With the rapid development of cutting-edge technologies such as big data, cloud computing, the Internet of Things, mobile Internet and artificial intelligence, State Grid Corporation of China has thoroughly implemented the latest requirements of the "New Generation Artificial Intelligence Development Plan" and further refined the "State Grid Corporation of China Artificial Intelligence Special Plan", aiming to comprehensively build a coordinated three-dimensional intelligent operation and inspection model for overhead transmission lines in all units of the company system. This measure aims to significantly improve the benefits, efficiency and quality of inspections, significantly reduce operation and maintenance costs, accelerate the transformation and upgrading of the operation and inspection model, and promote the comprehensive construction and development of smart grids. At the same time, the iterative upgrades of technologies such as big data and cloud computing around the world have laid a solid foundation for the widespread application of artificial intelligence. In particular, the release of the "China Internet Development Report (2024)" clearly pointed out that data has become the commanding heights of the new round of artificial intelligence competition. Against this background, State Grid Corporation of China has actively responded to the national call, made full use of data resources, and promoted the deep integration and innovative application of artificial intelligence technology in power inspections.

[0003] In terms of drone technology, it has achieved leapfrog development in recent years. Drone intelligent inspection technology not only realizes autonomous flight and intelligent data analysis, but also has the ability to monitor equipment status in real time and finely control defects and hidden dangers. In particular, the integrated application of "power + Beidou" technology has significantly improved the accuracy and safety of drone inspections, providing strong technical support for transmission line inspections. State Grid Corporation actively promotes the popularization and application of drone inspection technology. Through the inspection mode of "helicopter + drone + manual collaboration", it has achieved full coverage of 500kV and above lines, and gradually expanded to low-voltage level lines.

[0004] Xinjiang Kashgar Power Supply Company closely follows this technological trend and actively promotes the use of drones in power inspections, significantly improving the efficiency and safety of inspections. However, despite the many conveniences brought by autonomous drone inspection technology, it still faces some challenges in its actual application, especially in remote areas, where targeted technical research is urgently needed.

[0005] One of the core technologies of autonomous inspection by drones is to rely on RTK (real-time dynamic carrier phase differential technology) for high-precision positioning. However, the implementation of RTK technology requires stable network signals and wide coverage of RTK base stations. In Kashgar and its surrounding areas, due to the complex terrain, many power transmission lines are located in mountainous areas or remote areas, where network signals are often weak or even lack RTK base stations. This means that in these areas, drones cannot rely on RTK technology for autonomous inspections, which greatly limits the coverage and application effect of drone inspections.

[0006] At the same time, considering the unique climatic conditions in remote areas of Xinjiang, such as wind direction, wind speed, and the drone's own status, which will affect the autonomous flight status of the drone, its positioning accuracy may deviate, and this deviation will cause the image taken by the drone to deviate from the center of the target, or the target taken is not clear, thus affecting the inspection effect. Especially under complex weather conditions, such as strong winds, rain and snow, the positioning accuracy of the drone is more easily affected, further increasing the difficulty of inspection.

[0007] In addition, route planning for autonomous inspections by drones is also an important issue. When planning a route, many factors need to be considered, such as terrain, obstacles, and flight altitude. However, due to the complex and changeable terrain in remote areas and the lack of detailed terrain data support, route planning is often difficult to achieve accuracy and rationality. This can cause the drone to deviate from its route during flight or fail to reach the scheduled inspection point. At the same time, due to unreasonable route planning, the image angle taken by the drone may also be limited, and it is impossible to fully and clearly reflect the operating status of the power facilities.

[0008] Therefore, it is necessary to design a method and device for edge intelligent visual inspection of drones in remote areas to solve the above problems. Summary of the invention

[0009] The technical problem to be solved by the present invention is to solve the deficiencies in the prior art and to design a method and device for edge intelligent visual inspection by drones in remote areas. The present invention provides a new autonomous inspection solution based on GPS, which can solve the problem that the existing drone inspection technology relies on the coverage of RTK base stations.

[0010] The solution adopted by the present invention to solve the technical problem is:

[0011] A method for edge intelligent visual inspection by drones in remote areas is applied to areas with complex terrain, lack of RTK base stations, or weak network signals. The drones inspect target power grid areas including power grid facilities and power lines, including the following steps:

[0012] Step S1: Based on the monocular vision of the airborne edge device, three-dimensional reconstruction of the power towers and power lines in the target power grid area is performed in real time, specifically including the following steps S11 to S12:

[0013] Step S11: Based on the analysis of heterogeneous power tower components, a power tower model is established based on the ontology to provide accurate information for the identification of power towers and the positioning of key checkpoints, and then enter step S12,

[0014] Step S12: A three-dimensional model of the power line is established in combination with the airborne edge device. In the stage where the posture changes slowly, a direct method is used to build the map. At the same time, the colinear and coplanar attributes of the features are supplemented in the dense map generated by the direct method. In the stage where the posture changes rapidly, a feature-based method is used to build the map.

[0015] Step S2: The drone autonomously and dynamically plans the inspection route based on machine vision, specifically including steps S21 to S23.

[0016] Step S21: Based on existing technical research and corresponding point cloud scanning data, typical tower data training is performed to form a tower library of different specifications and voltage levels;

[0017] Step S22: Combined with the dead reckoning algorithm, before takeoff, the inspection route is automatically generated on site according to different tower types, and the inspection route is predicted through point cloud display according to the longitude, latitude and height information of the tower, realizing automatic recognition of the tower shape and automatic generation of shooting points.

[0018] Step S23: Set the inspection shooting viewpoint, that is, determine the hovering position of the drone; plan the inspection route; execute automatic flight; the drone flies to the shooting viewpoint and hovers, the front-end intelligent recognition module identifies the key parts of the tower, determines the position of the key parts of the tower, plans the order of key part recognition, focuses and takes pictures in sequence; enters the next shooting viewpoint until the shooting is completed;

[0019] Step S3: Perform real-time detection and analysis based on the key photographing part information of the airborne edge device, specifically including steps S31 to S32.

[0020] Step S31: Based on the deep learning model compression technology, the algorithms required for inspection, such as target recognition, real-time detection of defects and anomalies, and control, are deployed on the airborne device to realize real-time analysis and processing of data, thus saving a large amount of network bandwidth and storage problems caused by data transmission.

[0021] Step S32: gradually establish a defect library review mechanism, quickly accumulate defect data through human-computer interaction, establish a defect sample library, perform type analysis on the accumulated defect sample library, determine the number of defects of different defect types and whether they are suitable for deep learning recognition, iterate data and algorithms for defects suitable for deep learning recognition, and further identify defects that are not suitable for deep learning recognition in combination with device targets and global semantic analysis, and finally determine the defect diagnosis algorithm.

[0022] As a preferred embodiment of the present invention, step S11 includes the following steps S111 to S114:

[0023] Step S111: analyzing basic components of the power tower according to the power tower design standard;

[0024] Step S112: analyzing key characteristics of different types of power pole tower components, and establishing a property set of the power pole tower components for accurately describing and recording key characteristics of the power pole tower components;

[0025] Step S113: Analyze the association relationship and constraint rules between different types of components and define them in detail, and establish an association relationship model and corresponding constraint rules for the connection between various components of the corresponding power tower;

[0026] Step S114: Constructing a knowledge base of power tower components based on the ontology:

[0027] Step S115: constructing and identifying a power pole tower model based on the power pole tower components.

[0028] As a preferred embodiment of the present invention, step S114 includes steps S1141 to S1145;

[0029] Step S1141: analyzing important terms in power tower components;

[0030] Step S1142: define power tower component classes and class hierarchies:

[0031] Step S1143: define the properties of each power tower component class:

[0032] Step S1144: define facets of power tower component attributes:

[0033] Step S1145: Create a corresponding power pole tower component instance.

[0034] As a preferred embodiment of the present invention, during the automatic flight of the drone in step S23, the dynamic planning of the autonomous inspection route is performed using the relative relationship between GPS and key points, which specifically includes the following steps S231 to S232:

[0035] Step S231: Use the onboard pan-tilt camera to visually locate and correct the power tower and the surrounding environment, and then proceed to step S232;

[0036] Step S232: According to the result of visual positioning, the aircraft is controlled to move according to a preset route, and then the process proceeds to step S233;

[0037] Step S233: During the flight, the drone automatically identifies and locks each key part, and then takes precise photos respectively.

[0038] As a preferred implementation of the present invention, the step S31 specifically includes the following steps S311 to S316:

[0039] Step S311: training the server-side power grid equipment detection model, and then proceeding to step S312;

[0040] Step S312: Pruning, model distillation and fine-tuning the power grid equipment detection model, and then proceeding to step S313;

[0041] Step S313: quantize after training using the Bit-split technology, or use perceptual quantization to quantize during the fine-tuning of the power grid equipment detection model to obtain a lightweight model, and then proceed to step S314;

[0042] Step S314: The corresponding tool chain of the final deployed chip converts the lightweight model into a format that meets the requirements of the airborne device deployment algorithm, and then proceeds to step S315;

[0043] Step S315: deploy the lightweight model algorithm on the airborne end and verify it by actual flight of the UAV.

[0044] As a preferred embodiment of the present invention, the direct method used to build the map in step S12 requires extracting feature points to complete loop detection and relocation functions, specifically by calculating the Shi-Tomas corner point score and selecting the point category as a feature point or an edge point according to the threshold.

[0045] As a preferred embodiment of the present invention, the basic components of the power tower include a tower crown component, a tower base component and electrical components.

[0046] As a preferred embodiment of the present invention, the configuration of the tower crown assembly includes a dry-shaped tower, a wine glass-shaped tower, a cat-head-shaped tower, a rotating tower, a single-circuit tower, a double-circuit tower, a two-phase double-circuit tower and a three-phase double-circuit tower;

[0047] The heights of the tower base components include 5M tower feet, 10M tower feet, 15M tower feet and 20M tower feet.

[0048] As a preferred embodiment of the present invention, the elements of the ontology include concepts, relations, attributes, axioms, functions and individual instances of concepts.

[0049] A device for edge intelligent visual inspection of drones in remote areas, comprising an edge intelligent terminal and a drone, wherein the hardware device of the edge intelligent terminal is connected to the drone via a serial port to control the drone; the hardware device of the edge intelligent terminal is connected to the drone via a USB port to transmit camera data to the edge intelligent terminal for identification;

[0050] The drone comprises a mobile terminal, an onboard computer terminal and a drone load terminal. The mobile terminal receives terminal instructions and controls the drone load terminal and the onboard computer to execute instruction tasks.

[0051] Beneficial effects:

[0052] 1. Assist and strengthen existing autonomous inspections: The method and device of the present invention make the UAV edge intelligent visual inspection technology no longer restricted by the coverage of the RTK base station. In areas where there are no RTK base stations, and where RTK signals are difficult to obtain or weak, innovative positioning and navigation technologies are used to effectively make up for the shortcomings of traditional methods, ensure the smooth progress of inspection tasks, reduce dependence on external facilities, and improve the flexibility and adaptability of inspections. It has great research and application value for autonomous inspections by UAVs in remote areas of Xinjiang. At the same time, it is an inevitable choice to respond to the national development strategy, promote industrial upgrading, and realize the construction of smart grids. Through the method and device of the present invention, the application of artificial intelligence technology in power inspections is continuously deepened, and the inspection benefits, efficiency and quality are continuously improved, providing a strong guarantee for the safe and stable operation of the power grid.

[0053] 2. New breakthrough in autonomous inspection in areas without RTK: For remote or complex terrain areas without RTK base station coverage, the fully autonomous navigation and inspection functions of drones are realized, greatly broadening the coverage of inspection operations. This technological breakthrough not only reduces the burden of manual work, but also ensures continuous and safe monitoring of power lines in these areas, providing a strong guarantee for the stable operation of the power grid. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 A flow chart of a method for edge intelligent visual inspection of unmanned aerial vehicles in remote areas proposed by the present invention;

[0055] Figure 2 A technical roadmap for modeling heterogeneous power tower components in a method for edge intelligent visual inspection of remote areas by drones proposed in the present invention;

[0056] Figure 3Schematic diagram of tower crown components of different configurations in a method for edge intelligent visual inspection of unmanned aerial vehicles in remote areas proposed by the present invention;

[0057] Figure 4 A schematic diagram of tower base components at different heights in a method for edge intelligent visual inspection of unmanned aerial vehicles in remote areas proposed by the present invention;

[0058] Figure 5 A schematic diagram of window sliding in a method for edge intelligent visual inspection of unmanned aerial vehicles in remote areas proposed by the present invention;

[0059] Figure 6 The distribution of λ1, λ2 and pixel points in a method for edge intelligent visual inspection of unmanned aerial vehicles in remote areas proposed by the present invention;

[0060] Figure 7 This is a block diagram of the classic Harris algorithm in a method for edge intelligent visual inspection of unmanned aerial vehicles in remote areas proposed by the present invention;

[0061] Figure 8 This is a capture design flow chart for a method for edge intelligent visual inspection of unmanned aerial vehicles in remote areas proposed by the present invention;

[0062] Fig. 9 A flow chart of the research on power inspection image defect recognition in a method for edge intelligent visual inspection of unmanned aerial vehicles in remote areas proposed by the present invention;

[0063] Fig.10 This is a hardware integration architecture diagram of edge intelligent terminal equipment in a method for edge intelligent visual inspection of unmanned aerial vehicles in remote areas proposed by the present invention; Fig.11 This is the communication process between various components of a drone in a method for edge intelligent visual inspection of drones in remote areas proposed by the present invention. DETAILED DESCRIPTION

[0064] The specific implementation of the present invention is described below in conjunction with the accompanying drawings and embodiments:

[0065] It should be noted that the structures, colors, proportions, sizes, etc. illustrated in the drawings of this specification are only used to match the contents disclosed in the specification so that people familiar with this technology can understand and read them, and are not used to limit the conditions under which the present invention can be implemented. Any structural modification, change in proportional relationship or adjustment of size should still fall within the scope of the technical content disclosed in the present invention without affecting the effects and purposes that can be achieved by the present invention.

[0066] like Figure 1 As shown, the present invention provides a method for managing abnormal area energy storage, comprising the following steps:

[0067] Step S1: Based on the monocular vision of the airborne edge device, the power towers and power lines in the target power grid area are reconstructed in real time in three dimensions, so that the drone knows where it is;

[0068] Step S2: The drone autonomously and dynamically plans the inspection route based on machine vision, so that the drone knows how to fly safely;

[0069] Step S3: Real-time detection and analysis is performed based on the key photographing part information of the airborne edge device, in order to let the drone know what parts to photograph.

[0070] Step S1 specifically includes 1. Research on modeling technology of heterogeneous power tower components: In view of the complexity of transmission lines, research on modeling technology of tower components under various tower types and voltages. By analyzing the unique properties of the tower and integrating them into the 3D reconstruction process, accurate reconstruction of key parts of the tower is achieved to accurately locate the target for shooting.

[0071] 2. Research on 3D reconstruction technology based on airborne edge devices: Use the camera equipment, lidar equipment and edge computing equipment carried by drones for real-time 3D reconstruction. Research the real-time 3D reconstruction algorithm implemented on edge computing devices to provide accurate 3D information in real-time dynamic route planning.

[0072] Step S2 specifically includes 1. Research on real-time inspection tower type recognition and autonomous shooting positioning technology based on machine vision: by building algorithms and model training, the key shooting parts of the tower can be recognized. Using the recognition model, research is conducted on how to locate key components on different sides and phases of the tower, so as to achieve the screening and autonomous adjustment of shooting targets.

[0073] 2. Research on autonomous inspection route dynamic planning technology based on the relative relationship between GPS and key points: Research on how to dynamically adjust the route of the drone in GPS mode using the relative position relationship of the tower structure. This research aims to ensure the accuracy of the drone's displacement and the accuracy of visual model recognition in GPS mode.

[0074] Step S3 specifically includes 1. Research on airborne lightweight edge computing equipment technology based on autonomous inspection of power transmission lines; research on edge computing equipment that can be carried on drones and build lightweight models. In order to achieve high recognition accuracy and fast recognition speed of the model in actual application scenarios, it is necessary to study the quantization and reasoning acceleration algorithms of the model.

[0075] 2. Research on defect recognition in autonomous inspection images based on edge vision recognition. Autonomous inspection based on edge vision recognition will make the target of the captured components clear, which will be more helpful in improving defect recognition. On the basis of the inspection images, the application of graph relationships will be deepened to further improve the accuracy of the defect model.

[0076] With the trend of refined and intelligent management of power grids, the demand for high-precision modeling of poles and towers is becoming increasingly strong. At present, most of the three-dimensional models of power transmission line towers at home and abroad rely on professional modelers to refer to the corresponding design parameters and use CAD software to manually create them. This is highly professional, labor-intensive, time-consuming and labor-intensive, and it is difficult to meet the needs of rapid digital modeling of a large number of heterogeneous power towers. The present invention intends to establish a power tower model based on ontology technology based on the analysis of heterogeneous power tower components, so as to provide accurate information for the identification of power towers and the positioning of key checkpoints. Its main research contents are:

[0077] 1. Analyze the basic components and related properties of power towers according to the power tower design standards;

[0078] 2. Research and establish the relationship between power tower components;

[0079] 3. Build a knowledge base of power tower components based on ontology;

[0080] 4. Research on the method of constructing and identifying power tower models based on power tower components. Specific research implementation plan and technical route are as follows: Figure 2 shown.

[0081] Step S1 specifically includes the following steps:

[0082] Step S11: On the basis of analyzing the heterogeneous power tower components, a power tower model is established based on the ontology to provide accurate information for the identification of the power tower and the positioning of key checkpoints, and then proceed to step S12;

[0083] Step S11 includes the following steps:

[0084] Step S111: analyzing basic components of the power tower according to the power tower design standard;

[0085] In the power system, poles and towers are the key supporting structures of transmission lines. Their design and construction must strictly follow the specific type standards formulated by the state and the industry. These standards not only ensure the structural safety and stability of the poles and towers, but also promote the standardization and modularization of the pole and tower design. Given that most poles and towers show a high degree of similarity in the tower foot structure and the tower head design of similar poles, this feature provides important convenience for the digital modeling of power line corridors.

[0086] Through detailed interpretation and analysis of the power tower design standards, the basic structure, size parameters, material requirements and installation specifications of various towers specified in the standards are clarified. Subsequently, by rationally decomposing these standardized models, the main components of the tower are identified, such as the tower crown, tower base, crossarm, insulator string, electrical components (such as lightning arresters, disconnectors, etc.) and connectors. Each component carries a specific function and role, and together constitutes the overall structure of the tower.

[0087] Explore the connection mode and interaction relationship between components. By simulating the assembly process in the real environment, the decomposed components are flexibly assembled in the digital environment to build a transmission line tower model that meets the design standards. This component modeling method not only improves the efficiency and accuracy of modeling, but also makes the model easier to maintain and update.

[0088] Specifically, when constructing the tower crown assembly, factors such as its shape, size, material, and load-bearing capacity are considered. Figure 3 When constructing the tower base assembly, you need to pay attention to its foundation form, burial depth, stability, and connection method with the ground; the tower base assembly height is as shown in Figure 4 As shown. For the modeling of electrical components, it is necessary to accurately reflect their electrical performance, installation methods, and interface relationships with other components. By comprehensively considering these factors, a digital model of power towers that meets both design standards and actual needs is constructed.

[0089] Step S112: analyzing key characteristics of different types of power pole tower components, and establishing a property set of the power pole tower components for accurately describing and recording key characteristics of the power pole tower components;

[0090] Analyze the key characteristics of each power tower component to establish the attribute set of the power tower component, and effectively and accurately describe and record the key characteristics of the power tower component, such as electrical characteristics, size configuration, location information, etc.

[0091] Step S113: Analyze the association relationship and constraint rules between different types of components and define them in detail, and establish an association relationship model and corresponding constraint rules for the connection between various components of the corresponding power tower;

[0092] In order to ensure seamless connection between the tower crown, tower base and electrical components, the association relationship and constraint rules between different types of components in the model library must also be defined in detail. The association relationship model and corresponding constraint rules between the components of the corresponding power tower are established, mainly including spatial position constraints, size constraints and functional constraints.

[0093] Step S114: Constructing a knowledge base of power tower components based on the ontology:

[0094] Ontology is a clear and standardized description of the conceptual model. Ontology technology can be used to achieve power tower component modeling, power tower structure identification and reasoning, etc.

[0095] Ontology has the following elements: (1) Concept. Also known as class. (2) Relation. Relation is used to describe the connection between concepts in the ontology. Typical binary relations include the is-a relationship between concepts. (3) Property. Property refers to the predicate in a statement used to describe an individual. Properties are divided into object properties and data properties. Object properties connect individuals, while data properties connect individuals and text values. (4) Axiom. Axioms in the ontology refer to recognized facts used for knowledge reasoning. (5) Function. Function is a specific expression of a relationship. The mapping relationship specified in the function can make the reasoning point from one concept to another. (6) Individual Instance of Concept. The individual instance of a concept is the concept at the lowest logical level. Its extension is only itself, referred to as instance.

[0096] The specific steps are as follows:

[0097] Step S1141: analyzing important terms in power tower components;

[0098] Step S1142: define power tower component classes and class hierarchies:

[0099] Step S1143: define the properties of each power tower component class:

[0100] Step S1144: define facets of power tower component attributes:

[0101] Step S1145: Create a corresponding power pole tower component instance.

[0102] Step S115: constructing and identifying a power pole tower model based on the power pole tower components.

[0103] Reasoners can infer logical results from a set of asserted facts or axioms, and they play an important role in writing ontology using OWL. We intend to study the reasoning rules for building power tower components to detect whether the corresponding components meet the consistency and inclusion relationship between classes and other rule requirements. This will detect the correctness of the power tower model built by components and automatically identify the types of various power towers, providing correct and reliable data information such as configuration and location for the inspection of power towers.

[0104] In this way, we can ensure that the model of the power tower is correct and that different types of towers can be automatically identified, which is very important for power inspections because it can provide accurate tower configuration and location information, helping inspectors better understand and maintain the towers.

[0105] The inference engine can be understood as an intelligent agent that uses logical reasoning techniques to analyze and interpret the information stored in the ontology. Ontology is a structured knowledge representation system that defines concepts, attributes, and the relationships between them in a certain field. Based on these definitions and relationships, the inference engine can discover new knowledge or verify whether existing knowledge is consistent through logical reasoning.

[0106] OWL (Web Ontology Language) is a language for describing the semantics and relationships of web resources. It allows us to define and describe entities such as people, places, things, and the relationships between them on the web in a machine-readable way. OWL is part of the semantic web technology, which enables computers to understand and process information on the web more intelligently. Although it is not a programming language, we can use it to describe data models, build ontologies, and analyze and reason about these data through reasoning engines.

[0107] Step S12: A three-dimensional model of the power line is established in combination with the airborne edge device. In the stage where the posture changes slowly, a direct method is used to build the map. At the same time, the colinear and coplanar attributes of the features are supplemented in the dense map generated by the direct method. In the stage where the posture changes rapidly, a feature-based method is used to build the map.

[0108] In step S12, the direct method for mapping needs to extract feature points to complete loop detection and relocalization functions. Specifically, the Shi-Tomas corner point score is calculated and the point category is selected as a feature point or an edge point according to the threshold.

[0109] A large number of experiments have shown that under the conditions of smooth motion and little illumination change, the direct method is superior to the current feature-based methods in terms of accuracy and robustness, and the established three-dimensional point cloud is also significantly denser than the feature point-based method.

[0110] The direct method is very different from other feature-based methods in two aspects:

[0111] The feature-based method calculates the camera pose and the position of the map point by minimizing the reprojection error, while the direct method minimizes the photometric error. The so-called photometric error means that the objective function to be minimized is usually determined by the error between images, rather than the geometric error after reprojection. Due to the limitation of the number of feature points, the feature-based method uses much less information than the direct method, which limits its positioning and mapping effects.

[0112] The direct method puts data association and pose estimation into a unified nonlinear optimization problem, while the feature-based method solves it in steps, that is, first find the association between data by matching feature points, and then estimate the pose based on the association. These two steps are usually independent. In the second step, the outliers in the data association can be determined by the reprojection error, which can also be used to correct the matching results. When the movement is intense, due to the prior brought by the first step of the feature-based method, the feature points are not easy to fail to track, and the effect is better than the direct method.

[0113] Experiments show that by combining the different advantages of the direct method and the feature-based method, more accurate and robust results can be obtained. The direct method is difficult to perform loop detection and relocation, and to a certain extent, it is also necessary to extract feature points to complete the loop detection and relocation functions. Therefore, the present invention calculates the Shi-Tomas corner point score and selects the point category as a feature point or an edge point based on the threshold at one time, thereby combining it with the feature point method. The characteristics of Shi-Tomas corner points are insensitivity to changes in brightness and contrast and rotation invariance. The present invention adopts the direct method in the stage where the posture changes slowly; and adopts the feature-based method when the posture changes quickly. At the same time, in the dense map generated by the direct method, the collinearity and coplanarity attributes of the features are supplemented.

[0114] Shi-Tomas corner detection algorithm was proposed by J. Shi and C. Tomasi in 1994. It is an improved algorithm of Harris algorithm and has achieved good results. Since the main framework of this algorithm is the same as Harris algorithm, Harris algorithm is introduced first.

[0115] The classic Harris algorithm is a corner feature extraction algorithm proposed by C. Harris et al. at the Alvey Vision Conference in the late 1980s. Its basic principle is to take the target pixel as the center, calculate the change of the grayscale curvature within its window, and select the point with the maximum curvature change value as the feature point. The calculation method is as follows:

[0116] Let G x,yis the grayscale change when the neighborhood window W of the pixel point (x, y) moves (u, v). When the movement (u, v) is a local minimum, the following can be obtained after diagonalization and quadratic calculation based on the Talyor series expansion:

[0117]

[0118] Among them, I is the image grayscale, I x ,I y are the first-order partial derivatives in the x and y directions, respectively. is the covariance matrix of the image gradient, is the window function of window W, which acts as noise reduction and smoothing. λ1 and λ2 are the eigenvalues ​​of matrix M, and R is a two-dimensional rotation matrix.

[0119] like Figure 5 and Figure 6 As shown in the figure, the λ1 and λ2 eigenvalues ​​indicate the distribution of the grayscale values ​​of the image pixels. When λ1 and λ2 are both relatively small, there is no obvious grayscale change when the image window moves in any direction, indicating that the corresponding pixel point in the image window is in a smooth area. When one of λ1 and λ2 is large and the other is small, the grayscale change of the image window in one direction is more obvious, and there is no obvious grayscale change in the other direction, indicating that the corresponding pixel point in the image window is located at the boundary of the object. When λ1 and λ2 are both relatively large, the curvature or grayscale gradient of the pixel point in the image window moving in any direction changes greatly, and this point is the desired feature point.

[0120] When extracting feature points from an entire image, it is necessary to calculate the eigenvalues ​​of all pixels in the image, which requires a lot of calculations. Since the matrix M is a real symmetric matrix, let:

[0121]

[0122] but:

[0123] Among them, tr(M) is the trace of matrix M, and det(M) is its determinant. In order to improve the computational efficiency, tr(M) and det(M) are usually used to avoid the calculation of λ1 and λ2. Thus, the corner feature detection function is obtained:

[0124] R(x,y)=det(M)-k(tr(M)) 2 =(AB-C 2 )-k(A+B) 2

[0125] Here, k=0.04-0.06, which is an empirical constant and can also be adaptively modified according to actual scenarios.

[0126] Given a threshold T, for a pixel point (x, y), when R(x, y)>T and it is the maximum value in the local neighborhood window, the pixel point (x, y) is the desired corner point.

[0127] To sum up, we can get the block diagram of the classic Harris algorithm, as follows Figure 7 shown.

[0128] Shi and Tomasi later improved the classic Harris algorithm. The original definition of the algorithm is to subtract the determinant value of the matrix M from the trace of M, and then compare the difference with a pre-given threshold T. Shi and Tomasi proposed an improved method. If the smaller of the two eigenvalues ​​is greater than the minimum threshold, then a strong corner point will be obtained. That is:

[0129] R(x,y)=min(λ1,λ2)

[0130] Since larger uncertainties depend on smaller eigenvalues, good feature points can be found by finding the maximum value of small eigenvalues.

[0131] The Shi-Tomas algorithm only modifies the calculation method of R(x,y), and the rest is the same as the Harris operator. The Shi-Tomas algorithm will get better results than the Harris algorithm in most cases. The operator can be implemented using the goodFeaturesToTrack function in Opencv.

[0132] Through comparative studies of algorithms, the three-dimensional reconstruction of power lines is finally achieved by combining airborne edge devices.

[0133] Step S2 includes steps S21 to S23,

[0134] Step S21: Based on existing technical research and corresponding point cloud scanning data, typical tower data training is performed to form a tower library of different specifications and voltage levels;

[0135] Step S22: Combined with the route calculation algorithm, before takeoff, the inspection route is automatically generated on site according to different tower types, and the inspection route is predicted through point cloud display according to the longitude, latitude and height information of the tower, realizing automatic recognition of the tower shape and automatic generation of shooting points, thereby providing support for the next step of research;

[0136] like Figure 8As shown, the snapshot design process based on edge computing, step S23: set the shooting viewpoint for inspection, that is, determine the hovering position of the drone; plan the inspection route; execute automatic flight; the drone flies to the shooting viewpoint and hovers, the front-end intelligent recognition module identifies the key parts of the tower, determines the position of the key parts of the tower, plans the order of identifying the key parts, and focuses on the photos in sequence; enters the next shooting viewpoint until the photo is taken;

[0137] In step S23, during the automatic flight of the drone, the relative relationship between GPS and key points is used to dynamically plan the autonomous inspection route, aiming to achieve the autonomous inspection task of the drone on the transmission line tower by using only GPS signals without relying on RTK equipment. Specifically, the following steps S231 to S232 are included:

[0138] Step S231: Use the onboard gimbal camera to visually locate and correct the power tower and its surroundings. During this process, the drone will capture high-definition images and use advanced image processing algorithms to identify key parts such as the overall structure of the tower, ground wire connections, insulators, and hardware on both sides. This step is crucial to ensure accurate positioning of the drone in a complex environment. Then proceed to step S232;

[0139] Step S232: According to the result of visual positioning, the aircraft is controlled to move along the preset route. During this process, the drone will monitor its own position and posture in real time and make fine adjustments according to the actual situation to ensure the accuracy and stability of the route. Then proceed to step S233;

[0140] Step S233: During the flight, the drone automatically identifies and locks each key part, and then takes accurate photos respectively. In order to ensure the quality of the photo, the drone will automatically adjust the parameters of the gimbal camera according to factors such as the distance, angle and lighting conditions of the target to obtain the best photo effect.

[0141] Step S3 specifically includes steps S11 to S32.

[0142] Step S31: Based on deep learning model compression technology, the algorithms required for inspection, such as target recognition, real-time detection of defects and anomalies, and control, are deployed on the airborne device to realize real-time analysis and processing of data, thus saving a large amount of network bandwidth and storage problems caused by data transmission.

[0143] Based on the lightweight airborne model through deep learning algorithm model, the model is directly deployed on the airborne end and performs real-time calculations. Based on deep learning model compression technology, the target recognition, real-time detection of defects and abnormalities, and control algorithms required for inspection are deployed on the airborne end device to achieve real-time analysis and processing of data, saving a large amount of network bandwidth and storage problems caused by data backhaul.

[0144] Lightweight compression and acceleration of neural networks has become a consensus in the industry, and related algorithm research and hardware design have received great attention. This paper mainly conducts research from the direction of network low-rank decomposition technology:

[0145] Since the convolution kernel is often represented in the form of a 4-dimensional Tensor in the convolution layer, this type of solution can be regarded as the embodiment of aggregating the tensor in some dimensions and performing low-rank approximation in some dimensions. According to the specific dimensions in which the low-rank approximation is performed, we can further subdivide the solution into: 1) 2D decomposition; 2) 3D decomposition; 3) 4D decomposition. In the following description, we will use [w,h,c,n], that is, the width, height, number of channels and number of convolution kernels to describe the weight tensor in CNN.

[0146] As a representative of low-rank decomposition, SVD is directly applied in 2-D matrix decomposition. By aggregating the four-dimensional weight tensor [w,h,c,n] into [w,h,c]*n, the matrix is ​​decomposed into the product of the matrix of [w,h,c]*d and d*[1,1,n], where d represents the rank of the matrix.

[0147] This solution achieves a theoretical speedup of 3x on VGG-16, but with some performance loss. Based on the above 2D decomposition strategy, it can be seen that the computational reduction of the first convolution operation is not obvious, which leads to a 3D decomposition strategy for further decomposing the first convolution operation. On this basis, the first 4D tensor is decomposed into three groups of convolutions: [w,1,c,d1]; [1,h,d1,d2]; [1,1,d2,d].

[0148] In this way, a theoretical 4x speedup is achieved on VGG-16 with almost no performance loss. If the original convolution operation is decomposed in the dimension of the number of input channels, the Tucker decomposition form is obtained: [1,1,c,d1]; [w,h,d1,d2]; [1,1,d2,d].

[0149] Furthermore, the scheme achieves a theoretical speedup of 4.93x on VGG-16 with a slight performance loss. A scheme is proposed to approximate the original 4D tensor by summing sub-tensors that are formally the same as Tucker decomposition. If the sub-matrices of the scheme are arranged in the order of Tucker decomposition, it can be regarded as replacing the second tensor block in Tucker decomposition with a block diagonalized tensor.

[0150] Finally, the scheme achieved a theoretical speedup of 12.2x on VGG-16 with a slight performance loss. Each dimension of the 4-dimensional tensor was decomposed to obtain the approximate form of the multiplication of the four sub-tensors: [1,1,c,d1]; [w,1,d1,d2]; [h,1,d2,d3]; [1,1,d3,d]. Due to the large approximation error of the scheme, it only caused a 1% performance loss when accelerating the shallow convolutional neural network by 4.5x.

[0151] Step S31 specifically includes the following steps S311 to S316:

[0152] Step S311: training the server-side power grid equipment detection model, and then proceeding to step S312;

[0153] Step S312: Pruning, model distillation and fine-tuning the power grid equipment detection model, and then proceeding to step S313;

[0154] Step S313: quantize after training using the Bit-split technology, or use perceptual quantization to quantize during the fine-tuning of the power grid equipment detection model to obtain a lightweight model, and then proceed to step S314;

[0155] Step S314: The corresponding tool chain of the final deployed chip converts the lightweight model into a format that meets the requirements of the airborne device deployment algorithm, and then proceeds to step S315;

[0156] Step S315: deploy the lightweight model algorithm on the airborne end and verify it by actual flight of the UAV.

[0157] Step S32: gradually establish a defect library review mechanism, quickly accumulate defect data through human-computer interaction, establish a defect sample library, perform type analysis on the accumulated defect sample library, determine the number of defects of different defect types and whether they are suitable for deep learning recognition, iterate data and algorithms for defects suitable for deep learning recognition, and further identify defects that are not suitable for deep learning recognition in combination with device targets and global semantic analysis, and finally determine the defect diagnosis algorithm.

[0158] At present, autonomous inspection of power transmission lines by drones has certain shooting standards and specifications.

[0159] (1) Shooting Instructions:

[0160] 1) Shooting content: overall view of the tower, tower head, tower body, tower base, pole number plate, insulators, various hanging points, hardware, and passages.

[0161] 2) Shooting principles: facing the large side, first left then right, from bottom to top (the opposite side from top to bottom), first small side then large side.

[0162] (2) Shooting location

[0163] The overall picture of the tower, tower head, tower body, tower base, number plate, ground wire, passage, cross arm end hanging point, insulator, and conductor end hanging point.

[0164] (3) Examples of common shooting point descriptions

[0165] The overall view of the tower, tower head, tower body, tower base, tower number plate, left and right hanging points of the ground wire, I string cross arm end hanging point, I string insulator string, I string conductor end hanging point, V string left insulator cross arm end hanging point, left return upper phase cross arm end hanging point, left return upper phase small side jumper insulator cross arm end hanging point, left return upper phase large side cross arm end hanging point, etc.

[0166] (4) Naming rules

[0167] Voltage level + line + tower number + phase side.

[0168] According to the State Grid photography specifications, detailed inspections of pole towers require taking photos of the entire tower, tower head, tower base, passages, and connection hardware, insulators, ancillary facilities, shock-absorbing hammers, and spacers on the towers. The above-mentioned component categories are collected and labeled. Based on the existing inspection images, the target detection sample set is labeled through labelimg. It can be found that different photographed parts can avoid different defects, which can also reduce the false alarm rate of defect identification and, in a sense, provide a recall rate for identification. For example, for channel identification, other defect types outside the channel category can be removed. When shooting the tower base, only tower base and signboard defects appear, and other defects can be avoided.

[0169] In addition, there are many types of defects in the equipment in the power transmission and transformation lines. If we want to perform intelligent detection one by one, the workload is very large. We need to build an effective defect library, especially the high-quality images obtained by autonomous inspection. We need to determine which defect types can be trained using deep learning methods to complete defect detection in one go, which can recognize different defects through learning with small samples, and thus generalize and identify other different types of defects, and which need to locate the location of the equipment for further defect analysis.

[0170] like Fig. 9As shown in the figure, currently, drone and helicopter inspections have acquired a massive amount of data, but the defect database data is very small. We need to gradually establish a defect database review mechanism, quickly accumulate defect data through human-computer interaction, and establish a defect sample library, which will be very useful for the next step of defect analysis and identification. We need to analyze the accumulated defect database by type, determine the number of defects of different defect types and whether they are suitable for deep learning identification, iterate the data and algorithms for defects suitable for deep learning identification, and further identify defects that are not suitable for deep learning identification by combining equipment target identification and global semantic analysis, and finally determine the defect diagnosis algorithm.

[0171] A device for edge intelligent visual inspection by drones in remote areas, including an edge intelligent terminal and a drone, a CUDA environment is deployed on the edge intelligent terminal, a defect detection program is written based on the python language and the pytorch deep learning framework, and a GPU is used for model acceleration. The hardware integration structure of the edge intelligent terminal is as follows: Fig.10 shown.

[0172] Edge intelligent terminal devices are used for:

[0173] 1. Identification of typical power targets: high-precision identification of insulator hanging points, conductor hanging points, insulators, connecting fittings, and suspension clamps;

[0174] 2. It can identify and track the wires.

[0175] 3. Processing speed: power fittings target recognition speed 15FPS, wire recognition speed 10FPS;

[0176] 4. Control capability: The gimbal can be controlled to perform operations such as target alignment, focus pulling, and focusing; the aircraft can be controlled to perform dynamic route planning.

[0177] 5. Communication capability: Integrated 4G / 5G communication module, which can transmit video images and alarm information to the back-end platform in real time

[0178] The edge intelligent terminal hardware device connects to the drone through the serial port to control the drone; it connects to the aircraft gimbal through the USB port to transmit the camera data to the edge intelligent terminal for identification.

[0179] The drone consists of a mobile terminal, an onboard computer terminal, and a drone payload terminal. The communication process between them is as follows: Fig.11 shown.

[0180] Applications developed based on DJIOSDK can communicate with mobile apps developed based on MSDK and payloads developed based on PSDK. For example, users can send control instructions to drones from mobile apps developed based on MSDK to control payloads and onboard computers to perform specified tasks. Applications developed based on OSDK can control payloads to perform required actions and send status information to mobile apps developed based on MSDK. Payloads developed based on PSDK can send data such as video streams or files to mobile apps developed based on MSDK and applications developed based on OSDK.

[0181] The preferred embodiments of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above embodiments, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

[0182] Many other changes and modifications may be made without departing from the concept and scope of the present invention.It should be understood that the present invention is not limited to the specific embodiments, and the scope of the present invention is defined by the appended claims.

Claims

1. A method for edge intelligent visual inspection of unmanned aerial vehicles in remote areas, characterized in that: Applicable to areas with complex terrain, lack of RTK base stations or weak network signals, drones are used to inspect target power grid areas including power grid facilities and power lines, including the following steps: Step S1: Based on the monocular vision of the airborne edge device, three-dimensional reconstruction of the power towers and power lines in the target power grid area is performed in real time, specifically including the following steps S11 to S12: Step S11: Based on the analysis of heterogeneous power tower components, a power tower model is established based on the ontology to provide accurate information for the identification of power towers and the positioning of key checkpoints, and then enter step S12, Step S12: A three-dimensional model of the power line is established in combination with the airborne edge device. In the stage where the posture changes slowly, a direct method is used to build the map. At the same time, the colinear and coplanar attributes of the features are supplemented in the dense map generated by the direct method. In the stage where the posture changes rapidly, a feature-based method is used to build the map. Step S2: The drone autonomously and dynamically plans the inspection route based on machine vision, which specifically includes steps S21 to S23. Step S21: Based on existing technical research and corresponding point cloud scanning data, typical tower data training is performed to form a tower library of different specifications and voltage levels; Step S22: Combined with the dead reckoning algorithm, before takeoff, the inspection route is automatically generated on site according to different tower types, and the inspection route is predicted through point cloud display according to the longitude, latitude and height information of the tower, realizing automatic recognition of the tower shape and automatic generation of shooting points. Step S23: Set the inspection shooting viewpoint, that is, determine the hovering position of the drone; plan the inspection route; execute automatic flight; the drone flies to the shooting viewpoint and hovers, the front-end intelligent recognition module identifies the key parts of the tower, determines the position of the key parts of the tower, plans the order of key part recognition, focuses and takes pictures in sequence; enters the next shooting viewpoint until the shooting is completed; Step S3: Perform real-time detection and analysis based on the key photographing part information of the airborne edge device, specifically including steps S31 to S32. Step S31: Based on the deep learning model compression technology, the algorithms required for inspection, such as target recognition, real-time detection of defects and anomalies, and control, are deployed on the airborne device to realize real-time analysis and processing of data, thus saving a large amount of network bandwidth and storage problems caused by data transmission. Step S32: gradually establish a defect library review mechanism, quickly accumulate defect data through human-computer interaction, establish a defect sample library, perform type analysis on the accumulated defect sample library, determine the number of defects of different defect types and whether they are suitable for deep learning recognition, iterate data and algorithms for defects suitable for deep learning recognition, and further identify defects that are not suitable for deep learning recognition in combination with device targets and global semantic analysis, and finally determine the defect diagnosis algorithm.

2. A method for edge intelligent visual inspection of unmanned aerial vehicles in remote areas as claimed in claim 1, characterized in that: The step S11 includes the following steps S111 to S114: Step S111: analyzing basic components of the power tower according to the power tower design standard; Step S112: analyzing key characteristics of different types of power pole tower components, and establishing a property set of the power pole tower components for accurately describing and recording key characteristics of the power pole tower components; Step S113: Analyze the association relationship and constraint rules between different types of components and define them in detail, and establish an association relationship model and corresponding constraint rules for the connection between various components of the corresponding power tower; Step S114: Constructing a knowledge base of power tower components based on the ontology: Step S115: constructing and identifying a power pole tower model based on the power pole tower components.

3. A method for edge intelligent visual inspection of unmanned aerial vehicles in remote areas as claimed in claim 2, characterized in that: The step S114 includes steps S1141 to S1145; Step S1141: analyzing important terms in power tower components; Step S1142: define power tower component classes and class hierarchies: Step S1143: define the properties of each power tower component class: Step S1144: define facets of power tower component attributes: Step S1145: Create a corresponding power pole tower component instance.

4. The method for edge intelligent visual inspection of unmanned aerial vehicles in remote areas as claimed in claim 1, characterized in that: In the step S23, during the automatic flight of the drone, the relative relationship between GPS and key points is used to dynamically plan the autonomous inspection route, which specifically includes the following steps S231 to S232: Step S231: Use the onboard pan-tilt camera to visually locate and correct the power tower and the surrounding environment, and then proceed to step S232; Step S232: According to the result of visual positioning, the aircraft is controlled to move according to a preset route, and then the process proceeds to step S233; Step S233: During the flight, the drone automatically identifies and locks each key part, and then takes precise photos respectively.

5. The method for edge intelligent visual inspection of unmanned aerial vehicles in remote areas as claimed in claim 1, characterized in that: The step S31 specifically includes the following steps S311 to S316: Step S311: training the server-side power grid equipment detection model, and then proceeding to step S312; Step S312: Pruning, model distillation and fine-tuning the power grid equipment detection model, and then proceeding to step S313; Step S313: quantize after training using the Bit-split technology, or use perceptual quantization to quantize during the fine-tuning of the power grid equipment detection model to obtain a lightweight model, and then proceed to step S314; Step S314: The corresponding tool chain of the final deployed chip converts the lightweight model into a format that meets the requirements of the airborne terminal device deployment algorithm, and then proceeds to step S315; Step S315: deploy the lightweight model algorithm on the airborne end and verify it by actual flight of the UAV.

6. The method for edge intelligent visual inspection of unmanned aerial vehicles in remote areas as claimed in claim 1, characterized in that: In step S12, the direct method for mapping needs to extract feature points to complete loop detection and relocation functions, specifically by calculating the Shi-Tomas corner point score and selecting the point category as a feature point or an edge point according to a threshold.

7. A method for edge intelligent visual inspection of unmanned aerial vehicles in remote areas as claimed in claim 2, characterized in that: The basic components of the power tower include a tower crown component, a tower base component and electrical components.

8. A method for edge intelligent visual inspection of unmanned aerial vehicles in remote areas as claimed in claim 7, characterized in that: The configuration of the tower crown assembly includes a dry-shaped tower, a wine glass-shaped tower, a cat-head-shaped tower, a rotating tower, a single-circuit tower, a double-circuit tower, a two-phase double-circuit tower and a three-phase double-circuit tower; The heights of the tower base components include 5M tower feet, 10M tower feet, 15M tower feet and 20M tower feet.

9. A method for edge intelligent visual inspection of unmanned aerial vehicles in remote areas as claimed in claim 2, characterized in that: The elements of the ontology include concepts, relations, attributes, axioms, functions, and individual instances of concepts.

10. A device for edge intelligent visual inspection of unmanned aerial vehicles in remote areas, using a method for edge intelligent visual inspection of unmanned aerial vehicles in remote areas as described in any one of claims 1 to 9, characterized in that: It includes an edge intelligent terminal and a drone. The hardware device of the edge intelligent terminal is connected to the drone through a serial port to control the drone; it is connected to the aircraft gimbal through a USB port to transmit the camera data to the edge intelligent terminal for identification; The drone comprises a mobile terminal, an onboard computer terminal and a drone load terminal. The mobile terminal receives terminal instructions and controls the drone load terminal and the onboard computer to execute instruction tasks.

Images