Power component defect recognition detection method based on unmanned aerial vehicle inspection image positioning

Abstract

The application discloses a kind of power component defect identification detection methods based on unmanned aerial vehicle inspection image positioning, comprising: the spatial relationship model of power inspection is constructed, realize the coordinate uniformity of multi-source spatial data, wherein spatial relationship model includes power component monomer model, unmanned aerial vehicle image and image positioning information;In spatial relationship model, using power component monomer model, unmanned aerial vehicle image with positioning information as spatial reference, the initial target area of power component monomer model on image is calculated by relative position relationship;The intersection relationship of three-dimensional light ray of initial target area and spatial relationship model is calculated using ray tracing method, the projection order of initial target area is determined according to intersection relationship, and the optimization target area is screened from initial target area according to projection order;Call pre-trained defect identification network, and the power component defect identification of optimization target area is carried out, and the defect category of power component is determined.

Description

Technical Field

[0001] This invention relates to the field of intelligent power grid inspection technology, and more specifically, to an intelligent identification and detection method for defects in power components. Background Technology

[0002] The power system is an indispensable infrastructure for the lives and production activities of the people in a country, and its stable operation is essential. Power transmission lines are vast in scale, contain numerous components, and have complex structures. Long-term exposure to outdoor conditions makes them highly susceptible to various degrees of loss, damage, and breakage. Damage to some components can lead to serious consequences such as the paralysis of transmission or protection functions, including insulators and pins. Insulators are crucial components of overhead lines, primarily responsible for securing busbars and live conductors, providing sufficient conductor spacing, and providing electrical insulation. Insulators in overhead lines are prone to defects such as glaze peeling, insecure fixing, and breakage. Pins are the fasteners connecting various electrical components, playing a vital role in preventing loosening; pin detachment is one of the most common defects. To promptly detect these defects and faults, my country's power system conducts regular power line inspections, repairing faulty electrical components and clearing foreign objects accumulated on poles to ensure proper and normal operation.

[0003] The methods for inspecting power equipment are constantly improving with advancements in technology. The development of drone technology has had a significant impact on power equipment inspection methods. Compared to traditional methods, drones are more flexible and can acquire high spatial resolution and high spectral resolution data from multiple angles and close-up shots of power equipment, greatly improving the efficiency of spatial data acquisition. Power inspection methods based on drone data are faster than manual inspections, and also offer a certain level of accuracy.

[0004] Traditional image-based defect detection techniques rely solely on conventional image processing methods. This involves setting up detectors to analyze features such as color, texture, and geometry to extract defects from the complex background of electrical equipment, followed by modeling for further detection. While some researchers have explored using texture information for defect detection, this approach presents challenges. The unevenness of texture can lead to incorrect contour segmentation, resulting in inaccurate defect detection.

[0005] Machine learning algorithms analyze data, fit intermediate models, and then use the trained models to predict the results. The development of deep learning has further advanced object detection tasks, but current deep learning-based methods for object detection and defect identification in power components still struggle to accurately identify all defects. Summary of the Invention

[0006] To address at least one of the problems described in the background section, this invention provides a method for identifying and detecting defects in power components based on UAV inspection image localization.

[0007] According to one aspect of the present invention, a method for defect identification and detection of power components based on UAV inspection image localization is provided, wherein defect detection is based on a three-dimensional model, and the method includes:

[0008] A spatial relationship model for power line inspection is constructed to achieve the unification of coordinates for multi-source spatial data. The spatial relationship model includes power equipment, individual models of power equipment components, UAV image location and image attitude and positioning information. The spatial relationship model contains multi-source spatial data, which are located in their respective independent coordinate systems.

[0009] In the spatial relationship model, the individual power component model and UAV imagery with attitude and positioning information are used as spatial references. The initial target area of ​​the individual power component model on the imagery is calculated through the relative positional relationship.

[0010] The intersection relationship between the 3D rays of the initial target area and the spatial relationship model is calculated using the ray tracing method. The projection order of the initial target area is determined based on the intersection relationship, and the optimized target area is selected from the initial target area based on the projection order.

[0011] A pre-trained defect identification network is invoked to identify power component defects in the optimization target area and determine the defect category of the power component.

[0012] Optionally, the construction of a spatial relationship model for power line inspection to achieve unified coordinates of multi-source spatial data includes:

[0013] The individual model points of the power pole, which contain rich three-dimensional structural information and are based on the relative spatial coordinate system of the power pole model, are transformed and mapped to the absolute spatial coordinate system through rotation, translation, and scaling. The mapping formula is p. i =Rq i +t;

[0014] The pixels of a two-dimensional image located in the image plane coordinate system are transformed to a three-dimensional absolute space coordinate system using camera principal distance, image pose, and UAV positioning information. The mapping formula is p. i =kRq i +T;

[0015] By transforming UAV images and individual power component models to the same coordinate system, spatial guidance from 3D models to 2D images can be achieved, resulting in a spatial relationship model for power inspection.

[0016] Where, p iq represents the coordinates of a point in the absolute space coordinate system. i This represents the coordinates of the model points in the relative coordinate system. k is the scaling factor of the model. When the power pole model is obtained by the proportional method, k = 1. R and T are the registration parameters of the power pole model, including rotation and translation.

[0017] Optionally, in the spatial relationship model, using a single power component model and UAV imagery with attitude and positioning information as spatial references, the initial target area of ​​the single power component model on the imagery is calculated based on relative positional relationships, including:

[0018] The single-point perspective projection method in three-dimensional space is used to project the individual model of the power component onto the image plane coordinate system;

[0019] Transform the coordinates of the image point in the image space coordinate system to the absolute space coordinate system by rotating R and translating t.

[0020] The projection points of all individual power equipment component models within the visible range of the image are divided according to the equipment component category. After the division is completed, the projection points are aggregated by setting a preset interval threshold based on the regional distribution.

[0021] Based on the aggregated set of projection points, and following the rule of parallel to the width and height of the image, a recognition target area for a specific category of power component is constructed, which serves as the initial target area for the individual power component model on the image.

[0022] Optionally, the individual power component model is replaced by a bounding box for transformation between different spaces, and by obtaining the imaging range of a component in the image, it helps to determine whether the target has been detected, specifically including:

[0023] The bounding box used is a hexahedron larger than the volume of the individual power component model, and is the circumscribed hexahedron of the individual component; the automatic bounding box solving algorithm is as follows:

[0024] Let the point set of a single component model be P. i {(x, y, z)};

[0025] Determine the type of equipment component;

[0026] For insulator components, an outer rectangle is generated based on the position of the uppermost shed. The coordinates of the center point of the outer rectangle are then calculated based on its coordinates, which is the center point I of that shed. h (x, y, z); calculate the center point I of the lowest umbrella skirt in the same way. L (x, y, z), according to I h (x, y, z) and I L The central axis of the insulator is calculated from (x, y, z). L yL , z L ],N[A,B,C]]. Calculate the circumscribed rectangle A of the insulator skirt with the largest area. m B m C m D m To calculate the bounding box edges N by vertices i＝a，b，c，d ; passing through point I L (x, y, z) and normal vector N i Construct the bottom surface of the bounding box, face N. i ·(x L y L , z L )+d=0, passing through point I H (x, y, z) and normal vector N i Construct the top surface N of the bounding box i ·(x H y H , z H )+d=0; base and edge N i＝a，b，c，d The intersection point is the vertex A of the bounding box. L B L C L D L Top surface and edge N i＝a，b，c，d The intersection point is the vertex A of the bounding box. H B H C H D H ;

[0027] For conductor-type components, select a section of conductor in the 3D model, take the cross-section of the conductor along the direction perpendicular to the conductor, obtain the bounding rectangle of the cross-section, and construct the bounding box with the two bounding rectangles of the conductor section.

[0028] For other components, determine the set of individual model points P. i The coordinates of the maximum and minimum values ​​of the three coordinate axes Q{x max x min y max y min z max z min}; Based on the coordinates of the 6 extreme points P i＝1，2，3，4，5，6 Calculate the plane γ perpendicular to the coordinate axes. i = (Q, 0, 0)·P Q +d i (i = 1, 2, 3, 4, 5, 6); with 6 faces γ i Construct the bounding box and calculate its vertices.

[0029] Optionally, when projecting the bounding box of a single power component model onto the image plane, the points in the three-dimensional model coordinate system are projected to the two-dimensional image coordinate system through matrix transformation, including:

[0030] The projection transformation is performed only on the coordinates of the set of points along the boundary edges of the bounding box, not on the coordinates of the entire point set P of the individual object. i Transformation;

[0031] The transformation formula for the bounding box vertices is as follows:

[0032] [xy 1] T =F·P·M·[X M Y M Z M 1] T (1)

[0033] F·P·M~{R,c} (2)

[0034] Among them, [xy 1] T Let [X] be the image coordinates of the bounding box vertices. M Y M Z M 1] T The bounding box's 3D model coordinates, where F·P·M are the affine matrix, projection matrix, and model transformation matrix; and R is the orthogonal rotation matrix of the camera's exterior orientation elements, R∈R 3×3 Let c be the three-dimensional coordinate of the camera in the object coordinate system at the moment of capture, c∈R 3×1 The model points are rotated, translated, and scaled by the model matrix M to enter the image space coordinate system, and then transformed by the perspective projection matrix P. After perspective transformation, the bounding box image coordinates are obtained by affine transformation F.

[0035] By connecting the bounding box boundaries on the corresponding image, the two-dimensional image target box of the individual component can be determined.

[0036] Optionally, the step of calculating the intersection relationship between the three-dimensional rays of the initial target area and the spatial relationship model using ray tracing, determining the projection order of the initial target area based on the intersection relationship, and selecting optimized target areas from the initial target area based on the projection order includes:

[0037] Calculate the intersection point of the photographic ray P(t) = S + V with the triangular plane represented by N·(x, y, z) + d = 0; where N is the normal vector of the triangular plane; the photographic ray P(t) intersects the triangular plane if and only if N·V ≠ 0, that is, there is no parallel relationship. The values ​​of d and S can be obtained at this time. Substituting them into the photographic ray formula, the coordinates of the intersection point P(t) can be calculated.

[0038] Determine whether the intersection point is located inside the triangle;

[0039] When the intersection point is located inside the triangle, a two-dimensional projection area is obtained on the plane of the image captured by the UAV after performing a three-dimensional perspective projection on the intersection point. The coordinates of each image point in the projection area and the individual power component model corresponding to the area are known.

[0040] The projection order of all individual power component models within the projection area was calculated using ray tracing.

[0041] The set of visible points of the actual visible electrical components on the two-dimensional projection area is determined according to the order of projection. This set of visible points is the optimized identification target area.

[0042] Optionally, the step of invoking a pre-trained defect identification network to identify power component defects in the optimized target area and determine the defect category of the power component includes:

[0043] An improvement to the Faster-RCNN network is achieved by replacing the VGG network in the Faster-RCNN network with the deep residual network ResNet50.

[0044] The improved Faster-RCNN network is invoked to identify power component defects in the optimized target area and determine the defect categories of the power components.

[0045] Optionally, guided by the spatial relationship model of the power pole tower model and the image pose and positioning information, the recognition range of the defect identification network is narrowed from the entire image to a specified area, and only specific target detection algorithms need to be executed, reducing the false alarm rate. Furthermore, a deep residual network ResNet50 is used to replace the VGG network in the Faster-RCNN network to increase the richness of feature extraction. A feature pyramid network is used for multi-scale feature fusion, organically combining the resolution information of the image and the semantic information of the target. The ROIAlign pooling method is used to solve the regional mismatch problem and improve the target positioning accuracy. A recurrent generative adversarial network is used to augment small sample data, making the entire network more adaptable to the task of small target detection. Different neural networks are called for different target categories, thereby completing the automatic defect identification in power inspection tasks.

[0046] According to another aspect of the present invention, a power component defect identification and detection device based on UAV inspection image localization is provided, comprising:

[0047] The spatial relationship model construction module is used to construct a spatial relationship model for power inspection, and realize the unification of coordinates of multi-source spatial data. The spatial relationship model includes power equipment, individual models of power equipment components, UAV image location and image attitude and positioning information. The spatial relationship model contains multi-source spatial data, which are located in their respective independent coordinate systems.

[0048] The initial target area determination module is used in the spatial relationship model to calculate the initial target area of ​​the power component unit model on the image by using the individual power component model and UAV imagery with attitude and positioning information as spatial references and through relative positional relationships.

[0049] The optimized target area determination module is used to calculate the intersection relationship between the three-dimensional rays and the spatial relationship model of the initial target area using the ray tracing method, determine the projection order of the initial target area based on the intersection relationship, and select the optimized target area from the initial target area based on the projection order.

[0050] The power component defect identification module is used to call a pre-trained defect identification network to identify power component defects in the optimized target area and determine the defect category of the power component.

[0051] Optionally, the spatial relationship model construction module is specifically used for:

[0052] The individual model points of the power pole, which contain rich three-dimensional structural information and are based on the relative spatial coordinate system of the power pole model, are transformed and mapped to the absolute spatial coordinate system through rotation, translation, and scaling. The mapping formula is p. i =Rq i +t;

[0053] The pixels of a two-dimensional image located in the image plane coordinate system are transformed to a three-dimensional absolute space coordinate system using camera principal distance, image pose, and UAV positioning information. The mapping formula is p. i =kRq i +T;

[0054] By transforming UAV images and individual power component models to the same coordinate system, spatial guidance from 3D models to 2D images can be achieved, resulting in a spatial relationship model for power inspection.

[0055] Where, p i q represents the coordinates of a point in the absolute space coordinate system. i This represents the coordinates of the model points in the relative coordinate system. k is the scaling factor of the model. When the power pole model is obtained by the proportional method, k = 1. R and T are the registration parameters of the power pole model, including rotation and translation.

[0056] According to another aspect of the present invention, a computer-readable storage medium is provided, the storage medium storing a computer program for performing the methods described in any of the above aspects of the present invention.

[0057] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising: a processor; a memory for storing executable instructions of the processor; the processor being configured to read the executable instructions from the memory and execute the instructions to implement the method described in any of the preceding aspects of the present invention.

[0058] This invention first constructs a spatial relationship model consisting of a single power component model, UAV imagery, and image pose and positioning information. Guided by this model, and utilizing spatial information references provided by 3D data, the single power component model and UAV imagery are transformed into a unified coordinate reference system through multi-source coordinate space datum mapping. A 3D spatial projection is used to obtain the 2D projection region of the single power component model. Then, a ray tracing intersection algorithm is used to remove occluded areas, completing target region optimization. Finally, the target region of the power component is obtained, and a neural network is invoked to execute a target detection algorithm within this region, completing the defect identification of the power component. This invention refines the multi-target detection problem of a single 2D image into a single-target detection problem with a specified target and region, providing a new technical method for intelligent single-image recognition and improving the accuracy of image defect detection based on artificial intelligence technology. Even under poor UAV imagery shooting conditions and low target clarity, the guidance of the spatial relationship model can effectively improve the efficiency and accuracy of defect identification. Attached Figure Description

[0059] Exemplary embodiments of the present invention can be more fully understood by referring to the following figures:

[0060] Figure 1 This is a flowchart illustrating a method for identifying and detecting defects in power components based on UAV inspection image localization, provided in an exemplary embodiment of the present invention.

[0061] Figure 2 This is a flowchart of a defect identification process guided by a spatial relationship model provided in an exemplary embodiment of the present invention;

[0062] Figure 3 This is a conceptual diagram of a spatial relationship model provided by an exemplary embodiment of the present invention;

[0063] Figure 4 This is a multi-source coordinate space reference mapping relationship diagram provided by an exemplary embodiment of the present invention;

[0064] Figure 5 This is a flowchart of a multi-identification network invocation provided by an exemplary embodiment of the present invention;

[0065] Figure 6 This is a schematic diagram of the structure of a power component defect identification and detection device based on UAV inspection image localization provided in an exemplary embodiment of the present invention;

[0066] Figure 7 This is the structure of an electronic device provided in an exemplary embodiment of the present invention. Detailed Implementation

[0067] Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. It is obvious that the described embodiments are merely some embodiments of the present invention, and not all embodiments of the present invention, and it should be understood that the present invention is not limited to the exemplary embodiments described herein.

[0068] It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps described in these embodiments do not limit the scope of the invention.

[0069] With the development and rise of artificial intelligence technology, machine learning methods have gradually come into view. Machine learning algorithms analyze data, fit intermediate models, and then use the trained models to predict the results of the data. The development of deep learning has further advanced object detection tasks. Using drones to quickly, easily, and rapidly acquire high-resolution images, and then employing deep learning methods for object detection and defect identification of power components, is a relatively novel approach to power inspection. This invention aims to improve the accuracy and practicality of intelligent defect detection in existing intelligent inspection of power equipment, and solve the problem of manual review of massive inspection images.

[0070] Figure 1 This diagram illustrates the flowchart of the power component defect identification and detection method based on UAV inspection image localization provided by the present invention. Figure 1 As shown, a method for defect identification and detection of power components based on UAV inspection image localization is used. This method detects defects based on a 3D model and includes:

[0071] Step S101: Construct a spatial relationship model for power line inspection to achieve unified coordinates of multi-source spatial data. The spatial relationship model includes power equipment, individual models of power equipment components, UAV image positions and image attitude and positioning information. The spatial relationship model contains multi-source spatial data, which are located in their respective independent coordinate systems.

[0072] Optionally, the construction of the spatial relationship model for power line inspection to achieve unified coordinates of multi-source spatial data includes: transforming and mapping individual model points of power poles containing rich three-dimensional structural information in a relative spatial coordinate system based on the power pole model to an absolute spatial coordinate system through rotation, translation, and scaling, using the mapping formula p. i =Rq i +t; This transforms the pixel coordinates of a 2D image located in the image plane coordinate system to a 3D absolute space coordinate system using camera principal distance, image pose, and UAV positioning information. The mapping formula is p.i =kRq i +T; After transforming the UAV imagery and individual power component models to the same coordinate system, spatial guidance from the 3D model to the 2D imagery can be achieved, resulting in a spatial relationship model for power line inspection; where p i q represents the coordinates of a point in the absolute space coordinate system. i This represents the coordinates of the model points in the relative coordinate system. k is the scaling factor of the model. When the power pole model is obtained by the proportional method, k = 1. R and T are the registration parameters of the power pole model, including rotation and translation.

[0073] In this embodiment of the invention, the algorithm flow for identifying power component defects in UAV images guided by a spatial relationship model is as follows: Figure 2 As shown, firstly, the UAV imagery and the individual power component model are transformed into a unified coordinate system. Then, a 3D perspective projection method is used to obtain the target region of the power component on the 2D imagery. Based on this, a ray tracing method is used to calculate the actual visibility of the target region, retaining the effective area. Finally, the category label and region sub-image of the effective target region are used as input to call the corresponding category recognition network to complete the identification of target defects.

[0074] Specifically, the spatial relationship model for power line inspection is managed to unify the coordinates of multi-source spatial data. The spatial relationship model includes the following: Figure 3 As shown, the data includes 3D models of individual power components, UAV imagery, and other 3D data, along with accompanying attitude and positioning parameters, registration parameters, camera parameters, and photographic parameters. These data, unified by the coordinates and scale of the 3D information, form a strict spatial relationship. Guided by this spatial relationship model, and utilizing the spatial reference provided by the 3D data, a single image can be refined to extract the range of the target, thus refining the multi-target detection problem of a single 2D image into a single-target detection problem for a specified target and region.

[0075] In the overall architecture of the spatial relationship model for power line inspection, a three-dimensional model system for defect identification in power line inspection is constructed from three-dimensional data such as individual power component models, UAV imagery, and image attitude and positioning information. This system contains multi-source spatial data, each located in its own independent coordinate system, such as... Figure 4As shown. The smallest data unit of a single power component model is the power component unit node. Each node stores the structural and semantic information of the component. The structural information mainly consists of 3D vertices, supplemented by connecting lines or structural surfaces. This 3D structural information is usually located in a relative spatial coordinate system based on the single power component model and needs to be mapped to an absolute spatial coordinate system through transformations such as rotation, translation, and scaling. The smallest data unit of a 2D image is a pixel, located in the image plane coordinate system. Using camera principal distance, image pose, and UAV positioning information, the pixel coordinates are transformed to an absolute 3D spatial coordinate system.

[0076] To perform attitude restoration and coordinate transformation of UAV images, let the image plane coordinate system of the two-dimensional image be a left-handed system with the upper left corner of the image as the origin, the horizontal axis as the X-axis, and the vertical axis as the Y-axis. Then the image plane coordinates of each pixel are {(j,i)|j∈[1,w],i∈[1,h]}, where w and h are the width and height of the image (in pixels), respectively. The image point coordinate units are converted from pixels to millimeters (mm) and translated to a two-dimensional coordinate system with the principal point of the image as the origin, as shown in formula (1).

[0077]

[0078] Where (x,y) represents the image plane coordinates in mm, s is the pixel size, and (j,i) and (x i ,y iLet f represent the pixel coordinates of the image point and the principal point, respectively. If the principal distance of the camera is f, then in the image space coordinate system with the camera center as the origin, the image point coordinates are denoted as (x, y, -f). When a UAV with an attitude determination and positioning system captures images, it can record the camera's position and attitude at the time of capture. The position information is obtained by the GNSS positioning module, recording the longitude, latitude, and elevation at the time of capture, denoted as (lng, lat, hei). This is then converted to a geodetic coordinate system to obtain the absolute position coordinates (X, Y, Z). The attitude information is obtained by the IMU module, and common recorded information is (roll, yaw, pitch), representing roll (around the forward axis), yaw (around the vertical axis), and pitch (orthogonal to the vertical and forward directions), respectively. Since the attitude recovery process causes the coordinate axes to rotate, the order of rotation affects the final attitude recovery result. Therefore, theoretically, determining a strict spatial attitude requires four parameters. In actual sensors, for ease of recording, only three parameters are typically acquired, and the order of coordinate axis rotation is manually defined, resulting in various rotation systems such as XYZ, YXZ, and ZXY. Although different rotation systems can affect the attitude recovery results, from a numerical calculation perspective, the reason for the influence lies in the different calculation order of the rotation matrices. Therefore, in practical applications, it is only necessary to adjust the calculation position of each coordinate axis rotation matrix according to the different rotation systems to obtain the correct calculation results.

[0079] By transforming coordinates, the pose and positioning information of the image is utilized. The principal distance f and the pixel size s convert the pixel coordinates (j, i) of the image point into three-dimensional coordinates (X, φ) in the absolute space coordinate system. i ,Y i Z i This allows for the attitude restoration and coordinate transformation of UAV images.

[0080] Then, coordinate transformation of the individual power component model is performed. In the file of the individual power component model, each power component is stored in the format of a node with cascading information. In each node, the shape, position and other information of the component are recorded in the form of three-dimensional points, lines and surfaces. The coordinate reference of these three-dimensional data is usually a three-dimensional spatial coordinate system with the tower itself as the reference. Since the individual power component models in the individual power component model have relatively strict positional relationships, it is only necessary to scale, rotate and translate the coordinates proportionally to transform them into the absolute spatial coordinate system. In order to transform the three-dimensional tower model into the geodetic coordinate system, point cloud data is usually used for three-dimensional registration. If the individual power component model is built based on point cloud, the relevant registration parameters are known data. The smallest data unit of coordinate transformation is a point. All spatial structure data can use point-based coordinate transformation. The transformation process depends on the registration parameters, including scaling, rotation and translation. The calculation formula for transforming the local model point coordinates into the absolute spatial coordinate system is shown in formula (2):

[0081] p i =kRq i +T (2)

[0082] Where, p i =(X i ,Y i Z i ) T and q i =(X' i ,Y' i ,Z' i ) T These represent the coordinates of the model points in the absolute spatial coordinate system and the model's relative coordinate system, respectively. k is the model's scaling factor; when the individual power component model is obtained using a proportional method, k = 1. R and T are the registration parameters of the individual power component model, including rotation and translation. Using the registration parameters of the individual power component model, and based on single-point coordinate transformation, the coordinates of the individual power component model are transformed to the geodetic coordinate system.

[0083] Step S102: In the spatial relationship model, the individual power component model and UAV imagery with attitude and positioning information are used as spatial references to calculate the initial target area of ​​the individual power component model on the imagery through relative positional relationships.

[0084] Optionally, in the spatial relationship model, using individual power component models and UAV images with attitude and positioning information as spatial references, the initial target area of ​​the individual power component models on the image is calculated through relative positional relationships. This includes: projecting the individual power component models onto the image plane coordinate system using a three-dimensional spatial single-point perspective projection method; transforming the coordinates of image points in the image space coordinate system to the absolute space coordinate system by rotation R and translation t; dividing the projection points of all individual power component models within the visible range of the image according to equipment component categories; after the division, a preset interval threshold is set to aggregate the projection points based on the regional distribution; and based on the aggregated projection point set, a recognition target area for a specific power component category is constructed according to the rule of parallel to the width and height of the image, serving as the initial target area of ​​the individual power component models on the image.

[0085] In this embodiment of the invention, in the power inspection spatial relationship model, a single power component model and UAV imagery with attitude and positioning information are used as spatial references. The target area of ​​the single power component model on the imagery is calculated through the relative positional relationship, thereby completing the guidance.

[0086] A unified coordinate transformation is used to transform the UAV imagery and individual power component models to the same coordinate system, typically a geodetic coordinate system or a relative spatial coordinate system proportional to the actual tower scale. Then, using 3D perspective projection, the individual power component models are projected onto the image plane coordinate system of the 2D UAV imagery. All projected areas of the power components are merged into similar categories and segmented into different target regions, completing the initial extraction of the image target range. Considering the sequential occlusion relationships between the power components in the individual models, after obtaining the perspective projection target areas, ray tracing is used to calculate the sequential order of the target areas within the photographic field of view, thereby extracting the effective target areas.

[0087] The process of projecting the individual power component model onto the image plane coordinate system uses a three-dimensional single-point perspective projection method. In the individual model, the stored structural information consists of points, lines, and surfaces in space; the smallest data unit is a spatial point. When only considering the geometric region of the projection, it is only necessary to calculate the positions of all points in the two-dimensional plane and then combine these points to form an envelope, thus obtaining the outline of the projection region. Therefore, the projection of the individual power component model onto the two-dimensional image is accomplished using single-point perspective projection.

[0088] In a three-dimensional coordinate system, p = (X0, Y0, Z0) T It is a point c = (X) on the individual model of the power component. c ,Y c Z c ) T It is a photography center. These are the image pose parameters, with the rotation system being YXZ and the camera principal distance being f. Image points in the image space coordinate system can be transformed to the absolute space coordinate system by rotation R and translation t. The inverse of this process is to transform points in the three-dimensional space coordinate system to the image space coordinate system. The calculation formula is shown in formula (3):

[0089]

[0090] Where R' is the inverse transformation matrix of the image space pose recovery rotation matrix, (X'0, Y'0, Z'0). T This represents the coordinates of point p in the image space rectangular coordinate system of the single model. Then, using spatial similarity transformation, the three-dimensional coordinates can be projected onto the two-dimensional plane, as shown in formula (4):

[0091]

[0092] Where (x0, y0) T It is the model point (X0, Y0, Z0). T Projection coordinates on the image. The projection points of all individual models within the visible range of the image are divided into categories. After the division, the projection points are aggregated according to a certain interval threshold based on the regional distribution. Then, based on the aggregated point set, the identification target area of ​​a specific power component category is constructed according to the rule of parallel to the width and height of the image, thus completing the initial guidance of the spatial relationship model.

[0093] Optionally, the individual power component model is replaced by a bounding box for transformation between different spaces. Furthermore, by obtaining the imaging range of a component in the image, it helps determine whether the target has been detected. Specifically, the bounding box used is a hexahedron larger than the individual power component model, serving as the circumscribed hexahedron of the component. The automatic bounding box solving algorithm is as follows:

[0094] Let the point set of a single component model be P. i {(x,y,z)};

[0095] Determine the type of equipment component;

[0096] For insulator components, an outer rectangle is generated based on the position of the uppermost shed. The coordinates of the center point of the outer rectangle are then calculated based on its coordinates, which is the center point I of that shed. h (x, y, z); calculate the center point I of the lowest umbrella skirt in the same way. L (x, y, z), according to I h (x, y, z) and I L The central axis of the insulator is calculated from (x, y, z). L y L , zL ],N[A,B,C]]. Calculate the circumscribed rectangle A of the insulator skirt with the largest area. m B m C m D m To calculate the bounding box edges N by vertices i＝a，b，c，d ; passing through point I L (x, y, z) and normal vector N i Construct the bottom surface of the bounding box, face N. i ·(x L y L , z L )+d=0, passing through point I H (x, y, z) and normal vector Ni construct the top surface N of the bounding box. i ·(x H y H , z H )+d=0; base and edge N i＝a，b，c，d The intersection point is the vertex A of the bounding box. L B L C L D L Top surface and edge N i＝a，b，c，d The intersection point is the vertex A of the bounding box. H B H C H D H ;

[0097] For conductor-type components, select a section of conductor in the 3D model, take the cross-section of the conductor along the direction perpendicular to the conductor, obtain the bounding rectangle of the cross-section, and construct the bounding box with the two bounding rectangles of the conductor section.

[0098] For other components, determine the set of individual model points P. i The coordinates of the maximum and minimum values ​​of the three coordinate axes Q{x max x min y max y min z max z min}; Based on the coordinates of the 6 extreme points P i＝1，2，3，4，5，6 Calculate the plane γ perpendicular to the coordinate axes. i = (Q, 0, 0)·P Q +d i (i = 1, 2, 3, 4, 5, 6); with 6 faces γ i Construct the bounding box and calculate its vertices.

[0099] In this embodiment of the invention, given the viewpoint information of a single image in a unified coordinate system, perspective projection is used to project the bounding box onto the image plane. Projection is essentially the process of projecting points from the three-dimensional model coordinate system to the two-dimensional image coordinate system through matrix transformation. The transformation formula is as follows:

[0100] [xy 1] T =F·P·M·[x M Y M Z M 1] T (1)

[0101] F·P·M~{R,c} (2)

[0102] Among them, [xy 1] T Let [X] be the image coordinates of the bounding box vertices. M Y M Z M 1] T The bounding box's 3D model coordinates, where F·P·M are the affine matrix, projection matrix, and model transformation matrix; and R is the orthogonal rotation matrix of the camera's exterior orientation elements, R∈R 3×3 Let c be the three-dimensional coordinate of the camera in the object coordinate system at the moment of capture, c∈R 3×1 The model points are rotated, translated, and scaled by the model matrix M to enter the image space coordinate system, and then transformed by the perspective projection matrix P. After perspective transformation, the bounding box image coordinates are obtained by affine transformation F.

[0103] By connecting the bounding box boundaries on the corresponding image, the two-dimensional image target box of a single component can be determined. By comparing and screening this target box with the target detection candidate boxes identified by the deep learning network, component targets that the network missed can be identified. At the same time, for components with detected defects, the location of the defective component can be quickly located through this two-dimensional and three-dimensional mapping relationship, thereby improving the accuracy and efficiency of power inspection.

[0104] Step S103: Calculate the intersection relationship between the three-dimensional rays of the initial target area and the spatial relationship model using the ray tracing method, determine the projection order of the initial target area based on the intersection relationship, and select the optimized target area from the initial target area based on the projection order;

[0105] Optionally, the step of calculating the intersection relationship between the three-dimensional rays and the spatial relationship model of the initial target area using ray tracing, determining the projection order of the initial target area based on the intersection relationship, and selecting optimized target areas from the initial target area based on the projection order includes: calculating the intersection point of the photographic ray P(t) = S + V with the triangular plane represented by N·(x,y,z) + d = 0; where N is the normal vector of the triangular plane; the photographic ray P(t) intersects the triangular plane if and only if N·V ≠ 0, i.e., there is no parallel relationship. The values ​​of d and S at this time can be obtained by substituting them into the photographic ray... The formula can calculate the coordinates P(t) of the intersection point; determine whether the intersection point is located inside the triangle; when the intersection point is located inside the triangle, perform a three-dimensional perspective projection on the intersection point to obtain a two-dimensional projection area on the plane of the UAV image, where the coordinates of each image point in the projection area and the corresponding power component individual model are known; use ray tracing to calculate the sequential projection order of all power component individual models in the projection area; determine the set of visible points of the actual visible power components on the two-dimensional projection area based on the sequential projection order, and this set of visible points is the optimized identification target area.

[0106] In this embodiment of the invention, ray tracing is used to further optimize the results of the 3D perspective projection. Assuming the light source originates from the center of the image, passes through the image points on the image plane, and reaches the surface of the 3D model, the pixel values ​​corresponding to the image points are calculated based on the light source, the surface reflectance, and the atmospheric attenuation coefficient. After performing 3D perspective projection on the individual power component model, a 2D projection area is obtained on the UAV image plane. At this point, the coordinates of each image point in the projection area and the corresponding 3D individual model are known. Therefore, by calculating the sequential projection order of all 3D individual models within the projection area, the actual visible power components within that area can be determined, thus completing the optimization of the target area identification.

[0107] Step S104: Call the pre-trained defect recognition network to identify power component defects in the optimization target area and determine the defect category of the power component.

[0108] Optionally, the step of calling a pre-trained defect identification network to identify power component defects in the optimized target area and determine the defect category of the power component includes: replacing the VGG network in the Faster-RCNN network with a deep residual network ResNet50 to improve the Faster-RCNN network; calling the improved Faster-RCNN network to identify power component defects in the optimized target area and determine the defect category of the power component.

[0109] Optionally, guided by the spatial relationship model of the power pole tower model and the image pose and positioning information, the recognition range of the defect identification network is narrowed from the entire image to a specified area, and only specific target detection algorithms need to be executed, reducing the false alarm rate. Furthermore, a deep residual network ResNet50 is used to replace the VGG network in the Faster-RCNN network to increase the richness of feature extraction. A feature pyramid network is used for multi-scale feature fusion, organically combining the resolution information of the image and the semantic information of the target. The ROIAlign pooling method is used to solve the regional mismatch problem and improve the target positioning accuracy. A recurrent generative adversarial network is used to augment small sample data, making the entire network more adaptable to the task of small target detection. Different neural networks are called for different target categories, thereby completing the automatic defect identification in power inspection tasks.

[0110] In this embodiment of the invention, target recognition is fused and invoked based on an improved Faster-RCNN multi-recognition network. To address the issues of low resolution and limited features for small targets, a deep residual network ResNet50 is used to replace VGG to increase the richness of feature extraction. To address the problems of large target scale spans and multiple scales coexisting, a Feature Pyramid Network (FPN) is used for multi-scale feature fusion, organically combining image resolution information and target semantic information. To address the problem of inaccurate localization caused by small target areas, the ROIAlign pooling method is used to solve the region mismatch problem and improve target localization accuracy. A Recurrent Generative Adversarial Network (CycleGAN) is used to augment small sample data, making the entire network more adaptable to small target detection tasks. Different neural networks are invoked for different target categories, thereby completing the automatic defect identification in power line inspection tasks.

[0111] Furthermore, for scenarios where the target is identified but the specific area is unknown, multiple independent recognition networks are invoked, such as... Figure 5 As shown, different network structures have varying detection capabilities for different types of power components, and the uneven distribution of sample data leads to multiple weights for different network parameters. Different network classes are assigned different labels, indicating the types of power components they can detect, serving as an index for invocation. During the recognition task, different neural networks are invoked for the target class, executing neural network target detection algorithms on the two-dimensional image or sub-regions of the image, ultimately obtaining the target class label, thus completing the automatic defect identification process in power inspection tasks.

[0112] In summary, this invention first constructs a spatial relationship model consisting of a single power component model, UAV imagery, and image attitude and positioning information. Then, guided by this spatial relationship model, and utilizing spatial information references provided by 3D data, it transforms the single power component model and UAV imagery into a unified coordinate reference system through multi-source coordinate space datum mapping. A 3D spatial projection is used to obtain the 2D projection region of the single power component model, and then a ray tracing intersection algorithm is used to remove occluded areas, thus optimizing the target area. Finally, the target region of the power component is obtained, and a neural network is invoked to execute a target detection algorithm within this region to complete the defect identification of the power component. This invention refines the multi-target detection problem of a single 2D image into a single-target detection problem with a specified target and region, providing a new technical method for intelligent single-image recognition and improving the accuracy of image defect detection based on artificial intelligence technology. Even under poor UAV imagery shooting conditions and low target clarity, the guidance of the spatial relationship model can effectively improve the efficiency and accuracy of defect identification.

[0113] Exemplary device

[0114] Figure 6 This is a schematic diagram of the structure of a power component defect identification and detection device based on UAV inspection image localization, provided in an exemplary embodiment of the present invention. Figure 6 As shown, the device 600 includes:

[0115] The spatial relationship model construction module 610 is used to construct a spatial relationship model for power inspection and realize the unification of coordinates of multi-source spatial data. The spatial relationship model includes power equipment, individual models of power equipment components, UAV image positions and image attitude and positioning information. The spatial relationship model contains multi-source spatial data, which are located in their respective independent coordinate systems.

[0116] The initial target area determination module 620 is used to calculate the initial target area of ​​the power component unit model on the image by using the power component unit model and UAV image with attitude and positioning information as spatial references in the spatial relationship model.

[0117] The optimized target area determination module 630 is used to calculate the intersection relationship between the three-dimensional rays and the spatial relationship model of the initial target area using the ray tracing method, determine the projection order of the initial target area based on the intersection relationship, and select the optimized target area from the initial target area based on the projection order.

[0118] The power component defect identification module 640 is used to call a pre-trained defect identification network to identify power component defects in the optimized target area and determine the defect category of the power component.

[0119] Optionally, the spatial relationship model construction module 610 is specifically used for:

[0120] The individual model points of the power pole, which contain rich three-dimensional structural information and are based on the relative spatial coordinate system of the power pole model, are transformed and mapped to the absolute spatial coordinate system through rotation, translation, and scaling. The mapping formula is p. i =Rq i +t;

[0121] The pixels of a two-dimensional image located in the image plane coordinate system are transformed to a three-dimensional absolute space coordinate system using camera principal distance, image pose, and UAV positioning information. The mapping formula is p. i =kRq i +T;

[0122] By transforming UAV images and individual power component models to the same coordinate system, spatial guidance from 3D models to 2D images can be achieved, resulting in a spatial relationship model for power inspection.

[0123] Where, p i q represents the coordinates of a point in the absolute space coordinate system. i This represents the coordinates of the model points in the relative coordinate system. k is the scaling factor of the model. When the power pole model is obtained by the proportional method, k = 1. R and T are the registration parameters of the power pole model, including rotation and translation.

[0124] The power component defect identification and detection device based on UAV inspection image localization in this embodiment corresponds to the power component defect identification and detection method based on UAV inspection image localization in another embodiment of this invention, and will not be described again here.

[0125] Exemplary electronic devices

[0126] Figure 7 This is the structure of an electronic device provided in an exemplary embodiment of the present invention. For example... Figure 7 As shown, the electronic device 70 includes one or more processors 71 and a memory 72.

[0127] The processor 71 may be a central processing unit (CPU) or other form of processing unit with data processing and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.

[0128] The memory 72 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 71 may execute the program instructions to implement the methods for information mining of historical change records and / or other desired functions of the software programs of the various embodiments of the present invention described above. In one example, the electronic device may also include an input device 73 and an output device 74, these components being interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0129] In addition, the input device 73 may also include, for example, a keyboard, a mouse, etc.

[0130] The output device 74 can output various information to the outside. The output device 74 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0131] Of course, for the sake of simplicity, Figure 7 Only some of the components of the electronic device relevant to the present invention are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device may include any other suitable components depending on the specific application.

[0132] Exemplary computer program products and computer-readable storage media

[0133] In addition to the methods and apparatus described above, embodiments of the present invention may also be computer program products, which include computer program instructions that, when executed by a processor, cause the processor to perform the steps in the methods according to various embodiments of the present invention described in the "Exemplary Methods" section above.

[0134] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of the present invention. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0135] Furthermore, embodiments of the present invention may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps of the methods for information mining of historical change records according to various embodiments of the present invention as described in the "Exemplary Methods" section above.

[0136] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0137] The basic principles of the present invention have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in the present invention are merely examples and not limitations, and should not be considered as essential features of each embodiment of the present invention. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the present invention to the necessity of employing the aforementioned specific details.

[0138] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments, since they largely correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0139] The block diagrams of devices, systems, devices, and systems involved in this invention are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, systems, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0140] The methods and systems of the present invention may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of the present invention are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, the present invention may also be implemented as a program recorded on a recording medium, the program comprising machine-readable instructions for implementing the methods according to the present invention. Thus, the present invention also covers recording media storing programs for performing the methods according to the present invention.

[0141] It should also be noted that in the systems, apparatus, and methods of the present invention, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered equivalents of the present invention. The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the invention. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of the invention. Therefore, the invention is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0142] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of the invention to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. A method for defect identification and detection of power components based on UAV inspection image localization, characterized in that, This method for defect detection based on a 3D model includes: A spatial relationship model for power line inspection is constructed to achieve the unification of coordinates for multi-source spatial data. The spatial relationship model includes power equipment, individual models of power equipment components, UAV image location and image attitude and positioning information. The spatial relationship model contains multi-source spatial data, which are located in their respective independent coordinate systems. In the spatial relationship model, the individual power component model and UAV imagery with attitude and positioning information are used as spatial references. The initial target area of ​​the individual power component model on the imagery is calculated through the relative positional relationship. The intersection relationship between the 3D rays of the initial target area and the spatial relationship model is calculated using the ray tracing method. The projection order of the initial target area is determined based on the intersection relationship, and the optimized target area is selected from the initial target area based on the projection order. The pre-trained defect identification network is invoked to identify power component defects in the optimization target area and determine the defect category of the power component. In the spatial relationship model, a single power component model and UAV imagery with attitude and positioning information are used as spatial references. The initial target area of ​​the single power component model on the imagery is calculated based on relative positional relationships, including: The single-point perspective projection method in three-dimensional space is used to project the individual model of the power component onto the image plane coordinate system; The image point in the image space coordinate system is rotated Peaceful relocation Transform the coordinates to an absolute space coordinate system; The projection points of all individual power component models within the visible range of the image are divided according to the equipment component category. After the division is completed, the projection points are aggregated by setting a preset interval threshold based on the regional distribution. Based on the aggregated projection point set, and following the rule of parallel to the width and height of the image, a recognition target area for a specific category of power component is constructed, which serves as the initial target area for the individual power component model on the image. Specifically, the individual power component model is replaced by a bounding box to perform transformations between different spaces, and by obtaining the imaging range of a component in the image, it helps to determine whether the target has been detected. This includes: The bounding box used is a hexahedron larger than the volume of the individual power component model, and is the circumscribed hexahedron of the individual component.

2. The method according to claim 1, characterized in that, The construction of a spatial relationship model for power line inspection, achieving unified coordinates for multi-source spatial data, includes: The individual model points of the power pole tower, which contain rich three-dimensional structural information, are transformed and mapped to the absolute spatial coordinate system through rotation, translation, and scaling, based on the relative spatial coordinate system of the power pole tower model. The pixels of a 2D image located in the image plane coordinate system are transformed to a 3D absolute space coordinate system using camera principal distance, image pose, and UAV positioning information. The mapping formula is as follows: ; By transforming UAV images and individual power component models to the same coordinate system, spatial guidance from 3D models to 2D images can be achieved, resulting in a spatial relationship model for power inspection. in, This represents the coordinates of a point in the absolute space coordinate system. This represents the coordinates of a model point in a relative coordinate system. k K is the scaling factor for the model. When the power pole model is obtained by the proportional method, k=1. R and T are the registration parameters of the power pole model, including rotation and translation.

3. The method according to claim 1, characterized in that, The automatic bounding box solving algorithm is as follows: Let the point set of a single component model be P. i {(x,y,z)}; Determine the type of equipment component; For insulator components, an outer rectangle is generated based on the position of the uppermost shed. The coordinates of the center point of the outer rectangle are then calculated based on its coordinates, which is the center point I of that shed. h (x, y, z); Calculate the center point I of the lowest umbrella skirt in the same way. L (x,y,z), according to I h (x,y,z) and I L The central axis of the insulator is calculated using (x, y, z). ; Calculate the circumscribed rectangle A of the insulator skirt with the largest area. m B m C m D m To calculate the bounding box edges N by vertices i=a,b,c,d ; passing through point I L (x, y, z) and normal vector N i Construct the bottom surface and the surface of the bounding box. , through point I H (x, y, z) and normal vector N i Construct the top surface of the bounding box ; Bottom and edge N i=a,b,c,d The intersection point is the vertex A of the bounding box. L B L C L D L Top surface and edge N i=a,b,c,d The intersection point is the vertex A of the bounding box. H B H C H D H ; For conductor-type components, select a section of conductor in the 3D model, take the cross-section of the conductor along the direction perpendicular to the conductor, obtain the bounding rectangle of the cross-section, and construct the bounding box with the two bounding rectangles of the conductor section. For other components, determine the set of individual model points P. i The coordinates of the maximum and minimum values ​​of the three coordinate axes Q{x max x min y max y min z max z min }; Based on the coordinates of the 6 extreme points P i=1,2,3,4,5,6 Calculate the plane perpendicular to the coordinate axes (i=1,2,3,4,5,6); with 6 faces Construct the bounding box and calculate its vertices.

4. The method according to claim 1, wherein when projecting the bounding box of a single power component model onto an image plane, the points in the three-dimensional model coordinate system are projected to the two-dimensional image coordinate system through matrix transformation, comprising: The projection transformation is performed only on the coordinates of the set of points along the boundary edges of the bounding box, not on the coordinates of the entire point set P of the individual object. i Transformation; The transformation formula for the bounding box vertices is as follows: in, Let be the image coordinates of the bounding box vertices. The coordinates of the bounding box's 3D model. Let R be the affine matrix, projection matrix, and model transformation matrix; where R is the orthogonal rotation matrix of the camera's exterior orientation elements. c represents the three-dimensional coordinates of the object in the object coordinate system at the moment the camera captures the image. The model points are rotated, translated, and scaled by the model matrix M to enter the image space coordinate system, and then transformed by the perspective projection matrix P. After perspective transformation, the bounding box image coordinates are obtained by affine transformation F. By connecting the bounding box boundaries on the corresponding image, the two-dimensional image target box of the individual component can be determined.

5. The method according to claim 1, characterized in that, The process of calculating the intersection relationship between the 3D rays and the spatial relationship model of the initial target area using ray tracing, determining the projection order of the initial target area based on the intersection relationship, and selecting optimized target areas from the initial target area based on the projection order includes: Calculate photographic lighting with by The intersection of the triangular planes represented; where, N It is the normal vector of the triangular plane; if and only if That is, when there is no parallel relationship, the photographic light rays It intersects with the plane of the triangle at a point, at which time d Given the values ​​of S and S, the coordinates of the intersection point can be calculated by substituting them into the photographic ray formula. ; Determine whether the intersection point is located inside the triangle; When the intersection point is located inside the triangle, a two-dimensional projection area is obtained on the plane of the image captured by the UAV after performing a three-dimensional perspective projection on the intersection point. The coordinates of each image point in the projection area and the individual power component model corresponding to the area are known. The projection order of all individual power component models within the projection area was calculated using ray tracing. The set of visible points of the actual visible electrical components on the two-dimensional projection area is determined according to the order of projection. This set of visible points is the optimized identification target area.

6. The method according to claim 1, characterized in that, The process of calling a pre-trained defect identification network to identify power component defects in the optimized target area and determine the defect category of the power component includes: Using deep residual networks ResNet 50 replacements Faster - RCNN In the network VGG Network, to achieve Faster - RCNN Network improvements; Call the improved version Faster - RCNN The network identifies power component defects in the target area and determines the defect categories of the power components.

7. The method according to claim 1, characterized in that, Guided by the spatial relationship model of power pole towers and image pose and positioning information, the defect identification network's recognition range is narrowed from the entire image to a specified area, and it only needs to execute target detection algorithms for specific categories, reducing the false alarm rate; furthermore, a deep residual network ResNet50 is used to replace... Faster - RCNN VGG network within the network is used to enhance the richness of feature extraction; Multi-scale feature fusion is performed using a feature pyramid network to organically combine image resolution information and target semantic information; the ROI Align pooling method is used to solve the region mismatch problem and improve target localization accuracy. A recurrent generative adversarial network was used to augment small sample data, making the entire network more adaptable to small target detection tasks. Different neural networks were invoked for different target categories, thereby enabling automatic defect identification in power line inspection tasks.

8. A power component defect identification and detection device based on UAV inspection image localization, characterized in that, include: The spatial relationship model construction module is used to construct a spatial relationship model for power inspection, and realize the unification of coordinates of multi-source spatial data. The spatial relationship model includes power equipment, individual models of power equipment components, UAV image location and image attitude and positioning information. The spatial relationship model contains multi-source spatial data, which are located in their respective independent coordinate systems. The initial target area determination module is used in the spatial relationship model to calculate the initial target area of ​​the power component unit model on the image by using the individual power component model and UAV imagery with attitude and positioning information as spatial references and through relative positional relationships. The optimized target area determination module is used to calculate the intersection relationship between the three-dimensional rays and the spatial relationship model of the initial target area using the ray tracing method, determine the projection order of the initial target area based on the intersection relationship, and select the optimized target area from the initial target area based on the projection order. The power component defect identification module is used to call a pre-trained defect identification network to identify power component defects in the optimization target area and determine the defect category of the power component. In the spatial relationship model, a single power component model and UAV imagery with attitude and positioning information are used as spatial references. The initial target area of ​​the single power component model on the imagery is calculated based on relative positional relationships, including: The single-point perspective projection method in three-dimensional space is used to project the individual model of the power component onto the image plane coordinate system; The image point in the image space coordinate system is rotated Peaceful relocation Transform the coordinates to an absolute space coordinate system; The projection points of all individual power component models within the visible range of the image are divided according to the equipment component category. After the division is completed, the projection points are aggregated by setting a preset interval threshold based on the regional distribution. Based on the aggregated projection point set, and following the rule of parallel to the width and height of the image, a recognition target area for a specific category of power component is constructed, which serves as the initial target area for the individual power component model on the image. Specifically, the individual power component model is replaced by a bounding box to perform transformations between different spaces, and by obtaining the imaging range of a component in the image, it helps to determine whether the target has been detected. This includes: The bounding box used is a hexahedron larger than the volume of the individual power component model, and is the circumscribed hexahedron of the individual component.

9. The apparatus according to claim 8, characterized in that, The spatial relationship model construction module is specifically used for: The individual model points of the power pole tower, which contain rich three-dimensional structural information, are transformed and mapped to the absolute spatial coordinate system through rotation, translation, and scaling, based on the relative spatial coordinate system of the power pole tower model. The pixels of a 2D image located in the image plane coordinate system are transformed to a 3D absolute space coordinate system using camera principal distance, image pose, and UAV positioning information. The mapping formula is as follows: ; By transforming UAV images and individual power component models to the same coordinate system, spatial guidance from 3D models to 2D images can be achieved, resulting in a spatial relationship model for power inspection. in, This represents the coordinates of a point in the absolute space coordinate system. This represents the coordinates of a model point in a relative coordinate system. k K is the scaling factor for the model. When the power pole model is obtained by the proportional method, k=1. R and T are the registration parameters of the power pole model, including rotation and translation.

Images