A transmission line hardware based on orientation inference graph convolution detection method and system

Abstract

The application relates to a power transmission line hardware map convolution detection method and system based on orientation reasoning, which comprises the following steps: obtaining a basic feature map based on a feature extraction network model; performing adaptive learning on the basic feature map based on an orientation prior matrix to obtain a regional orientation adjacency matrix; taking the basic features in the basic feature map as nodes and taking the regional orientation adjacency matrix as the connection relationship between the nodes to construct a spatial connection graph; inputting the spatial connection graph into a graph convolution neural network to perform orientation reasoning and obtain an enhanced feature map; cascading the enhanced feature map to the basic feature map; and inputting the cascaded feature map into a full connection layer to identify the hardware type and orientation. The application takes the orientation prior matrix as prior knowledge of different hardware spatial structures, constructs a spatial connection graph, and designs a graph convolution neural network to adopt an orientation reasoning mode to fuse the spatial connection graph into the detection and identification process, thereby improving the precision of power transmission line hardware detection.

Description

Technical Field

[0001] This invention relates to the field of power transmission line testing technology, and in particular to a method and system for detecting power transmission line fittings using convolution based on orientation reasoning. Background Technology

[0002] Transmission lines, as a crucial link in the power grid, play a vital role in the safety and development of the grid. Fittings, being the most common and frequently used components on transmission lines, have a significant impact on the normal and stable operation of the entire transmission line. Fittings are often prone to various problems such as cracking, falling off, and rusting due to harsh external environments, thus affecting their normal function. Therefore, regularly inspecting transmission lines for defects and the condition of various fittings, and promptly repairing or replacing them, will effectively reduce the frequency of transmission line failures.

[0003] The rapid development of image processing technology, combined with drone photography, has significantly improved the effectiveness of power transmission line fitting inspection. Currently, power transmission line fitting inspection technologies can be broadly categorized into three types: those based on traditional image processing algorithms, those based on machine learning algorithms, and those based on deep learning algorithms. Traditional image processing algorithms typically involve multiple stages of image processing, including feature extraction, image segmentation, and target localization, utilizing descriptions of features such as shape, edges, and contours to extract features from fittings. While traditional image processing algorithms achieve good results under experimental conditions, they are often affected by the external environment in practical applications, and can only identify fittings with prominent features, making it difficult to meet actual needs. Machine learning algorithms, which combine feature extraction with classifiers for fitting identification, are more reliable than traditional image processing algorithms to some extent. However, the features constructed by the model are easily affected by background, target material, shape, size, and other factors, making it difficult to deeply mine information from sample features.

[0004] Deep learning methods have received widespread attention since their inception, and using them for the localization and detection of hardware datasets offers advantages such as less influence of hyperparameters on the results, stronger feature extraction capabilities, and greater robustness against interference. For example, consider the following two approaches:

[0005] Solution 1: To address the problem of low accuracy in hardware inspection due to large variations in hardware dimensions and the fact that some hardware components are small-scale targets, a method for detecting typical hardware components and some defects based on an improved Cascade R-CNN is proposed, which effectively improves the accuracy of hardware inspection.

[0006] Option 2: To address the multi-angle requirements and limited sample size in detecting vibration dampers and wire clamps, an improved training method based on transfer learning is proposed to train the YOLOv3 model, thereby improving the model's final accuracy and generalization ability.

[0007] However, the above schemes one and two are improvements made by combining relevant models with the inherent characteristics of the target, without incorporating knowledge from the power field. As a result, the above models often cannot achieve good detection results in situations with dense fittings. Summary of the Invention

[0008] In view of this, the present invention provides a method for detecting transmission line fittings based on orientation reasoning convolution, which provides a detection method that can integrate knowledge in the power field and improve the detection accuracy in cases of dense fittings.

[0009] To achieve the above objectives, the present invention provides the following solution:

[0010] A method for detecting transmission line fittings based on orientation reasoning using convolutional mapping, the method comprising the following steps:

[0011] Aerial images of hardware are input into a feature extraction network model for feature extraction to obtain basic feature maps;

[0012] Based on the orientation prior matrix, adaptive learning is performed on the basic feature map to obtain the regional orientation adjacency matrix;

[0013] A spatial connectivity graph is constructed by using the basic features in the basic feature map as nodes and the regional orientation adjacency matrix as the connection relationship between nodes.

[0014] The spatial connectivity graph is input into a graph convolutional neural network for orientation reasoning to obtain an enhanced feature map.

[0015] The enhanced feature map is concatenated onto the base feature map to obtain a concatenated feature map;

[0016] The cascaded feature map is input into the fully connected layer for identification of fitting type and orientation.

[0017] Optionally, the step of adaptively learning the basic feature map based on the azimuth prior matrix to obtain the regional azimuth adjacency matrix specifically includes:

[0018] The distance between the basic feature vectors in the basic feature map is calculated using the following formula, and a distance matrix is ​​constructed.

[0019]

[0020] Where, d ij f is the distance between the i-th and j-th basic feature vectors. i Let i be the i-th basic feature vector. This is the vector obtained by expanding the dimensions of the j-th basic feature vector and transposing the first two dimensions.

[0021] The distance matrix is ​​input into an adaptive convolutional activation network to obtain the predicted edges for the current iteration;

[0022] Based on the predicted edges of the current iteration and the expected edges in the orientation prior matrix, the loss function value of the current iteration is calculated using the following formula;

[0023]

[0024] Among them, Loss e This represents the value of the loss function; Let i be the predicted edge between the i-th basic feature vector and the j-th basic feature vector. Let be the (i, j)th element in the orientation prior matrix, used to represent the expected edge between the i-th and j-th basic eigenvectors. The value of is equal to the divergence between the i-th row vector and the j-th row vector in the relative category orientation probability map, and N represents the number of basic feature vectors, the value of which is equal to the number of hardware categories;

[0025] Update the parameters of the adaptive convolutional activation network based on the loss function value of the current iteration, return to the step "input the distance matrix into the adaptive convolutional activation network to obtain the predicted edges of the current iteration", until the iteration termination condition is met, and output the predicted edges obtained in the last iteration;

[0026] The predicted edges obtained in the last iteration are normalized using the softmax function, and the normalized predicted edges are used as elements to construct the regional orientation adjacency matrix.

[0027] Optionally, the step of adaptively learning the basic feature map based on the azimuth prior matrix to obtain the regional azimuth adjacency matrix further includes:

[0028] Aerial images of hardware with known types and locations are obtained as samples to construct a sample set;

[0029] Based on the sample set, the probability of category y appearing in the s orientation relative to category x is calculated using the following formula, and a relative category orientation probability map is constructed.

[0030]

[0031] Among them, P s (C y |C x Let y be the probability of category y appearing relative to category x at position s, where s∈{0,1,2,3}, 0,1,2,3 are up, down, left, and right respectively, x,y=1,2,3,…,M, and M is the number of hardware categories; N s (C y |C xN represents the number of times category y appears in the s-position of category x. s' (C y |C x ) represents the number of times category y appears in the s' position of category x;

[0032] The divergence between any two row vectors in the relative class orientation probability map is calculated using the following formula, and the orientation prior matrix with the divergence as its elements is constructed.

[0033]

[0034] Among them, JS(R) i |R j R represents the divergence between the i-th and j-th row vectors in the relative class orientation probability plot. i and R j Let D represent the i-th and j-th row vectors in the relative class orientation probability map, respectively. KL (·) is the function for calculating relative entropy.

[0035] Optionally, the feature extraction network model includes a backbone network ResNet101, a region proposal network, and a region of interest pooling unit.

[0036] A transmission line fittings diagram convolution detection system based on orientation reasoning, the system comprising:

[0037] The feature extraction module is used to input aerial images of hardware into the feature extraction network model, perform feature extraction, and obtain basic feature maps.

[0038] The adaptive learning module is used to adaptively learn the basic feature map based on the orientation prior matrix to obtain the regional orientation adjacency matrix.

[0039] The spatial connectivity graph construction module is used to construct a spatial connectivity graph by taking the basic features in the basic feature graph as nodes and the regional orientation adjacency matrix as the connection relationship between the nodes.

[0040] The orientation reasoning module is used to input the spatial connectivity graph into the graph convolutional neural network for orientation reasoning to obtain an enhanced feature map;

[0041] A cascading module is used to cascade the enhanced feature map onto the base feature map to obtain a cascaded feature map.

[0042] The identification module is used to input the cascaded feature map into the fully connected layer for identification of the hardware type and orientation.

[0043] Optionally, the adaptive learning module specifically includes:

[0044] The distance calculation submodule is used to calculate the distance between the basic feature vectors in the basic feature map using the following formula, and to construct a distance matrix.

[0045]

[0046] Where, d ij f is the distance between the i-th and j-th basic feature vectors. i Let i be the i-th basic feature vector. This is the vector obtained by expanding the dimensions of the j-th basic feature vector and transposing the first two dimensions.

[0047] The prediction submodule is used to input the distance matrix into the adaptive convolutional activation network to obtain the predicted edges for the current iteration;

[0048] The loss function value calculation submodule is used to calculate the loss function value of the current iteration based on the predicted edge and the expected edge in the orientation prior matrix of the current iteration, using the following formula;

[0049]

[0050] Among them, Loss e This represents the value of the loss function; Let i be the predicted edge between the i-th basic feature vector and the j-th basic feature vector. Let be the (i, j)th element in the orientation prior matrix, used to represent the expected edge between the i-th and j-th basic eigenvectors. The value of is equal to the divergence between the i-th row vector and the j-th row vector in the relative class orientation probability map, and N represents the number of basic feature vectors;

[0051] The parameter update submodule is used to update the parameters of the adaptive convolutional activation network according to the loss function value of the current iteration, and return to the step "input the distance matrix into the adaptive convolutional activation network to obtain the predicted edge of the current iteration" until the iteration termination condition is met, and output the predicted edge obtained in the last iteration.

[0052] The softmax function normalization submodule is used to normalize the predicted edges obtained in the last iteration using the softmax function, and then uses the normalized predicted edges as elements to construct the regional orientation adjacency matrix.

[0053] Optionally, the system further includes:

[0054] The sample set construction module is used to acquire aerial images of hardware with known categories and locations as samples to construct a sample set;

[0055] The probability calculation module is used to calculate the probability of category y appearing in the s orientation relative to category x based on the sample set using the following formula, and to construct a relative category orientation probability map.

[0056]

[0057] Among them, P s (C y |C x Let y be the probability of category y appearing relative to category x at position s, where s∈{0,1,2,3}, 0,1,2,3 are up, down, left, and right respectively, x,y=1,2,3,…,M, and M is the number of hardware categories; N s (C y |C x N represents the number of times category y appears in the s-position of category x. s' (C y |C x ) represents the number of times category y appears in the s' position of category x;

[0058] The divergence calculation module is used to calculate the divergence between any two row vectors in the relative class orientation probability map using the following formula, and to construct an orientation prior matrix with the divergence as its element.

[0059]

[0060] Among them, JS(R) i |R j R represents the divergence between the i-th and j-th row vectors in the relative class orientation probability plot. i and R j Let D represent the i-th and j-th row vectors in the relative class orientation probability map, respectively. KL (·) is the function for calculating relative entropy.

[0061] Optionally, the feature extraction network model includes a backbone network ResNet101, a region proposal network, and a region of interest pooling unit.

[0062] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0063] This invention discloses a method and system for detecting transmission line fittings using a convolutional graph based on azimuth reasoning. The method includes the following steps: inputting aerial images of the fittings into a feature extraction network model for feature extraction to obtain a basic feature map; adaptively learning the basic feature map based on an azimuth prior matrix to obtain a regional azimuth adjacency matrix; constructing a spatial connectivity graph by using the basic features in the basic feature map as nodes and the regional azimuth adjacency matrix as the connection relationship between nodes; inputting the spatial connectivity graph into a graph convolutional neural network for azimuth reasoning to obtain an enhanced feature map; concatenating the enhanced feature map onto the basic feature map to obtain a cascaded feature map; and inputting the cascaded feature map into a fully connected layer for fitting type and azimuth identification. This invention uses the azimuth prior matrix as prior knowledge of different fitting spatial structures to construct a spatial connectivity graph, and designs a graph convolutional neural network to integrate the spatial connectivity graph into the detection and recognition process using azimuth reasoning, thereby improving the accuracy of transmission line fitting detection. Attached Figure Description

[0064] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0065] Figure 1 This is a network framework diagram for power transmission line fittings provided in an embodiment of the present invention;

[0066] Figure 2 This is a structural diagram of the hardware assembly provided in an embodiment of the present invention; Figure 2 Figures (a) and (b) in the figure are structural diagrams of the anti-galling structure and the single-span suspension structure, respectively;

[0067] Figure 3 This is a structural diagram of the planar orientation model provided in an embodiment of the present invention;

[0068] Figure 4 The relative category orientation probability map provided in the embodiments of the present invention; Figure 4 Figures (a), (b), (c), and (d) are relative category orientation probability diagrams for the four directions;

[0069] Figure 5 A flowchart of the adaptive learning process provided in an embodiment of the present invention;

[0070] Figure 6 This is a structural diagram of the adaptive convolutional activation network provided in an embodiment of the present invention. Detailed Implementation

[0071] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0072] The purpose of this invention is to provide a method for detecting transmission line fittings based on orientation reasoning through convolution, thereby providing a detection method that can integrate knowledge from the power industry and improve detection accuracy in situations with dense fittings.

[0073] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0074] To address the occlusion problem caused by densely packed hardware fittings and promote the deep integration of deep learning models with the structure of power fittings, this invention, based on UAV inspection and photography standards, conducts a top-down study of the structural rules governing the hardware fittings in power transmission lines. It proposes a relative category orientation concept for each hardware fitting and constructs a relative category orientation matrix using aerial images of power transmission lines. This matrix serves as prior knowledge of the spatial structure of different hardware fittings, and a graph convolutional fusion orientation inference module is designed and integrated into the target detection model to further improve the model's classification and localization performance.

[0075] Example 1

[0076] Embodiment 1 of the present invention provides a method for convolution detection of transmission line fitting diagrams based on orientation reasoning. The method provided in Embodiment 1 of the present invention is based on... Figure 1 The network framework shown is as follows: Figure 1As shown, this invention first uses aerial images of hardware as input, extracts features through a ResNet101 backbone network, and then obtains a base feature map through a Region Proposal Network (RPN) and a Region of Interest Pooling (RoIPooling) unit. Next, relative category orientation knowledge is extracted from the aerial image dataset and adaptively learned together with the base feature to obtain a region orientation adjacency matrix. Then, each base feature of the base feature is used as a node in a GCN graph convolutional network to form a spatial connectivity graph. The information contained in the features is considered as the signal strength of the nodes. The region orientation adjacency matrix is ​​used as the relationship in the spatial connectivity graph to establish an orientation inference model for feature aggregation and enhancement. Finally, the enhanced features generated by the orientation inference model are concatenated to the original base features to jointly complete the hardware category detection and localization task. In this invention, all network models used in the network framework are pre-trained models. The training method for the network models is as follows: Fourteen types of metal fittings were selected as experimental objects. Referring to the construction method of the PASCALVOC2007 dataset, a metal fittings dataset was constructed, with 1092 training images and 363 test images, a ratio of 3:1, containing a total of 9101 metal fitting targets. This invention uses an NVIDIA 1080Ti professional accelerator card for training and testing; the operating system is Ubuntu 16.04.6LTS, with CUDA 10.0 used for accelerated training; the computer language is Python 3.8, and the network development framework is PyTorch. During the training phase, the batch size was set to 1, and the stochastic gradient descent (SGD) algorithm was used. The initial learning rate was set to 0.0024, and after every 9 epochs, the learning rate decreased to 10% of its original value, with a maximum training iteration of 20 epochs. The enhanced feature dimension K is set to 256, and the ACAN network is scaled down using 1×1 convolutional kernels. By using GPUs, this method can be easily extended to distributed systems. The model of this invention is only suitable for hardware datasets with stable, static spatial structures. It does not significantly improve class detection performance for datasets with less spatially related structures, such as PASCALVOC2007.

[0077] The method based on the above network framework includes the following steps:

[0078] Step 101: Input the aerial image of the hardware into the feature extraction network model to extract features and obtain the basic feature map.

[0079] Step 102: Based on the azimuth prior matrix, adaptive learning is performed on the basic feature map to obtain the regional azimuth adjacency matrix.

[0080] Before performing step 102, the azimuth prior matrix must first be obtained. The method for obtaining the azimuth prior matrix includes the following steps:

[0081] Step 1: Intuitive Information about the Fittings Space

[0082] In the design and construction of power transmission lines, to ensure the safety and stability of power transmission, the hardware of power transmission lines often follows certain combination rules and structures. The fixed structure between hardware combinations indicates that they have a close orientational relationship. For example... Figure 2 As shown, the anti-galloping structure consists of a bag-type suspension clamp, a connecting plate, and a counterweight. Figure 2 (Figure (a)); a single-link suspension structure consisting of a bag-type suspension clamp, a U-shaped hanging ring, a connecting plate, and an equalizing ring. Figure 2 (See Figure (b)). There is spatial structural information about objects of one category relative to objects of another category, especially for static and stable objects like hardware. Images taken by drones show that the intuitive information about the left-right orientation between relative categories is affected by the drone's shooting angle, but the intuitive information about the up-down orientation is not affected. This allows for the effective extraction of spatial structural knowledge between hardware categories.

[0083] Step 2: Extraction of relative category location knowledge

[0084] Based on the rules of orientation, this invention proposes the concept of relative category orientation. Prior knowledge of relative category orientation is extracted through a planar orientation model, which is... Figure 3 As shown, it is defined by humans, dividing the direction of objects in the image into four directions symmetrically and proportionally, namely top(0), top(1), down(2) and down(3), which strictly conforms to the relevant habits of drone shooting, strictly distinguishing the two directions of the object up and down, while the two directions of left and right are organically combined within the same interval, so that the orientation information between relative categories can be extracted relatively effectively.

[0085] Co-occurrence relationships refer to the obvious structural symbiotic relationship between different hardware components, which often appear in the same image; for example, equalizing rings and bag-type suspension clamps often appear simultaneously. This invention innovatively extracts relative category orientation knowledge based on co-occurrence relationships. This invention uses conditional probability modeling. First, for a specific category y, it uses co-occurrence relationships to statistically analyze the number of occurrences of different categories x relative to y in different orientations, N. s (C y |C xLet y represent the number of times category y appears in position s of category x, where s∈{0,1,2,3}. The conditional probability P is obtained by dividing the number of times category y appears relative to category x in all four positions by the total number of times category y appears relative to category x. s (C y |C x ), representing the probability of category y appearing relative to category x at position s. The calculation formula is shown in equation (1):

[0086]

[0087] The relative category orientation probability map calculated according to equation (1) is as follows: Figure 4 As shown, Figure 4 Figures (a), (b), (c), and (d) are relative category orientation probability diagrams for the four directions, combined with... Figure 3 It is evident that when the equalizing ring and the counterweight appear together, the counterweight is most likely to appear at position 3 below the equalizing ring, P3(C 重锤 |C 均压环 = 0.85. Due to the influence of the hardware structure, the probability of the equalizing ring appearing below the weight is relatively reduced, P3(C) = 0.85. 均压环 |C 重锤 ) = 0.01. For P1(C 重锤 |C 均压环 )=P3(C 均压环 |C 重锤 The value of 0.85 indicates that the relative orientation knowledge graph is symmetrical across orientations. The probability of the bag-shaped suspension clamp relative to the weight in the planar orientation model is basically the same. However, due to the influence of the bag-shaped suspension clamps that assist in fixing adjacent weights, the probability of the horizontal orientation is appropriately increased. This is consistent with the hardware structure where each weight requires four bag-shaped suspension clamps for fixation and at least one equalizing ring to balance the voltage. Therefore, the relative orientation matrix can effectively express the orientational relationships between categories, helping the model learn efficient and reasonable knowledge.

[0088] Step 3: Orientation Prior Matrix

[0089] The Faster R-CNN's RPN network generates a series of regions of interest (ROIs). In this embodiment, these ROIs are used to construct a graph, perform graph reasoning, and adaptively learn the region orientation adjacency matrix. First, the N feature map base feature vectors with fixed-scale information extracted by the Faster R-CNN algorithm are used to construct the orientation map G: G=<v,ε,W> Let be the graph nodes, where v is the number of graph nodes (|v| = N), which is also the number of basic feature vectors. ε represents the graph edges, which need to be obtained through adaptive learning and expressed as node edge weights. W represents the graph propagation weights, which are also learned during training.

[0090] For the adjacency matrix of the orientation map, this invention adopts a supervised self-learning method to fuse the orientation features of the relative category structure of the hardware into the target detection model through the orientation adjacency matrix form of GCN. In order to obtain the orientation adjacency matrix, this embodiment first performs similarity calculation on the obtained relative category orientation probability map, and uses JS divergence to calculate the distance between each row vector of the orientation map from the perspective of probability distribution, enriching the expression of the knowledge graph, as shown in the following equation (2):

[0091]

[0092] Where D KL The relative entropy calculation is performed, as shown in equation (3):

[0093]

[0094] The value of the JS divergence is equal to the relative class orientation prior matrix E, according to... Figure 2 As shown, the relative orientation probability map is first similarized, and the JS divergence is calculated from the distance between the row vectors of the relative orientation probability map, as shown in formula (2). Formula (3) is based on R i and R j D in formula (2) is obtained KL .

[0095] In equations (2) and (3), R i and R j R represents the probability function of the row vectors of the orientation map. i (x)>0,R j (x) > 0. The result of JS divergence is symmetric, and its value is between 0 and 1. This yields the desparsed relative class orientation prior matrix E. (The value of JS divergence is equal to the relative class orientation prior matrix E, i.e., E = JS, and the two are equal.) This can be used as the truth value of prior knowledge and embedded into the model.

[0096] The specific implementation of step 102 of this invention is as follows:

[0097] like Figure 5 As shown, this step uses elements of the orientation prior matrix E as the expected edges. Where i,j∈[1,N], then a neural network is used to transform the features between nodes, with the aim of adaptively learning the required predicted edges. This indicates that the model has learned the directional relationship between the i-th node and the j-th node in the orientation map. To obtain the predicted edges... First, consider the basic feature vector f i f jWe perform dimensional expansion, transforming the original f∈[N,D] into [1,N,D], where D is the feature dimension of the node. Then, we transpose the first two dimensions of the expanded f to obtain f. T ∈[N,1,D], and then L1 normal form processing is performed to obtain the element distance matrix d∈[N,N,D], which is expressed by the following equation (4):

[0098]

[0099] In equation (4), the distance matrix d between the elements in the basic feature vectors reflects the similarity between elements of different basic vectors and expresses the correlation of visual feature regions. The calculated distance matrix is ​​then input into the stacked adaptive convolutional activation network (ACAN), as shown in equation (5) below.

[0100]

[0101] ACAN Figure 6 As shown, the network consists of a four-layer fully connected network, where each layer includes a convolutional layer and a Rectified Linear Unit (ReLU) activation function. ACAN is used for scaling down, inference learns the edge relationships between different nodes, and the matrix parameters α learned by ACAN are iteratively updated. This enables effective knowledge learning and allows the model to have a certain degree of adaptability to static knowledge.

[0102] In order to efficiently learn the appropriate ACAN α The matrix parameters are updated in a supervised manner to obtain the desired edge parameters. The constructed matrix represents the truth values, where... The edges are extracted using the rows and columns of the relative class orientation prior matrix E corresponding to the true class of the RPN bounding box in the Base feature. The predicted edges learned by ACAN are used as training values, and the loss function of Equation (6) is used to learn the edge weights of N candidate regions, and the network parameters are updated during training.

[0103]

[0104] To embed it into the GCN network later, the weighted form of the graph adjacency matrix is ​​used to generate edge weights, which are then normalized using the softmax function to obtain the adaptively learned region orientation adjacency matrix. As shown in equation (7), it will be integrated into the GCN network in the next step.

[0105]

[0106] Step 103: Use the basic features in the basic feature map as nodes and the regional orientation adjacency matrix as the connection relationship between nodes to construct a spatial connection graph.

[0107] Step 104: Input the spatial connectivity graph into the graph convolutional neural network for orientation reasoning to obtain an enhanced feature map.

[0108] Step 105: Concatenate the enhanced feature map onto the base feature map to obtain a cascaded feature map.

[0109] Step 106: Input the cascaded feature map into the fully connected layer to identify the type and orientation of the fittings.

[0110] In steps 103-106, to integrate the learned orientation knowledge into the target detection model, this embodiment of the invention constructs a graph convolutional fusion orientation inference module. First, a base feature map is obtained through Region Proposal Network (RPN). The base features in the base feature map form a connected graph, containing spatial information between targets. Each base feature is treated as a node in the graph convolutional network. The graph convolutional network propagates information between nodes through a region adjacency matrix, mining the spatial relationship information between target boxes, thereby assisting the model in locating and regressing targets. For the specific network framework, please refer to [reference needed]. Figure 1 .

[0111] The specific steps for target localization are as follows: First, based on the drone inspection and shooting specifications, spatial intuitive information between hardware categories is analyzed through top-down shooting standards to propose the relative category orientation concept between each hardware item, explicitly expressing the orientational relationship between hardware structures in a data-driven manner. To integrate spatial orientation information into the deep learning model, the visual features output from the Region of Interest (ROI) results are used as nodes in the spatial connectivity graph, and the adaptively learned region orientation adjacency matrix is ​​used as the relationship in the spatial connectivity graph. Then, a graph convolutional network is used to establish an orientation inference model.

[0112] To illustrate the effects of the present invention, the present invention also provides the following embodiments that illustrate the effects of the present invention.

[0113] To verify the effectiveness of the orientation reasoning module, Faster R-CNN was used as the basic framework for improvement. Faster R-CNN allows the RPN and downstream detection networks to share convolutional layers, enabling a unified deep learning-based object detection system to run at near real-time frame rates. This paper uses the mean Average Precision (mAP), a commonly used evaluation metric in current object detection models, to evaluate the model overall. The mean AP is defined as follows:

[0114]

[0115] Where C represents the number of categories of hardware, R n P represents the recall rate of category n. n (R n ) is when the recall rate of category n is R n The accuracy of the corresponding class n at that time. mAP is used to quantitatively evaluate the training effect of the model and reflects the performance of the trained model.

[0116] This invention can correctly identify the hammer under the anti-galling structure, and the introduction of GCN network and orientation knowledge module makes the positioning of the hammer more accurate.

[0117] Example 2

[0118] Embodiment 2 of the present invention provides a transmission line fitting diagram convolution detection system based on orientation reasoning, the system comprising:

[0119] The feature extraction module is used to input aerial images of hardware into the feature extraction network model for feature extraction to obtain basic feature maps. The feature extraction network model includes a backbone network ResNet101, a region proposal network, and region of interest pooling units.

[0120] The adaptive learning module is used to adaptively learn the basic feature map based on the orientation prior matrix to obtain the regional orientation adjacency matrix.

[0121] The adaptive learning module specifically includes:

[0122] The distance calculation submodule is used to calculate the distance between basic feature vectors in the basic feature map using the following formula, and to construct a distance matrix.

[0123]

[0124] Where, d ij f is the distance between the i-th and j-th basic feature vectors. i Let i be the i-th basic feature vector. This is the vector obtained by expanding the dimensions of the j-th basic feature vector and transposing the first two dimensions.

[0125] The prediction submodule is used to input the distance matrix into the adaptive convolutional activation network to obtain the predicted edges for the current iteration.

[0126] The loss function value calculation submodule is used to calculate the loss function value of the current iteration based on the predicted edge and the expected edge in the orientation prior matrix of the current iteration, using the following formula.

[0127]

[0128] Among them, Loss e This represents the value of the loss function; Let i be the predicted edge between the i-th basic feature vector and the j-th basic feature vector. Let be the (i, j)th element in the orientation prior matrix, used to represent the expected edge between the i-th and j-th basic eigenvectors. The value of is equal to the divergence between the i-th row vector and the j-th row vector in the relative class orientation probability map, and N represents the number of basic feature vectors.

[0129] The parameter update submodule is used to update the parameters of the adaptive convolutional activation network based on the loss function value of the current iteration, and return to the step "input the distance matrix into the adaptive convolutional activation network to obtain the predicted edges of the current iteration" until the iteration termination condition is met, and output the predicted edges obtained in the last iteration.

[0130] The softmax function normalization submodule is used to normalize the predicted edges obtained in the last iteration using the softmax function, and then uses the normalized predicted edges as elements to construct the regional orientation adjacency matrix.

[0131] The spatial connectivity graph construction module is used to construct a spatial connectivity graph by using basic features from the basic feature graph as nodes and the regional orientation adjacency matrix as the connection relationship between nodes.

[0132] The orientation reasoning module is used to input the spatial connectivity graph into the graph convolutional neural network for orientation reasoning to obtain an enhanced feature map.

[0133] A cascading module is used to cascade the enhanced feature map onto the base feature map to obtain a cascaded feature map.

[0134] The identification module is used to input the cascaded feature map into the fully connected layer for identification of the hardware type and orientation.

[0135] The system also includes:

[0136] The sample set construction module is used to acquire aerial images of hardware with known categories and locations as samples to construct a sample set.

[0137] The probability calculation module is used to calculate the probability of category y appearing in the s orientation relative to category x based on the sample set using the following formula, and to construct a relative category orientation probability map.

[0138]

[0139] Among them, P s (C y |C xLet y be the probability of category y appearing relative to category x at position s, where s∈{0,1,2,3}, 0,1,2,3 are up, down, left, and right respectively, x,y=1,2,3,…,M, and M is the number of hardware categories; N s (C y |C x N represents the number of times category y appears in the s-position of category x. s' (C y |C x ) represents the number of times category y appears in the s' position of category x.

[0140] The divergence calculation module is used to calculate the divergence between any two row vectors in the relative class orientation probability map using the following formula, and to construct an orientation prior matrix with the divergence as its elements.

[0141]

[0142] Among them, JS(R) i |R j R represents the divergence between the i-th and j-th row vectors in the relative class orientation probability plot. i and R j Let D represent the i-th and j-th row vectors in the relative class orientation probability map, respectively. KL (·) is the function for calculating relative entropy.

[0143] The beneficial effects of this invention are as follows:

[0144] To address the widespread problem of dense hardware targets and mutual occlusion in aerial images of power transmission lines, this paper proposes a convolution detection method for power transmission line hardware images based on orientation reasoning.

[0145] First, based on the drone aerial photography specifications, we deeply learn from the hardware dataset and design a planar orientation model to extract relative orientation knowledge between different hardware items. Then, we enrich the representation of orientation knowledge through sparsification and learn the region orientation adjacency matrix through an adaptive learning module. Next, we treat the visual features output by the Region of Interest (ROI) as nodes in the spatial connectivity graph and the region orientation adjacency matrix as the relationships in the spatial connectivity graph. We then use a graph convolutional network to build a graph convolutional fusion model based on orientation reasoning, fusing external knowledge with the deep learning model. Finally, we perform classification and regression tasks.

[0146] Experimental results show that this invention can effectively solve the problem of dense targets, and its detection performance on multi-fitting datasets of power transmission lines is superior to other advanced models. This invention uses a planar orientation model to extract relevant orientation knowledge and embeds a self-learned regional orientation matrix into the GCN network to improve the detection performance of fittings, providing a new approach for intelligent inspection technology of power transmission line equipment components.

[0147] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0148] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for detecting transmission line fittings diagrams using convolution based on orientation reasoning, characterized in that, The method includes the following steps: Aerial images of hardware are input into a feature extraction network model for feature extraction to obtain basic feature maps; Based on the orientation prior matrix, adaptive learning is performed on the basic feature map to obtain the regional orientation adjacency matrix; A spatial connectivity graph is constructed by using the basic features in the basic feature map as nodes and the regional orientation adjacency matrix as the connection relationship between nodes. The spatial connectivity graph is input into a graph convolutional neural network for orientation reasoning to obtain an enhanced feature map. The enhanced feature map is concatenated onto the base feature map to obtain a concatenated feature map; The cascaded feature map is input into a fully connected layer for identification of fitting type and orientation; The method of adaptively learning the basic feature map based on the azimuth prior matrix to obtain the regional azimuth adjacency matrix also includes: Aerial images of hardware with known types and locations are obtained as samples to construct a sample set; Based on the sample set, the probability of category y appearing in the s orientation relative to category x is calculated using the following formula, and a relative category orientation probability map is constructed. Among them, P s (C y |C x Let y be the probability of category y appearing relative to category x at position s, where s∈{0,1,2,3}, 0,1,2,3 are up, down, left, and right respectively, x,y=1,2,3,…,M, and M is the number of hardware categories; N s (C y |C x N represents the number of times category y appears in the s-position of category x. s' (C y |C x ) represents the number of times category y appears in the s' position of category x; The divergence between any two row vectors in the relative class orientation probability map is calculated using the following formula, and the orientation prior matrix with the divergence as its elements is constructed. Among them, JS(R) i |R j R represents the divergence between the i-th and j-th row vectors in the relative class orientation probability plot. i and R j Let D represent the i-th and j-th row vectors in the relative class orientation probability map, respectively. KL (·) is the function for calculating relative entropy.

2. The method for convolution detection of transmission line fitting diagrams based on orientation reasoning according to claim 1, characterized in that, The method of adaptively learning the basic feature map based on the azimuth prior matrix to obtain the regional azimuth adjacency matrix specifically includes: The distance between the basic feature vectors in the basic feature map is calculated using the following formula, and a distance matrix is ​​constructed. Where, d ij f is the distance between the i-th and j-th basic feature vectors. i Let i be the i-th basic feature vector. This is the vector obtained by expanding the dimensions of the j-th basic feature vector and transposing the first two dimensions. The distance matrix is ​​input into an adaptive convolutional activation network to obtain the predicted edges for the current iteration; Based on the predicted edges of the current iteration and the expected edges in the orientation prior matrix, the loss function value of the current iteration is calculated using the following formula; Among them, Loss e This represents the value of the loss function; Let i be the predicted edge between the i-th basic feature vector and the j-th basic feature vector. Let be the (i, j)th element in the orientation prior matrix, used to represent the expected edge between the i-th and j-th basic eigenvectors. The value of is equal to the divergence between the i-th row vector and the j-th row vector in the relative class orientation probability map, and N represents the number of basic feature vectors; Update the parameters of the adaptive convolutional activation network based on the loss function value of the current iteration, return to the step "input the distance matrix into the adaptive convolutional activation network to obtain the predicted edges of the current iteration", until the iteration termination condition is met, and output the predicted edges obtained in the last iteration; The predicted edges obtained in the last iteration are normalized using the softmax function, and the normalized predicted edges are used as elements to construct the regional orientation adjacency matrix.

3. The method for convolutional detection of transmission line fitting diagrams based on orientation reasoning according to claim 1, characterized in that, The feature extraction network model includes a backbone network ResNet101, a region proposal network, and a region of interest pooling unit.

4. A transmission line fitting diagram convolution detection system based on orientation reasoning, characterized in that, The system includes: The feature extraction module is used to input aerial images of hardware into the feature extraction network model, perform feature extraction, and obtain basic feature maps. The adaptive learning module is used to adaptively learn the basic feature map based on the orientation prior matrix to obtain the regional orientation adjacency matrix. The spatial connectivity graph construction module is used to construct a spatial connectivity graph by taking the basic features in the basic feature graph as nodes and the regional orientation adjacency matrix as the connection relationship between the nodes. The orientation reasoning module is used to input the spatial connectivity graph into the graph convolutional neural network for orientation reasoning to obtain an enhanced feature map; A cascading module is used to cascade the enhanced feature map onto the base feature map to obtain a cascaded feature map. The identification module is used to input the cascaded feature map into the fully connected layer to identify the type and orientation of the fittings; The system also includes: The sample set construction module is used to acquire aerial images of hardware with known categories and locations as samples to construct a sample set; The probability calculation module is used to calculate the probability of category y appearing in the s orientation relative to category x based on the sample set using the following formula, and to construct a relative category orientation probability map. Among them, P s (C y |C x Let y be the probability of category y appearing relative to category x at position s, where s∈{0,1,2,3}, 0,1,2,3 are up, down, left, and right respectively, x,y=1,2,3,…,M, and M is the number of hardware categories; N s (C y |C x N represents the number of times category y appears in the s-position of category x. s' (C y |C x ) represents the number of times category y appears in the s' position of category x; The divergence calculation module is used to calculate the divergence between any two row vectors in the relative class orientation probability map using the following formula, and to construct an orientation prior matrix with the divergence as its element. Among them, JS(R) i |R j R represents the divergence between the i-th and j-th row vectors in the relative class orientation probability plot. i and R j Let D represent the i-th and j-th row vectors in the relative class orientation probability map, respectively. KL (·) is the function for calculating relative entropy.

5. The transmission line fitting diagram convolution detection system based on orientation reasoning according to claim 4, characterized in that, The adaptive learning module specifically includes: The distance calculation submodule is used to calculate the distance between the basic feature vectors in the basic feature map using the following formula, and to construct a distance matrix. Where, d ij f is the distance between the i-th and j-th basic feature vectors. i Let i be the i-th basic feature vector. This is the vector obtained by expanding the dimensions of the j-th basic feature vector and transposing the first two dimensions. The prediction submodule is used to input the distance matrix into the adaptive convolutional activation network to obtain the predicted edges for the current iteration; The loss function value calculation submodule is used to calculate the loss function value of the current iteration based on the predicted edge and the expected edge in the orientation prior matrix of the current iteration, using the following formula; Among them, Loss e This represents the value of the loss function; Let i be the predicted edge between the i-th basic feature vector and the j-th basic feature vector. Let be the (i, j)th element in the orientation prior matrix, used to represent the expected edge between the i-th and j-th basic eigenvectors. The value of is equal to the divergence between the i-th row vector and the j-th row vector in the relative class orientation probability map, and N represents the number of basic feature vectors; The parameter update submodule is used to update the parameters of the adaptive convolutional activation network according to the loss function value of the current iteration, and return to the step "input the distance matrix into the adaptive convolutional activation network to obtain the predicted edge of the current iteration" until the iteration termination condition is met, and output the predicted edge obtained in the last iteration. The softmax function normalization submodule is used to normalize the predicted edges obtained in the last iteration using the softmax function, and then uses the normalized predicted edges as elements to construct the regional orientation adjacency matrix.

6. The transmission line fitting diagram convolution detection system based on orientation reasoning according to claim 4, characterized in that, The feature extraction network model includes a backbone network ResNet101, a region proposal network, and a region of interest pooling unit.

Images