A method for detecting missing bolt pins in transmission lines by integrating graph theory reasoning

Abstract

The present invention proposes a method for detecting missing pins in transmission line bolts that integrates graph theory reasoning. The target detection algorithm based on the detection method uses an image semantic algorithm to extract structural semantic features of images, learns the spatial relationship of transmission line components and the connection relationship between components, and integrates them with the visual features output by the feature extraction network to enhance the features. A classifier and a regressor output the position of the bolts in the image and determine whether the pins are missing. The image semantic algorithm uses the Faster R-CNN framework as a basis to add a GRM module and construct a GR-RCNN algorithm to enhance the target detection algorithm's ability to recognize small bolt-like targets in the image. When determining whether the pins are missing, bolt-like images with intact pins are used as positive samples, and the Fastflow algorithm is used to determine whether the bolts in the image to be detected are missing pins. The present invention can improve the detection accuracy of small targets and solve the problem of insufficient defect samples for bolts with missing pins.

Description

Technical Field

[0001] The present invention relates to the technical fields of machine learning, image processing and unmanned aerial vehicle (UAV) power transmission line inspection, and in particular to a method for detecting missing pins in transmission line bolts by integrating graph theory reasoning. Background Art

[0002] The existing technical solution is to strengthen the algorithm's ability to recognize small targets by using a classic object detection algorithm framework (such as Faster R-CNN), employing multi-scale strategies, data augmentation, and improved feature resolution. A generative adversarial network is then used to generate new defect samples to train the defect detection model, enabling defect detection of bolts with missing pins.

[0003] However, existing defect detection generally uses target detection algorithms to find targets in aerial images and then conducts defect detection on them, or directly uses target detection algorithms to perform defect detection on components. Existing target detection algorithms have high accuracy in detecting larger components on transmission lines, such as insulators, equalizing rings, and shock-absorbing hammers, and can meet the requirements of intelligent detection. However, small targets such as bolts are generally difficult to detect by target detection algorithms in aerial images. In addition, most current algorithms perform independent detection on each component, while ignoring the connection relationship between components on the tower, and rarely use business knowledge related to transmission line inspections. At the same time, the problem of insufficient samples of bolt missing pin defects will greatly affect the accuracy of defect detection for deep learning algorithms that require a large number of samples. Therefore, the technical problem to be solved by this application proposal is to improve the accuracy of small target detection and solve the problem of insufficient samples of bolt missing pin defects. Summary of the Invention

[0004] The present invention proposes a method for detecting missing pins in transmission line bolts by integrating graph theory reasoning, which can improve the detection accuracy of small targets and solve the problem of insufficient defect samples for missing pins in bolts.

[0005] The present invention adopts the following technical solutions.

[0006] A method for detecting missing pins in power transmission line bolts that integrates graph theory reasoning. This method uses an object detection algorithm that extracts structural semantic features from images using an image semantic algorithm. It then learns the spatial relationships between transmission line components and the connectivity between them. This is then combined with visual features output by a feature extraction network to enhance the features. A classifier and regressor then output the bolt locations in the image and determine whether the pins are missing.

[0007] The image semantic algorithm uses the Faster R-CNN framework as a foundation to incorporate the GRM module and construct the GR-RCNN algorithm. This enhances the target detection algorithm's ability to identify small bolt-like targets in images. This module is used, but not limited to, on Faster R-CNN and can also be used with other target detection algorithms. When determining whether a pin is missing, images of bolts with intact pins are used as positive samples, and the Fastflow algorithm is used to determine whether the bolts in the image to be detected are missing.

[0008] The detection method includes the following process:

[0009] Process 1: First, aerial images and image data are input into the structural semantic expression model for training. The trained model outputs semantic features that can express the spatial structure contained in the image.

[0010] Process 2: The output of the structural semantic expression model and the aerial image are input into the object detection model for recognition detection to determine whether there are bolts in the image;

[0011] Process 3: After determining that there are bolts in the image, a defect detection algorithm is used to detect whether the bolts in the image are missing pins.

[0012] The method adopted in the process 1 and the process 2 specifically includes the following steps:

[0013] Step S1, creating an image dataset of power inspection transmission line components;

[0014] Step S2: Extract structural semantic features. The GRM module (Graph Reasoning Module) is used as the structural semantic expression model of the feature extraction network. The input of the model is the visual features of the image. The output graph data of the model contains image feature data with spatial structure and component connection relationship. The nodes of the graph data are the image features and image labels of each component, and the edges are the connection relationships. The formula is expressed as

[0015] P(S=(V,E,O,R)|I) Formula 1;

[0016] Where I represents the image, V is the node corresponding to the image features of each component in I, Represents the relationship between nodes, that is, the edge in the graph data, O represents the label of each component, and R represents the label of each connection relationship; the GRM module includes ORPObject Region Proposal (target candidate region), RsPN (Relationship Proposal Network) and Graph Labeling (graph data labeling), which is expressed as follows:

[0017] P(S|I)=P(V|I)P(E|V,I)P(R,O|V,E,I) Formula 2;

[0018] The structural semantic expression model is embedded in Faster-RCNN to form the GR-RCNN algorithm (Graph Reasoning Region Convolutional Neural Networks). The GR-RCNN algorithm includes a one-stage algorithm and a two-stage algorithm. It enhances and improves the final detection effect by integrating the image visual features of the structural semantic expression model.

[0019] In step S3, ORP uses the Faster R-CNN framework to extract a set of n object proposals from the input image. Each object proposal corresponds to a target node and contains a spatial region. The pooled feature vector The label distribution of the initial predictions in the class C = {1, ..., k} The vector set of all proposals is represented as matrix R o ∈R n×4 , X o ∈R n×d , P o ∈R n×|C| ;

[0020] Step S4: introducing the RsPN network to learn and predict the correlation between the transmission line components in the tower; performing a sparse operation on the structural semantic model by pruning unrelated edges to retain valid edges and suppress the noise generated by unrelated edges;

[0021] Initialize class distribution P o , there are a total of n*(n-1) pairs of relationships in different directions Calculate its correlation Where f(x, y) is a multilayer perceptron that can output a correlation score, and is expressed as an asymmetric kernel function:

[0022]

[0023] Among them, φ(x) and ψ(y) are the projection functions of the subject and object in the relationship respectively, which are composed of two multi-layer perceptrons with the same structure. The score matrix S={s ij} n×n ;Finally, the sigmoid activation function is used to map each element of the score matrix S to [0, 1];

[0024] Step S5: sort the processed score matrix S in descending order, select the top K pairs, and then use NMS (NonMaximum Suppression) to filter out object pairs that have obvious overlap with other objects;

[0025] The formula for calculating the degree of overlap between two relation pairs {u, v} and {p, q} is:

[0026]

[0027] The character I represents the intersection and U represents the union. After removing the relationship pairs with high overlap, the remaining m relationship pairs are all valid relationships E. Finally, we can get the graph data G = (V, E), which is the graph data after the sparse operation.

[0028] Step S6: Construct an AGCN (Attention-based Graph Convolutional Network) to learn the structural features and contextual information of graph data. The AGCN consists of two layers of GCN networks and an attention module. The layer-by-layer propagation of the network is as follows: for the target node i in the graph data, the trained hidden weight matrix W is used to calculate the adjacent nodes {z i ∈R d} performs a linear transformation, sets a weight α, and finally passes through the activation function ReLU; the entire layer-by-layer propagation process is expressed as follows:

[0029]

[0030] Or equivalently as a matrix representation:

[0031]

[0032] where α i ∈[0, 1] n ;

[0033] Step S7: The connection relationship of the graph data in GCN is known and α is predicted based on node features. i , is α i Add a trainable attention module, formulated as

[0034]

[0035] where w h and W a is a learning parameter, and the expression in the form of [x, y] is a connection operation. Let α ii =1,α ij =0, The attention module is a function of learning node features, and each iteration will cause the value to change, thus affecting subsequent iterations. The formula is

[0036] α i =softmax(u i ) Formula 8;

[0037] Step S8, calculate the loss function, specifically: Formula 2 expresses that GR-RCNN is divided into three parts, P(V|I) uses the same loss function as RPN in FasterR-CNN; P(E|V, I) uses binary cross entropy loss and anchor regression loss function on RsPN; P(R, O|V, E, I) uses two multi-class cross entropy loss functions for classification;

[0038] Step S9: Enhance the image features based on the graph data. After building the semantic structure model through the above steps, the visual features extracted by Faster R-CNN are fused with the structural features of the graph data. The formula is as follows:

[0039] f′=εfW e Formula 9;

[0040] ε is the adjacency matrix representing the nodes of the graph data, f is the visual feature extracted by Faster R-CNN, and W e ∈R n×d is the weight matrix;

[0041] Step S10: Input the enhanced fusion feature f′ into the bounding box regression layer and classification layer of the Faster R-CNN network to obtain the recognition result of whether the bolt exists;

[0042] The method adopted in process three includes the following steps;

[0043] Step S11: Detecting the area where the bolt is located based on the recognition result of step S10, and obtaining a bolt image after cropping; the bolt image is used to input a defect detection model to detect whether the bolt is missing a pin;

[0044] Step S12: establishing a defect detection model, wherein the network structure of the defect detection model is a feature extractor and a FastFlow module;

[0045] Step S13: The feature extractor directly uses the feature map extracted from the last layer of the first three blocks of the CNN network ResNet, performs a multi-scale convolution operation, and then inputs it into the FastFlow module;

[0046] Step S14: The FastFlow module is composed of multiple reversible transformation blocks of 3×3 and 1×1 sizes, as shown in the following formula:

[0047]

[0048]

[0049] f1, f2, f3…f k There are K transform blocks. The transform process of each transform block is as follows:

[0050] y a ,y b =split(y)

[0051] y′ a =y a

[0052] y′ b =s(y a )⊙y b +b(y a )

[0053] y′=concat(y′ a , y′ b ),

[0054] Formula 11;

[0055] Where s(y a ) and b(y a ) are the outputs of the two neural networks; the split(·) and concat(·) functions perform splitting and concatenation operations according to the channel dimension;

[0056] The two subnets s(·) and b(·) convert the original normalized flow into a two-dimensional manner. A two-dimensional convolutional layer is used in the default subnet to preserve the spatial information in the flow model and the loss function is adjusted accordingly. The model adopts a fully convolutional network in which 3×3 convolutions and 1×1 convolutions appear alternately to preserve spatial information in the FastFlow model.

[0057] In step S15 and step S14, f:X→Z represents the image feature x∈p x (x) is projected to the hidden variable z∈p z (z); for this function, the distribution of the model on X is defined by the transformation formula of the variable:

[0058] The log-likelihood estimation function value of the feature graph is calculated according to the following formula:

[0059]

[0060]

[0061] Among them z~N(o,I) and The Jacobian determinant of f is z = f θ (x), θ is the parameter of the f model;

[0062] An image with a bolt with a missing pin is an abnormal image. Its features exceed the distribution range. The likelihood value of the abnormal image is lower than that of the normal image, and its likelihood value can be used as the anomaly score. The specific operation is: the binary probability of each channel is added to obtain the final probability map, and it is upsampled to the pixels of the input image using bilinear interpolation. The final location where the abnormal pixels gather is the location of the missing bolt defect in the image.

[0063] The detection method is executed by a computer having a GPU computing device.

[0064] The image to be detected is an aerial image taken when a drone performs intelligent inspection on a power transmission line.

[0065] The present invention fully utilizes the spatial structure of transmission line components and the connection characteristics between components to improve the detection accuracy of bolts, making it more targeted at power inspection scenarios and solving the problem of insufficient number of defective samples.

[0066] The present invention can improve the small target detection accuracy for UAV power inspection scenarios and solve the problem of insufficient defect samples for bolts with missing pins.

[0067] The present invention proposes a method for detecting missing pins in tower bolts based on an image semantic algorithm. To address the problem of small bolt targets, a method is proposed to extract structural semantic features of images using an image semantic algorithm, and the method is applied to a target detection algorithm, so that the target detection algorithm can learn the spatial relationship of transmission line components and the connection relationship between components, and use this relationship feature to assist in finding the position of the bolts. The beneficial effects of the present invention are: the method is relatively flexible and can be used in any mainstream target detection algorithm. To address the problem of the small number of bolt missing pin samples, a detection algorithm using industrial anomalies is proposed, which only inputs positive samples to detect whether the bolt image is missing pins. The spatial structure of transmission line components and the connection characteristics between components are fully utilized to improve the algorithm's recognition accuracy for small targets. At the same time, the problem of lack of defective samples is also solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0069] Attachment Figure 1 It is a schematic flow diagram of the present invention;

[0070] Attachment Figure 2 It is a structural diagram of the GR-RCNN algorithm;

[0071] Attachment Figure 3It is a structural diagram of the algorithm used in the bolt missing pin detection method. DETAILED DESCRIPTION

[0072] As shown in the figure, a method for detecting missing bolt pins on transmission lines that integrates graph theory reasoning is described. The method's target detection algorithm uses an image semantic algorithm to extract structural semantic features from images, learns the spatial relationships between transmission line components and the connectivity between them, and integrates these features with the visual features output by the feature extraction network to enhance the features. A classifier and regressor output the bolt locations in the image and determine whether the pins are missing.

[0073] The image semantic algorithm uses the Faster R-CNN framework as a basis to add the GRM module and construct the GR-RCNN algorithm. This strengthens the target detection algorithm's ability to recognize small bolt-like targets in the image. When determining whether a pin is missing, bolt images with intact pins are used as positive samples, and the Fastflow algorithm is used to determine whether the bolts in the image to be detected are missing.

[0074] The detection method includes the following process:

[0075] Process 1: First, aerial images and image data are input into the structural semantic expression model for training. The trained model outputs semantic features that can express the spatial structure contained in the image.

[0076] Process 2: The output of the structural semantic expression model and the aerial image are input into the object detection model to perform recognition detection to determine whether there are bolts in the image;

[0077] Process 3: After determining that there are bolts in the image, a defect detection algorithm is used to detect whether the bolts in the image are missing pins.

[0078] The method used in process 1 and process 2 specifically includes the following steps:

[0079] Step S1, creating an image dataset of power inspection transmission line components;

[0080] Step S2: Extract structural semantic features. The GRM module (Graph Reasoning Module) is used as the structural semantic expression model of the feature extraction network. The input of the model is the visual features of the image. The output graph data of the model contains image feature data with spatial structure and component connection relationship. The nodes of the graph data are the image features and image labels of each component, and the edges are the connection relationships. The formula is expressed as

[0081] P(S=(V,E,O,R)|I) Formula 1;

[0082] Where I represents the image, V is the node corresponding to the image features of each component in I, Represents the relationship between nodes, that is, the edge in the graph data, O represents the label of each component, and R represents the label of each connection relationship; the GRM module includes ORPObject Region Proposal (target candidate region), RsPN (Relationship Proposal Network) and Graph Labeling (graph data labeling), which is expressed as follows:

[0083] P(S|I)=P(V|I)P(E|V,I)P(R,O|V,E,I) Formula 2;

[0084] The structural semantic expression model is embedded in Faster-RCNN to form the GR-RCNN algorithm (Graph Reasoning Region Convolutional Neural Networks). The GR-RCNN algorithm includes a one-stage algorithm and a two-stage algorithm. It enhances and improves the final detection effect by integrating the image visual features of the structural semantic expression model.

[0085] The specific algorithm structure of the GR-RCNN algorithm is as follows Figure 2 As shown;

[0086] In step S3, ORP uses the Faster R-CNN framework to extract a set of n object proposals from the input image. Each object proposal corresponds to a target node and contains a spatial region. The pooled feature vector The label distribution of the initial predictions in the class C = {1, ..., k} The vector set of all proposals is represented as matrix R o ∈R n×4 , X o ∈R n×d , P o ∈R n×|C| ;

[0087] Step S4: introducing the RsPN network to learn and predict the correlation between the transmission line components in the tower; performing a sparse operation on the structural semantic model by pruning unrelated edges to retain valid edges and suppress the noise generated by unrelated edges;

[0088] Initialize class distribution P o , there are a total of n*(n-1) pairs of relationships in different directions Calculate its correlation Where f(x, y) is a multilayer perceptron that can output a correlation score, and is expressed as an asymmetric kernel function:

[0089]

[0090] Among them, φ(x) and ψ(y) are the projection functions of the subject and object in the relationship respectively, which are composed of two multi-layer perceptrons with the same structure. The score matrix S={s ij} n×n ;Finally, the sigmoid activation function is used to map each element of the score matrix S to [0, 1];

[0091] Step S5: sort the processed score matrix S in descending order, select the top K pairs, and then use NMS (NonMaximum Suppression) to filter out object pairs that have obvious overlap with other objects;

[0092] The formula for calculating the degree of overlap between two relation pairs {u, v} and {p, q} is:

[0093]

[0094] The character I represents the intersection and U represents the union. After removing the relationship pairs with high overlap, the remaining m relationship pairs are all valid relationships E. Finally, we can get the graph data G = (V, E), which is the graph data after the sparse operation.

[0095] Step S6: Construct an AGCN (Attention-based Graph Convolutional Network) to learn the structural features and contextual information of graph data. The AGCN consists of two layers of GCN networks and an attention module. The layer-by-layer propagation of the network is as follows: for the target node i in the graph data, the trained hidden weight matrix W is used to calculate the adjacent nodes {z i ∈R d} performs a linear transformation, sets a weight α, and finally passes through the activation function ReLU; the entire layer-by-layer propagation process is expressed as follows:

[0096]

[0097] Or equivalently as a matrix representation:

[0098]

[0099] where α i ∈[0, 1] n ;

[0100] Step S7: The connection relationship of the graph data in GCN is known and α is predicted based on node features. i , is α iAdd a trainable attention module, formulated as

[0101]

[0102] where w h and W a is a learning parameter, and the expression in the form of [x, y] is a connection operation. Let α ii =1,α ij =0, The attention module is a function of learning node features, and each iteration will cause the value to change, thus affecting subsequent iterations. The formula is

[0103] α i =softmax(u i ) Formula 8;

[0104] Step S8, calculate the loss function, specifically: Formula 2 expresses that GR-RCNN is divided into three parts, P(V|I) uses the same loss function as RPN in FasterR-CNN; P(E|V, I) uses binary cross entropy loss and anchor regression loss function on RsPN; P(R, O|V, E, I) uses two multi-class cross entropy loss functions for classification;

[0105] Step S9: Enhance the image features based on the graph data. After building the semantic structure model through the above steps, the visual features extracted by Faster R-CNN are fused with the structural features of the graph data. The formula is as follows:

[0106] f′=εfW e Formula 9;

[0107] ε is the adjacency matrix representing the nodes of the graph data, f is the visual feature extracted by Faster R-CNN, and W e ∈R n×d is the weight matrix;

[0108] Step S10: Input the enhanced fusion feature f′ into the bounding box regression layer and classification layer of the Faster R-CNN network to obtain the recognition result of whether the bolt exists;

[0109] The method adopted in process three includes the following steps;

[0110] Step S11: Detecting the area where the bolt is located based on the recognition result of step S10, and obtaining a bolt image after cropping; the bolt image is used to input a defect detection model to detect whether the bolt is missing a pin;

[0111] Step S12: establishing a defect detection model, wherein the network structure of the defect detection model is a feature extractor and a FastFlow module;

[0112] The network structure of the defect detection model, that is, the algorithm structure of the bolt missing pin detection method, such as Figure 3 As shown;

[0113] Step S13: The feature extractor directly uses the feature map extracted from the last layer of the first three blocks of the CNN network ResNet, performs a multi-scale convolution operation, and then inputs it into the FastFlow module;

[0114] Step S14: The FastFlow module is composed of multiple reversible transformation blocks of 3×3 and 1×1 sizes, as shown in the following formula:

[0115]

[0116]

[0117] f1, f2, f3…f k There are K transform blocks. The transform process of each transform block is as follows:

[0118] y a ,y b =split(y)

[0119] y′ a =y a

[0120] y′ b =s(y a )⊙y b +b(y a )

[0121] y′=concat(y′ a , y′ b ),

[0122] Formula 11;

[0123] Where s(y a ) and b(y a ) are the outputs of the two neural networks; the split(·) and concat(·) functions perform splitting and concatenation operations according to the channel dimension;

[0124] The two subnets s(·) and b(·) convert the original normalized flow into a two-dimensional manner. A two-dimensional convolutional layer is used in the default subnet to preserve the spatial information in the flow model and the loss function is adjusted accordingly. The model adopts a fully convolutional network in which 3×3 convolutions and 1×1 convolutions appear alternately to preserve spatial information in the FastFlow model.

[0125] In step S15 and step S14, f:X→Z represents the image feature x∈p x (x) is projected to the hidden variable z∈p z (z); for this function, the distribution of the model on X is defined by the transformation formula of the variable:

[0126] The log-likelihood estimation function value of the feature graph is calculated according to the following formula:

[0127]

[0128]

[0129] Among them z~N(o,I) and The Jacobian determinant of f is z = f θ (x), θ is the parameter of the f model;

[0130] An image with a bolt with a missing pin is an abnormal image. Its features exceed the distribution range. The likelihood value of the abnormal image is lower than that of the normal image, and its likelihood value can be used as the anomaly score. The specific operation is: the binary probability of each channel is added to obtain the final probability map, and it is upsampled to the pixels of the input image using bilinear interpolation. The final location where the abnormal pixels gather is the location of the missing bolt defect in the image.

[0131] The detection method is executed by a computer having a GPU computing device.

[0132] The image to be detected is an aerial image taken when a drone performs intelligent inspection on a power transmission line.

Claims

1. A method for detecting missing bolt pins in power transmission lines using graph theory reasoning, characterized by: The target detection algorithm based on the detection method uses an image semantic algorithm to extract structural semantic features of the image, learn the spatial relationship between transmission line components and the connection relationship between components, and fuse them with the visual features output by the feature extraction network to enhance the features. The classifier and regressor output the position of the bolts in the image and determine whether the pins are missing. The image semantic algorithm uses the Faster R-CNN framework as the basis to add the GRM module and construct the GR-RCNN algorithm. When determining whether the pins are missing, the image of the bolt with the pins not missing is used as the positive sample, and the Fastflow algorithm is used to determine whether the bolts in the image to be detected are missing. Specifically include the following steps: Step S1, creating an image dataset of power inspection transmission line components; Step S2: Extract structural semantic features. The GRM module is used as the structural semantic expression model of the feature extraction network. The input of the model is the visual features of the image. The graph data output by the model contains image feature data with spatial structure and component connection relationship. The nodes of the graph data are the image features and image labels of each component, and the edges are the connection relationships. The formula is expressed as P(S=(V,E,O,R)|I) Formula 1; Where I represents the image, V is the node corresponding to the image features of each component in I, Represents the relationship between nodes, that is, the edge in the graph data, O represents the label of each component, and R represents the label of each connection relationship; The GRM module includes the target candidate region ORP, the relationship candidate network RsPN and the graph data labeling Graph Labeling, which is expressed as follows: P(S|I)=P(V|I)P(E|V,I)P(R,O|V,E,I) Formula 2; The structural semantic expression model is embedded in Faster-RCNN to form the GR-RCNN algorithm (Graph Reasoning Region Convolutional Neural Networks), which includes a one-stage algorithm and a two-stage algorithm; In step S3, ORP uses the Faster R-CNN framework to extract a set of n object proposals from the input image, each object proposal corresponds to a target node r i o , expressed as a spatial region The pooled feature vector The label distribution of the initial predictions in the class C = {1, ..., k} The vector set of all proposals is represented as matrix R o ∈R n×4 , X o ∈R n×d , p o ∈R n×|C| ; Step S4: introducing the RsPN network to learn and predict the correlation between the transmission line components in the tower; performing a sparse operation on the structural semantic model by pruning unrelated edges; Initialize class distribution P o , there are a total of n*(n-1) pairs of relationships in different directions Calculate its correlation Where f(x, y) is a multilayer perceptron that outputs the relevance score, and is expressed as an asymmetric kernel function: Among them, φ(x) and ψ(y) are the projection functions of the subject and object in the relationship respectively, which are composed of two multi-layer perceptrons with the same structure. The score matrix S={s ij } n×n ;Finally, the sigmoid activation function is used to map each element of the score matrix S to [0, 1]; Step S5: sort the processed score matrix S in descending order, select the top K pairs, and then use NMS (NonMaximumSuppression) to filter out object pairs that have obvious overlap with other objects; The formula for calculating the degree of overlap between two relation pairs {u, v} and {p, q} is: The character I represents intersection and U represents union. After removing the relationship pairs with high overlap, the remaining m relationship pairs are all valid relationships E. Finally, the graph data G = (V, E) is obtained, which is the graph data after the sparse operation. Step S6: Construct an AGCN to learn the structural features and contextual information of graph data. The AGCN consists of two layers of GCN networks and an attention module. The layer-by-layer propagation of the network is: for the target node i in the graph data, it is represented as {z i ∈R d }, the adjacent nodes z are trained by the hidden weight matrix W j |j∈N(i) undergoes linear transformation, sets a weight α, and finally passes through the activation function ReLU; the entire layer-by-layer propagation process is expressed as follows: Or it will be equivalent to a matrix Z (l) express: where α i ∈[0, 1] n ; Step S7: The connection relationship of the graph data in GCN is known, α i Based on node feature prediction, it is α i Add a trainable attention module, formulated as where w h and W a is a learning parameter, and the expression in the form of [x, y] is a connection operation. Let α ii =1,α ij =0, The attention module is a function of learning node features, and the formula is α i =softmax(u ij ) Formula 8; Step S8, calculate the loss function, specifically: Formula 2 expresses that GR-RCNN is divided into three parts, P(V|I) uses the same loss function as RPN in FasterR-CNN; P(E|V, I) uses binary cross entropy loss and anchor regression loss function on RsPN; P(R, O|V, E, I) uses two multi-class cross entropy loss functions for classification; Step S9: Enhance the image features based on the graph data. After building the semantic structure model through the above steps, the visual features extracted by FasterR-CNN are fused with the structural features of the graph data. The formula is as follows: f′=εfW e Formula 9; ε is the adjacency matrix representing the nodes of the graph data, f is the visual feature extracted by FasterR-CNN, and W e ∈R n×d is the weight matrix; Step S10: Input the enhanced fusion feature f′ into the bounding box regression layer and classification layer of the FasterR-CNN network to obtain the recognition result of whether the bolt exists.

2. The method for detecting missing bolt pins in power transmission lines using graph theory reasoning according to claim 1 is characterized by: The method adopted in process three includes the following steps; Step S11: Detecting the area where the bolt is located based on the recognition result of step S10, and obtaining a bolt image after cropping; the bolt image is used to input a defect detection model to detect whether the bolt is missing a pin; Step S12: establishing a defect detection model, wherein the network structure of the defect detection model is a feature extractor and a FastFlow module; Step S13: The feature extractor directly uses the feature map extracted from the last layer of the first three blocks of the CNN network ResNet, performs a multi-scale convolution operation to obtain X, and inputs it into the FastFlow module; Step S14: The FastFlow module is composed of multiple reversible transformation blocks of 3×3 and 1×1 sizes, as shown in the following formula: f1, f2, f3…f k There are K transform blocks; the transform process of each transform block is as follows: and a ,and b =split(y) and' a =and a and' b =s(and a )☉and b +b(and a ) y′=concat(y′ b ,and' b ), Formula 11; s(y a ) and b(y a ) are the outputs of the two neural networks; the split(·) and concat(·) functions perform splitting and concatenation operations according to the channel dimension; The two subnets s(·) and b(·) convert the original normalized flow into a two-dimensional manner. A two-dimensional convolutional layer is used in the default subnet to preserve the spatial information in the flow model and the loss function is adjusted accordingly. The model adopts a fully convolutional network in which 3×3 convolutions and 1×1 convolutions appear alternately to preserve spatial information in the FastFlow model. In step S15 and step S14, f:x→z represents the image feature x∈p x (x) is projected to the hidden variable z∈p z (z); for this function, the distribution of the model on x is defined by the transformation formula of the variable: The log-likelihood estimation function value of the feature graph is calculated according to the following formula: Among them z~N(o,I) and is the Jacobian determinant of f, z=f θ (x), θ is the parameter of the f model; An image with a bolt with a missing pin is an abnormal image. Its features exceed the distribution range, and the likelihood value of the abnormal image is lower than that of the normal image. Its likelihood value is used as the anomaly score. The specific operation is: the binary probability of each channel is added to obtain the final probability map, and it is upsampled to the pixels of the input image using bilinear interpolation. The final location where the abnormal pixels gather is the location of the missing bolt defect in the image.

3. The method for detecting missing bolt pins in power transmission lines using graph theory reasoning according to claim 1 is characterized by: The detection method is executed by a computer having a GPU computing device.

4. The method for detecting missing bolt pins in power transmission lines using graph theory reasoning according to claim 1 is characterized by: The image to be detected is an aerial image taken when a drone performs intelligent inspection on a power transmission line.

Images