Method and device for identifying missing bolts in power transmission lines based on graph knowledge reasoning
Through a graph-based knowledge reasoning method, using the bolt missing data set and probability matrix, co-occurrence information is spread and confidence scores are calculated to identify the missing bolts in the transmission line, and the prior art distinguishing between missing bolts and no pins is solved. The problem of bolt identification error improves recognition accuracy.
Patent Information
- Application Number
- CN202210492996.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-05-07
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2042-05-07
AI Technical Summary
The prior art is difficult to accurately distinguish between missing bolts and unpinned bolts in power transmission lines, resulting in identification errors.
Using a graph knowledge reasoning method, by obtaining the bolt missing data set and related probability matrix, the category representation vector of the metal tool picture is extracted and updated into a category representation vector, and combined with the graph convolution network to propagate co-occurrence information, calculate the confidence score to achieve the missing Identification of pin bolts.
It improves the recognition accuracy of missing bolts, reduces errors, and can accurately distinguish between missing bolts and pinless bolts in pictures taken by the drone.
Smart Images

Figure CN114898146B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of power equipment fault classification, and in particular to a method and device for identifying missing bolts of a power transmission line based on graph knowledge reasoning. Background Art
[0002] Bolts are indispensable components for fastening various hardware in power transmission lines. Bolt defects are easily caused by the outdoor environment and force. Pins are used as positioning and fixing connectors. Once they are missing, the fastened parts will become loose or missing, which will cause safety hazards. Using drones equipped with cameras to obtain aerial images of power transmission lines has gradually become the mainstream line inspection method. However, due to the small proportion of bolts in aerial images, the missing of bolt cotter pins is more difficult to identify compared with other parts of the transmission line.
[0003] The application publication number is CN 114037689 A, and the name is a semantic dictionary learning method and device for bolt defect classification. Based on the bolt defect category semantic dictionary and feature learning module, the bolt test image is classified and the bolt defect classification is obtained, so that the bolt defect classification accuracy and efficiency are high. The bolt defect category semantic dictionary and feature learning module are trained in advance, so the overall work efficiency is high.
[0004] The application publication number is CN 111210007 A, and its name is Super-resolution Bolt Image Generation Method Based on Improved SRGAN Model. By adding relative mean discriminator and other factors to the loss function of the SRGAN model, the SRGAN model is used to generate high-resolution images of bolts in power transmission lines in the combined application of autonomous inspection of power transmission lines by drones and deep learning.
[0005] Combining the above two patent documents and existing technical solutions, the inventors analyzed and found that the following technical problems exist in the existing technical solutions.
[0006] At present, there are two types of normal bolts commonly used in power transmission lines: bolts with pins and bolts without pins. When the cotter pin of a bolt with pins is missing, it will become a bolt without pins. At this time, the only difference between the bolt without pins and the bolt without pins is the pin hole. After the transmission line picture is obtained by the drone, the conventional method is to directly judge it through the existing model, but due to factors such as shooting angle and shadow, the pin hole is not clearly displayed or not displayed in the picture. The existing model only relies on surface features and cannot accurately distinguish between bolts without pins and bolts without pins, which causes errors in the identification of bolts without pins.
[0007] Existing technical problems and considerations:
[0008] How to solve the technical problem of being unable to distinguish between bolts with missing pins and bolts without pins. Summary of the invention
[0009] The technical problem to be solved by the present invention is to provide a method and device for identifying missing bolts in a power transmission line based on graph knowledge reasoning, so as to solve the technical problem that missing bolts cannot be identified.
[0010] In order to solve the above technical problems, the technical solution adopted by the present invention is: a method for identifying missing bolts in a transmission line based on graph knowledge reasoning comprises the following steps: obtaining a bolt missing data set, wherein the bolt missing data set comprises images and bolt labels, and the categories of bolt labels include missing bolts; obtaining a correlation probability matrix A s , obtain a hardware picture, obtain the class mapping and feature tensor of the bolt label category in the hardware picture, obtain the category representation vector V of the bolt label category in the hardware picture, update the category representation vector V of the bolt label category in the hardware picture to obtain the category representation vector Y, obtain the first confidence score vector S1 by global average pooling of the class mapping of the hardware picture, obtain the category score by the binary classifier, connect the category scores to obtain the second confidence score vector S2, and obtain the label prediction score S by weighted summation of the two confidence score vectors, wherein the label prediction score S includes the prediction score of the missing pin bolt. When the prediction score of the missing pin bolt is greater than 0.5, it is known that the hardware picture contains the missing pin bolt.
[0011] A further technical solution is to obtain a bolt missing pin data set and obtain a related probability matrix A s The step is a preliminary preparation step, which is specifically divided into the following steps: S1 obtains a bolt pin-missing dataset, obtains a bolt pin-missing dataset, and the bolt pin-missing dataset includes pictures and bolt labels. The categories of bolt labels include bolts with pins, bolts without pins, vertical bolts, horizontal bolts and bolts with pins missing, with a total of five types of bolt labels; S2 obtains a correlation probability matrix, obtains a training set from the bolt pin-missing dataset, counts the number of co-occurrences of bolt labels in all pictures in the training set, obtains a symmetric co-occurrence matrix N, normalizes the co-occurrence matrix N and obtains a correlation probability matrix A s .
[0012] A further technical solution is that the steps of obtaining a hardware picture and a class mapping, feature tensor, category representation vector V, category characterization vector Y and label prediction score S of the hardware picture are specific identification steps, which are specifically divided into the following steps: S3 obtains the category representation vector of the bolt in the hardware picture, passes the hardware picture through a feature extraction network and obtains a feature map, passes the feature map through a classifier and a sigmoid function in sequence to obtain a class mapping, performs channel downsampling on the feature map to obtain a feature tensor, converts the feature tensor through a class mapping and performs weighted summation to obtain a category representation vector V of the bolt label category in the hardware picture; S4 updates the category representation vector to obtain a category characterization vector, takes each category representation vector in the category representation vector V obtained in step S3 as a node, and takes the relevant probability matrix A obtained in step S2 as a node. s As an edge, the graph convolutional network is input to realize the propagation of co-occurrence information, and each category representation vector is used as a node to realize information update to obtain the category representation vector Y; S5 obtains the label prediction score, and the class mapping of the hardware picture obtained in step S3 is subjected to global average pooling to obtain the first confidence score vector S1, and the category representation vector Y is subjected to a binary classifier to obtain the category score, and the category scores are connected to obtain the second confidence score vector S2, and the two confidence score vectors are weighted and summed to obtain the label prediction score S, and the label prediction score S includes the prediction score of the pin-containing bolt, the prediction score of the pin bolt, the prediction score of the vertical bolt, the prediction score of the horizontal bolt and the prediction score of the pin-missing bolt.
[0013] A further technical solution is that: in step S3, the feature extraction network is a residual network.
[0014] A transmission line pin-missing bolt identification device based on graph knowledge reasoning includes the following program modules: a bolt pin-missing data set acquisition module, used to acquire a bolt pin-missing data set, wherein the bolt pin-missing data set includes pictures and bolt labels, and the categories of bolt labels include bolts with pins, bolts without pins, vertical bolts, horizontal bolts and pin-missing bolts, a total of five types of bolt labels; a correlation probability matrix acquisition module, used to acquire a training set from the bolt pin-missing data set, count the number of co-occurrences of bolt labels in all pictures in the training set, obtain a symmetric co-occurrence matrix N, normalize the co-occurrence matrix N and obtain a correlation probability matrix A s .
[0015] A further technical solution is that it also includes the following program modules: a module for obtaining a category representation vector of bolts in a hardware picture, which is used to pass the hardware picture through a feature extraction network and obtain a feature map, pass the feature map through a classifier and a sigmoid function in sequence to obtain a class map, downsample the feature map through channels to obtain a feature tensor, convert the feature tensor through a class map and perform weighted summation to obtain a category representation vector V of the bolt label category in the hardware picture.
[0016] A further technical solution is that it also includes the following program modules: updating the category representation vector to obtain the category representation vector module, which is used to obtain the category representation vector V obtained in the category representation vector module of the bolt in the hardware picture, and each category representation vector is a node to obtain the correlation probability matrix A obtained in the correlation probability matrix module s As the edge, the graph convolutional network is input to realize the propagation of co-occurrence information, and each category representation vector is used as a node to update the information to obtain the category representation vector Y.
[0017] A further technical solution is that it also includes the following program modules, a label prediction score obtaining module, which is used to obtain the first confidence score vector S1 by global average pooling of the class mapping of the hardware image obtained in the module for obtaining the category representation vector of the bolts in the hardware image, obtain the category score by passing the category representation vector Y through a binary classifier, connect the category scores to obtain the second confidence score vector S2, and obtain the label prediction score S by weighted summation of the two confidence score vectors, wherein the label prediction score S includes the prediction score of the pin-containing bolt, the prediction score of the pin bolt, the prediction score of the vertical bolt, the prediction score of the horizontal bolt and the prediction score of the pin-missing bolt.
[0018] A device for identifying missing bolts in a transmission line based on graph knowledge reasoning includes a memory, a processor, and a computer program stored in the memory and executable on the processor. The computer program includes the above-mentioned program module. When the processor executes the computer program, the corresponding steps in the above-mentioned method are implemented.
[0019] A device for identifying missing bolts in a transmission line based on graph knowledge reasoning includes a computer-readable storage medium, the computer-readable storage medium stores a computer program, the computer program includes the above-mentioned program module, and when the computer program is executed by a processor, the corresponding steps in the above-mentioned method are implemented.
[0020] The beneficial effects of adopting the above technical solution are:
[0021] A method for identifying missing bolts in a transmission line based on graph knowledge reasoning includes the following steps: obtaining a bolt missing data set, wherein the bolt missing data set includes images and bolt labels, and the categories of bolt labels include missing bolts; obtaining a correlation probability matrix A s, obtain a hardware picture, obtain the class mapping and feature tensor of the bolt label category in the hardware picture, obtain the category representation vector V of the bolt label category in the hardware picture, update the category representation vector V of the bolt label category in the hardware picture to obtain the category representation vector Y, obtain the first confidence score vector S1 by global average pooling of the class mapping of the hardware picture, obtain the category score by passing the category representation vector Y through a binary classifier, connect the category scores to obtain the second confidence score vector S2, and obtain the label prediction score S by weighted summation of the two confidence score vectors, the label prediction score S includes the prediction score of the missing pin bolt, when the prediction score of the missing pin bolt is greater than 0.5, it is known that the hardware picture contains the missing pin bolt. This technical solution is based on the already obtained bolt missing pin dataset and the related probability matrix A s By obtaining the hardware picture and processing it to obtain the label prediction score, when the prediction score of the missing pin bolt is greater than 0.5, it is known that the hardware picture contains the missing pin bolt, and the missing pin bolt can be identified and distinguished.
[0022] A transmission line pin-missing bolt identification device based on graph knowledge reasoning includes the following program modules: a bolt pin-missing data set acquisition module, used to acquire a bolt pin-missing data set, wherein the bolt pin-missing data set includes pictures and bolt labels, and the categories of bolt labels include bolts with pins, bolts without pins, vertical bolts, horizontal bolts and pin-missing bolts, a total of five types of bolt labels; a correlation probability matrix acquisition module, used to acquire a training set from the bolt pin-missing data set, count the number of co-occurrences of bolt labels in all pictures in the training set, obtain a symmetric co-occurrence matrix N, normalize the co-occurrence matrix N and obtain a correlation probability matrix A s The technical solution obtains the bolt shortage data set and the related probability matrix A in advance by obtaining the bolt shortage data set module and the related probability matrix A s This has made preliminary preparations and provided a foundation for being able to identify and distinguish missing pin bolts.
[0023] Please refer to the detailed description of the specific implementation method. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] Figure 1 is a flow chart of Embodiment 1 of the present invention;
[0025] Figure 2 is a principle block diagram of embodiment 2 of the present invention;
[0026] Figure 3 is a principle block diagram of embodiment 3 of the present invention;
[0027] Figure 4 is a principle block diagram of embodiment 4 of the present invention;
[0028] Figure 5 It is a screenshot of the bolt types of the bolt missing pin dataset in the present invention;
[0029] Figure 6 It is the data flow diagram of the present invention. DETAILED DESCRIPTION
[0030] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. The following description of at least one exemplary embodiment is actually only illustrative and is by no means intended to limit the present application and its application or use. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0031] In the following description, many specific details are set forth to facilitate a full understanding of the present application, but the present application may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present application. Therefore, the present application is not limited to the specific embodiments disclosed below.
[0032] Embodiment 1:
[0033] like Figure 1 As shown, the present invention discloses a method for identifying missing bolts of a transmission line based on graph knowledge reasoning, comprising the following steps:
[0034] S1 obtains bolt missing pin dataset
[0035] A bolt pin-missing dataset is obtained, wherein the bolt pin-missing dataset includes images and bolt labels. The categories of bolt labels include bolts with pins, bolts without pins, vertical bolts, horizontal bolts, and bolts with pins missing, which are five categories of bolt labels in total.
[0036] S2 obtains the relevant probability matrix
[0037] Get the training set from the bolt missing pin dataset, count the number of bolt label co-occurrences in all images in the training set, get the symmetric co-occurrence matrix N, normalize the co-occurrence matrix N and get the related probability matrix A s .
[0038] S3 obtains the category representation vector of the bolts in the hardware image
[0039] Obtain a hardware picture, input it into the feature extraction network and obtain a feature map, pass the feature map through the classifier to obtain a classified feature map, and pass the classified feature map through the sigmoid function to obtain a class map; at the same time, the feature map is passed through a convolution layer with a convolution kernel of 1×1 to perform channel downsampling to obtain a feature tensor; convert the feature tensor through the class map to obtain a feature tensor X', and perform weighted summation of the feature tensor X' to obtain a category representation vector V for each category of labels, V = [v1, v2, ..., v C ]∈R C×D , where v1 represents the vector of bolts with pins, v2 represents the vector of bolts without pins, v3 represents the vertical bolts, v4 represents the horizontal bolts, v5 represents the vector of bolts without pins, C is the total number of labels, which is 5, D is the feature dimension, and R is a C*D dimensional matrix.
[0040] The feature extraction network is a residual network.
[0041] S4 updates the category representation vector to obtain the category representation vector
[0042] Each category representation vector in the category representation vector V obtained in step S3 is used as a node, and the correlation probability matrix A obtained in step S2 is used as s As the edge, the graph convolutional network is input to realize the propagation of co-occurrence information, and each category representation vector is used as a node to update the information to obtain the category representation vector Y.
[0043] S5 obtains the label prediction score
[0044] The class map obtained in step S3 is subjected to global average pooling to obtain the first confidence score vector S1. in, represents the first confidence score of the bolt with pin, represents the first confidence score for bolts without pins, represents the first confidence score of the vertical bolt, represents the first confidence score of the horizontal bolt, The first confidence score representing the missing bolt is a decimal value ranging from 0 to 1. The category characterization vector Y obtained in step S4 is passed through a binary classifier to obtain the category score corresponding to each category, and then each category score is connected to generate a second confidence score vector S2. in, represents the second confidence score of the bolt with pin, represents the second confidence score for bolts without pins, represents the second confidence score of the vertical bolt, represents the second confidence score of the horizontal bolt, The second confidence score of the missing bolt is represented; the two confidence score vectors are weighted and summed to obtain the final label prediction score S, S = λS1 + (1-λ)S2, where λ is the weight correlation coefficient, which represents the proportion of the two confidence score vectors in the final label score S. In the present invention, the coefficient is simply set to 0.5 as the final label prediction score S = [S 1 ,S 2 ,…,S C ], where S 1 represents the prediction score of bolts with pins, S 2 represents the prediction score without pin bolts, S 3 represents the prediction score of the vertical bolt, S 4 represents the prediction score of horizontal bolts, S 5 It indicates the prediction score of the missing bolt. When c=5, the prediction result S C represents the predicted fraction of missing bolts, when S 5 When it is greater than 0.5, it indicates that the image, i.e., the hardware image, is predicted to contain a missing bolt.
[0045] Embodiment 2:
[0046] like Figure 2 As shown, the present invention discloses a device for identifying missing bolts of a power transmission line based on graph knowledge reasoning, including the following program modules:
[0047] The module for obtaining bolt pin-missing dataset is used to obtain bolt pin-missing dataset, which includes pictures and bolt labels. The categories of bolt labels include bolts with pins, bolts without pins, vertical bolts, horizontal bolts and bolts with missing pins, a total of five categories of bolt labels.
[0048] The module for obtaining the relevant probability matrix is used to obtain the training set from the bolt missing pin dataset, count the number of bolt label co-occurrences in all images in the training set, obtain the symmetrical co-occurrence matrix N, normalize the co-occurrence matrix N and obtain the relevant probability matrix A s .
[0049] The module for obtaining the category representation vector of the bolts in the hardware picture is used to pass the hardware picture through the feature extraction network and obtain the feature map, pass the feature map through the classifier and the sigmoid function in sequence to obtain the class map, downsample the feature map through the channel to obtain the feature tensor, convert the feature tensor through the class map and perform weighted summation to obtain the category representation vector V of the bolt label category in the hardware picture.
[0050] Update the category representation vector to obtain the category representation vector module, which is used to obtain the category representation vector V obtained in the category representation vector module of the bolt in the hardware picture. Each category representation vector is a node to obtain the relevant probability matrix A obtained in the relevant probability matrix module sAs the edge, the graph convolutional network is input to realize the propagation of co-occurrence information, and each category representation vector is used as a node to update the information to obtain the category representation vector Y.
[0051] A label prediction score obtaining module is used to obtain a first confidence score vector S1 by global average pooling of the class mapping of the hardware image obtained in the module for obtaining the category representation vector of the bolts in the hardware image, obtain a category score by passing the category representation vector Y through a binary classifier, connect the category scores to obtain a second confidence score vector S2, and obtain a label prediction score S by weighted summation of the two confidence score vectors. The label prediction score S includes a prediction score for a bolt containing a pin, a prediction score for a pin bolt, a prediction score for a vertical bolt, a prediction score for a horizontal bolt, and a prediction score for a bolt without a pin.
[0052] Embodiment 3:
[0053] like Figure 3 As shown, the present invention discloses a device for identifying missing bolts in a transmission line based on graph knowledge reasoning, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of Example 1 when executing the computer program.
[0054] Embodiment 4:
[0055] like Figure 4 As shown, the present invention discloses a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in Example 1 are implemented.
[0056] Purpose of this application:
[0057] The main purpose of the present invention is to propose a transmission line missing pin bolt recognition model based on graph knowledge reasoning, so as to solve the visual inseparability problem existing in the missing pin bolt recognition task and improve the missing pin bolt recognition accuracy.
[0058] The concept of this application:
[0059] After the transmission line images taken by drones are processed with image processing technology to obtain hardware images, the bolts at different positions on the hardware are used to have different orientations and installation types. The inherent combination relationship between hardware, bolt orientation and bolt type is summarized, and the relevant information of bolts with missing pins is summarized. The relevant information is input in the form of labels to realize the identification of bolts with missing pins. The present invention selects bolts on three types of hardware, namely, bag wire clamps, connecting plates and heavy hammers, as research objects. After inputting the above-mentioned types of hardware images into the model, if it is identified that the bolt is horizontally oriented and has no pin, the bolt is a bolt with missing pins. If it is identified that the bolt is vertically oriented and has no cotter pin, the bolt is a normal bolt without a pin, so that the model can realize the identification of bolts with missing pins.
[0060] Technical contributions of this application:
[0061] The present invention is achieved through the following technical solution, which comprises the following steps:
[0062] A bolt pin-missing dataset is constructed. The dataset labels are divided into five categories: bolts with pins, bolts without pins, vertical bolts, horizontal bolts, and bolts with pins missing.
[0063] The given hardware picture is input into the feature extraction network to obtain the feature map, and the feature map is processed in two ways. After classification, the class map is obtained and the channel down-sampling is performed. Then, the feature map after channel down-sampling is processed by the class map to obtain the category representation vector of each category label. At this time, each category representation vector aggregates the features with high correlation with its corresponding category.
[0064] The correlation matrix is obtained by statistically calculating the co-occurrence probability of the labels in the data set, and then normalized to obtain the correlation probability matrix. Each category representation vector obtained is used as a node, and the correlation probability matrix is used as an edge. The graph convolutional network is input to realize the propagation of co-occurrence information, and each category representation vector is used as a node to realize information update.
[0065] The obtained class map is subjected to global average pooling to obtain the confidence score vector S1. The updated category vectors are passed through a binary classifier to obtain the category score corresponding to each category. The category scores are then connected to generate a category score vector S2. The two category score vectors are weighted and summed to obtain the final label prediction score S.
[0066] Technical solution description:
[0067] Preliminary preparations:
[0068] Step 1: Use labelimg to manually label the aerial images, and use the PASCAL VOC2007 dataset construction method to construct a bolt pin-missing dataset. The dataset labels are divided into five categories: bolts with pins, bolts without pins, vertical bolts, horizontal bolts, and bolts with pins missing. (The category labels are represented by 1-5, specifically, bolts with pins are represented by 1, ..., bolts with pins missing are represented by 5, and so on)
[0069] like Figure 5 As shown in the figure, the bolt missing pin dataset contains 5 types of bolts, each of which can be labeled. The bolt missing pin dataset includes images and labels. The labels are manually labeled through labelimg. The aerial images are manually labeled through labelimg to obtain the bolt missing pin dataset.
[0070] The bolt missing pin dataset includes pictures and labels. The dataset labels have no pictures but only labels.
[0071] Step 2: First, we count the number of bolt labels co-occurring in all images in the training set to obtain a symmetrical co-occurrence matrix N. Then, for category labels i and j, we use the conditional probability formula: P ij =P(n j |n i ) represents the probability of label j appearing under the premise of category label i, thus normalizing the relevant probability matrix A s (i and j are aliases for labels and can be any value from 1 to 5 in step 1)
[0072] Specific implementation process:
[0073] like Figure 6 As shown, the data flow diagram of the model of the present invention is explained as follows.
[0074] Step 3: Input the image into the Residual Network (Resnet) Resnet-101 to obtain the feature map. First, the fully connected layer is used as a classifier to implement feature classification, and then the classified feature map is obtained through the sigmoid function to obtain the class map; at the same time, the feature map is downsampled using a convolution layer with a convolution kernel of 1×1 to reduce the channel scale and obtain the feature tensor. The feature tensor is converted into a category representation through the class mapping. By weighted summing of the feature tensor X', the model obtains each category representation V = [v1, v2, ..., v C ]∈R C×D (where v1 represents the pin-bolt vector..., C is the total number of labels, D is the feature dimension, and R is a C*D-dimensional matrix), then each category represents the ability to selectively aggregate features that are highly correlated with its corresponding category.
[0075] Step 4: Then use each category V as a node and the related probability matrix A obtained in step 2 s As an edge, it is input into the graph convolutional network (GCN). Using the relevant probability matrix A s and the weight update matrix W obtained by self-learning through gradient descent during training s To update the values of each category representation. During the training process, the activation function LeakyReLU is used to update the model through the state: Y = LReLU (A s VW s ) The relevant probability matrix As first propagates the co-occurrence graph information to all nodes, and then the nodes receive the information and update their own states after linear transformation to obtain the category representation vector Y.
[0076] Step 5: First, the class map obtained in step 3 is globally averaged pooled to obtain the first confidence score vector (in represents the confidence score of the bolt with pin, ..., represents the confidence score of the missing bolt, which is generally a decimal between 0 and 1). The category representation obtained in step 4 is put into the binary classifier to obtain the category score, and then each category score is connected to generate the second confidence score vector (in represents the confidence score of the bolt with pin, …, The two confidence scores are multiplied by the correlation coefficient respectively, and the two score vectors are aggregated by weighted summation through S = λS1 + (1-λ)S2 to obtain the final label prediction score S = [S 1 ,S 2 ,…,S C ](When c=5, the prediction result S C represents the predicted fraction of missing bolts, when S 5 When it is greater than 0.5, it indicates that the image is predicted to contain a bolt with a missing pin, and the identification of the bolt with a missing pin is completed).
[0077] Step 6: Under the Pytorch framework, the proposed model is trained and tested using a GTX1080ti GPU. The experiment shows that the accuracy of each category is 95.80% for bolts with pins, 97.72% for bolts without pins, 97.64% for vertical bolts, 89.94% for horizontal bolts, and 75.56% for bolts without pins, with an average accuracy of 91.33%. After using the proposed model, the accuracy of each category is 97.60% for bolts with pins, 98.80% for bolts without pins, 97.58% for vertical bolts, 92.96% for horizontal bolts, and 87.35% for bolts without pins, with an average accuracy of 94.86%. The recognition accuracy of bolts with missing pins is improved by 11.79%, and the average accuracy is improved by 3.53%, which proves the effectiveness of the proposed model.
[0078] After this application was run confidentially for a period of time, the on-site technicians reported that the benefits were:
[0079] The model proposed in the present invention transforms the task of identifying bolts with missing pins into a reasoning task for bolts without pins using the hardware to which the bolt belongs and the bolt orientation. When it is impossible to determine whether a bolt without a pin is a bolt with missing pins by visual features, the graph knowledge reasoning method is used to improve the recognition accuracy of bolts with missing pins.
[0080] At present, the technical solution of the present invention has been put into pilot production, that is, a small-scale test of the product before large-scale mass production; after the pilot production was completed, a user usage survey was carried out in a small range, and the survey results showed that user satisfaction was high; now preparations have begun for the formal production of the product for industrialization (including intellectual property risk warning survey).
Claims
1. A method for identifying missing bolts in power transmission lines based on graph knowledge reasoning, characterized in that: The method comprises the following steps: obtaining a bolt pin-missing dataset, wherein the bolt pin-missing dataset comprises images and bolt labels, and the categories of the bolt labels include bolt pin-missing bolts; obtaining a correlation probability matrix A s , obtain a hardware picture, obtain a class map and feature tensor of the bolt label category in the hardware picture, obtain a category representation vector V of the bolt label category in the hardware picture, update the category representation vector V of the bolt label category in the hardware picture to obtain a category representation vector Y, perform global average pooling on the class map of the hardware picture to obtain a first confidence score vector S1, perform a binary classifier on the category representation vector Y to obtain a category score, connect the category scores to obtain a second confidence score vector S2, perform weighted summation on the two confidence score vectors to obtain a label prediction score S, the label prediction score S includes a prediction score for a missing pin bolt, and when the prediction score for a missing pin bolt is greater than 0.5, it is known that the hardware picture contains a missing pin bolt; Obtain bolt missing pin data set and obtain related probability matrix A s The step is a preliminary preparation step, which is specifically divided into the following steps: S1 obtains a bolt pin-missing dataset, obtains a bolt pin-missing dataset, and the bolt pin-missing dataset includes pictures and bolt labels. The categories of bolt labels include bolts with pins, bolts without pins, vertical bolts, horizontal bolts and bolts with missing pins, a total of five types of bolt labels; S2 obtains a related probability matrix, obtains a training set from the bolt pin-missing dataset, counts the number of co-occurrences of bolt labels in all pictures in the training set, obtains a symmetric co-occurrence matrix N, normalizes the co-occurrence matrix N and obtains a related probability matrix A s ; The steps of obtaining a hardware picture and a class map, feature tensor, category representation vector V, category characterization vector Y and label prediction score S of the hardware picture are specific identification steps, which are specifically divided into the following steps: S3 obtains the category representation vector of the bolts in the hardware picture, passes the hardware picture through a feature extraction network and obtains a feature map, passes the feature map through a classifier and a sigmoid function in sequence to obtain a class map, performs channel downsampling on the feature map to obtain a feature tensor, converts the feature tensor through the class map and performs weighted summation to obtain a category representation vector V of the bolt label category in the hardware picture; S4 updates the category representation vector to obtain a category characterization vector, takes each category representation vector in the category representation vector V obtained in step S3 as a node, and takes the correlation probability matrix A obtained in step S2 as a node. s As an edge, the graph convolutional network is input to realize the propagation of co-occurrence information, and each category representation vector is used as a node to realize information update to obtain the category representation vector Y; S5 obtains the label prediction score, and the class mapping of the hardware picture obtained in step S3 is subjected to global average pooling to obtain the first confidence score vector S1, and the category representation vector Y is subjected to a binary classifier to obtain the category score, and the category scores are connected to obtain the second confidence score vector S2, and the two confidence score vectors are weighted and summed to obtain the label prediction score S, and the label prediction score S includes the prediction score of the pin-containing bolt, the prediction score of the pin bolt, the prediction score of the vertical bolt, the prediction score of the horizontal bolt and the prediction score of the pin-missing bolt.
2. The method for identifying missing bolts in power transmission lines based on graph knowledge reasoning according to claim 1 is characterized in that: In step S3, the feature extraction network is a residual network.
3. A device for identifying missing bolts in a power transmission line based on graph knowledge reasoning, used in the method for identifying missing bolts in a power transmission line based on graph knowledge reasoning as claimed in claim 1, characterized in that: The invention comprises the following program modules: a module for obtaining a bolt pin-missing dataset, which is used to obtain a bolt pin-missing dataset, wherein the bolt pin-missing dataset comprises pictures and bolt labels, and the categories of bolt labels include bolts with pins, bolts without pins, vertical bolts, horizontal bolts and bolts with pins missing, with a total of five categories of bolt labels; a module for obtaining a correlation probability matrix, which is used to obtain a training set from the bolt pin-missing dataset, count the number of co-occurrences of bolt labels in all pictures in the training set, obtain a symmetrical co-occurrence matrix N, normalize the co-occurrence matrix N and obtain a correlation probability matrix A s ; A module for obtaining a category representation vector of bolts in a hardware picture is used to pass the hardware picture through a feature extraction network and obtain a feature map, pass the feature map through a classifier and a sigmoid function in sequence to obtain a class map, perform channel downsampling on the feature map to obtain a feature tensor, convert the feature tensor through a class map and perform weighted summation to obtain a category representation vector V of the bolt label category in the hardware picture; Update the category representation vector to obtain the category representation vector module, which is used to obtain the category representation vector V obtained in the category representation vector module of the bolt in the hardware picture. Each category representation vector is a node to obtain the relevant probability matrix A obtained in the relevant probability matrix module s As an edge, a graph convolutional network is input to realize the propagation of co-occurrence information, and each category representation vector is used as a node to realize information update to obtain a category representation vector Y; a label prediction score module is obtained, which is used to obtain the class mapping of the hardware picture obtained in the module of obtaining the category representation vector of the bolts in the hardware picture by global average pooling to obtain a first confidence score vector S1, and the category representation vector Y is passed through a binary classifier to obtain a category score, and the category scores are connected to obtain a second confidence score vector S2, and the two confidence score vectors are weightedly summed to obtain a label prediction score S, and the label prediction score S includes the prediction score of the bolt containing the pin, the prediction score of the pin bolt, the prediction score of the vertical bolt, the prediction score of the horizontal bolt and the prediction score of the bolt without the pin.
4. A device for identifying missing bolts in a power transmission line based on graph knowledge reasoning, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: The computer program includes the program module in claim 3, and the processor implements the corresponding steps in the method of claim 1 or 2 when executing the computer program.
5. A device for identifying missing bolts in a power transmission line based on graph knowledge reasoning, comprising a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, characterized in that: The computer program includes the program module in claim 3, and implements the corresponding steps in the method of claim 1 or 2 when the computer program is executed by a processor.
Citation Information
Patent Citations
Super-resolution bolt image generation method based on improved SRGAN model
CN111210007A
Semantic dictionary learning method and device for bolt defect classification
CN114037689A
Vehicle pattern recognition method and device
CN112990048A
Bolt inspection device, bolt inspection system, bolt inspection method, and program
JP2021113755A
Real time, automatic diagnostic system and method for electric networks
WO2014165957A1
