Transmission tower line patrol path defect detection method and system

Abstract

The invention relates to a method and a system for detecting defects of a line patrol channel of a power transmission tower. The method comprises the following steps of: acquiring and preprocessing an image and text data of the line patrol channel of the tower; the defect target detection model comprises an image feature extraction module, a text information extraction module, a collaborative attention module and a target detection module; performing feature extraction on the tower line patrol path image through an image feature extraction module to obtain an image feature representation matrix of the tower line patrol path image; performing feature extraction on the tower line patrol track text data through a text information extraction module to obtain a text feature representation matrix of the tower line patrol track text data; enhancing the image feature representation matrix based on the text feature representation matrix through a collaborative attention module to obtain an enhanced image feature representation matrix; and inputting the enhanced image feature representation matrix into a target detection module for defect detection to obtain a defect detection result of the transmission tower line patrol channel.

Description

Technical Field

[0001] The present invention relates to a method and system for detecting defects in the inspection path of transmission towers, belonging to the technical field of computer vision. Background Art

[0002] Most of the inspection paths of transmission towers are located in remote mountainous areas. Due to environmental changes and the influence of bad weather, defects such as weed growth, potholes, and collapses on the inspection paths often occur. In severe cases, the inspection paths are even damaged, which often affects the normal line inspection of operation and maintenance personnel and makes it impossible to carry out inspection operations. The defect detection of the inspection path becomes particularly important.

[0003] Currently, the method for detecting defects in the inspection path of transmission towers mainly relies on manual inspection and judgment. The road conditions of the inspection path are checked in combination with manual inspection. After discovering defects, the defect categories, coordinate positions, and photographic records are manually registered. Manual detection not only has low efficiency and high cost, but more importantly, it cannot meet the growing inspection and detection requirements of transmission lines and the need for digital intelligent inspection.

[0004] The manual detection method not only requires a large amount of labor and time costs, but also due to the differences in the skills and experience of detection personnel, the detection results are affected by subjective factors. Manual detection not only has low efficiency and high cost, but more importantly, it cannot meet the growing detection and investigation requirements. Summary of the Invention

[0005] In order to solve the problems existing in the above-mentioned prior art, the present invention proposes a method and system for detecting defects in the inspection path of transmission towers.

[0006] The technical solution of the present invention is as follows:

[0007] On the one hand, the present invention provides a method for detecting defects in the inspection path of transmission towers, including the following steps:

[0008] Collect images of the inspection path of the transmission tower and preprocess the images of the inspection path of the transmission tower, collect text data of the inspection path of the transmission tower and preprocess the text data of the inspection path of the transmission tower;

[0009] Construct a defect target detection model, where the defect target detection model includes an image feature extraction module, a text information extraction module, a collaborative attention module, and a target detection module;

[0010] Extract features from the preprocessed images of the inspection path of the transmission tower through the image feature extraction module to obtain an image feature representation matrix of the images of the inspection path of the transmission tower;

[0011] Extract features from the preprocessed text data of the inspection path of the transmission tower through the text information extraction module to obtain a text feature representation matrix of the text data of the inspection path of the transmission tower;

[0012] Enhance the image feature representation matrix based on the text feature representation matrix through the collaborative attention module to obtain the enhanced image feature representation matrix;

[0013] Input the enhanced image feature representation matrix into the target detection module for defect detection to obtain the defect detection result of the transmission tower patrol path.

[0014] As a preferred embodiment of the present invention, the tower patrol path text data includes the remarks information of the tower patrol path and the historical inspection date of the tower patrol path.

[0015] As a preferred embodiment of the present invention, the defect target detection model is constructed based on an improved Faster R-CNN model.

[0016] As a preferred embodiment of the present invention, the image feature extraction module is constructed based on the ResNet101 network and the RPN network;

[0017] The preprocessed tower patrol path image extracts the overall features through the ResNet101 network, and then divides the overall features of the image into several regions, where the regions include the foreground region and the background region. Input the foreground region and the background region into the RPN network to extract the foreground features and the background features, map the coordinates of the corresponding regions of the foreground features and the background features to the overall features of the image to obtain the foreground feature vector and the background feature vector, and construct the foreground feature vector and the background feature vector into an image feature representation matrix.

[0018] As a preferred embodiment of the present invention, the text information extraction module is constructed based on the Bi-GRU algorithm, specifically as shown in the following formula:

[0019]

[0020] Where: x i represents the i-th word in the tower patrol path text data; n represents the total number of words in the tower patrol path text data; represents the corresponding feature vector obtained by inputting each word in the tower patrol path text data into the Bi-GRU algorithm in the order from front to back; represents the corresponding feature vector obtained by inputting each word in the tower patrol path text data into the Bi-GRU algorithm in the order from back to front; e i represents the final feature vector of the i-th word in the tower patrol path text data;

[0021] Construct the final feature vectors of all words in the tower patrol path text data into a text feature representation matrix.

[0022] As a preferred embodiment of the present invention, the step of constructing the enhanced image feature representation matrix is as follows:

[0023] Calculate the attention weights of each feature vector in the image feature representation matrix to all feature vectors in the text feature representation matrix through the collaborative attention module, and construct an attention weight matrix, as shown in the following formula:

[0024] S = H v W(H x ) T

[0025] A x = softmax(S) ∈ R m×n

[0026] Where: S represents the similarity matrix between the image feature representation matrix H v and the text feature representation matrix H x ; W represents the weight matrix; T represents the transpose operation; A x represents the attention weight matrix; R represents a real matrix of m×n; m represents the total number of regions divided by the overall image features;

[0027] Multiply the attention weight matrix by the text feature representation matrix to obtain a new text feature representation matrix, as shown in the following formula:

[0028]

[0029] Where: represents the new text feature representation matrix;

[0030] Concatenate the new text feature representation matrix with the image feature representation matrix to obtain the enhanced image feature representation matrix, as shown in the following formula:

[0031]

[0032] Where: Concat represents the concatenation operation; represents the enhanced image feature representation matrix.

[0033] On the other hand, the present invention also provides a transmission tower patrol path defect detection system, including a data acquisition module, an image feature extraction module, a text information extraction module, a collaborative attention module, and a target detection module;

[0034] The data acquisition module is used to collect tower patrol path images and preprocess the tower patrol path images, collect tower patrol path text data and preprocess the tower patrol path text data;

[0035] The image feature extraction module is used to extract features from the preprocessed tower line inspection path image to obtain an image feature representation matrix of the tower line inspection path image;

[0036] The text information extraction module is used to extract features from the preprocessed tower line inspection path text data to obtain a text feature representation matrix of the tower line inspection path text data;

[0037] The collaborative attention module is used to enhance the image feature representation matrix based on the text feature representation matrix to obtain an enhanced image feature representation matrix;

[0038] The target detection module is used to perform defect detection through the enhanced image feature representation matrix to obtain a defect detection result of the transmission tower line inspection path.

[0039] On the other hand, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the method described in any embodiment of the present invention is implemented.

[0040] On the other hand, the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, the method described in any embodiment of the present invention is implemented.

[0041] The present invention has the following beneficial effects:

[0042] 1. The present invention uses multi-modal data to enhance the feature quantity, fuses the note text features left by past line inspection personnel into the image features through a collaborative attention mechanism, and uses the text features to enhance the image features, thereby improving the detection accuracy.

[0043] 2. The present invention can be combined with the mobile management platform for the transmission line inspection path, change the traditional manual defect recording method, and realize the intelligent detection of inspection path defects. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 is a flowchart of the method of the present invention;

[0045] Figure 2 is a structural diagram of the defect target detection model of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0046] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0047] It should be understood that the step numbers used in the text are only for convenient description and do not limit the order of execution of the steps.

[0048] It should be understood that the terms used in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include the plural forms.

[0049] The terms "comprising" and "including" indicate the presence of the described features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or their combinations.

[0050] The term "and / or" refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0051] Embodiment 1:

[0052] See Figure 1 , a method for detecting defects in the inspection path of a transmission tower, comprising the following steps:

[0053] Collect images of the inspection path of the transmission tower and preprocess the images of the inspection path of the transmission tower, collect text data of the inspection path of the transmission tower and preprocess the text data of the inspection path of the transmission tower;

[0054] Construct a defect target detection model, which includes an image feature extraction module, a text information extraction module, a collaborative attention module and a target detection module;

[0055] Extract features from the preprocessed images of the inspection path of the transmission tower through the image feature extraction module to obtain an image feature representation matrix of the images of the inspection path of the transmission tower;

[0056] Extract features from the preprocessed text data of the inspection path of the transmission tower through the text information extraction module to obtain a text feature representation matrix of the text data of the inspection path of the transmission tower;

[0057] Enhance the image feature representation matrix based on the text feature representation matrix through the collaborative attention module to obtain an enhanced image feature representation matrix;

[0058] Input the enhanced image feature representation matrix into the target detection module for defect detection to obtain the defect detection result of the inspection path of the transmission tower.

[0059] As a preferred implementation manner of this embodiment, the text data of the pole tower patrol path includes the remarks information of the pole tower patrol path and the historical patrol date of the pole tower patrol path. Since the lengths of the remarks information of the patrol path are different, considering the actual situation of the transmission line patrol, the remarks of the patrol path edited by the patrol personnel during the operation are usually within 13 characters. Therefore, in this embodiment, the length of the text is set to 13. If the length of the text is greater than 13, it will be truncated. If the length is less than 13, the vacant part will be filled with empty characters, and the month information of the system date will be appended after this text. After such processing, the lengths of the texts at the final input end are kept consistent, all being 15 characters, which is convenient for calculation;

[0060] As a preferred implementation manner of this embodiment, refer to Figure 2 , the defect target detection model is constructed based on an improved Faster R-CNN model;

[0061] As a preferred implementation manner of this embodiment, the image feature extraction module is constructed based on the ResNet101 network and the RPN network;

[0062] The preprocessed pole tower patrol path image extracts the overall features through the ResNet101 network, and then divides the overall features of the image into several regions. The regions include the foreground region and the background region (there are more foreground regions and fewer background regions). The foreground region and the background region are input into the RPN network to extract the foreground features and the background features, and the coordinates of the corresponding regions of the foreground features and the background features are mapped to the overall features of the image to obtain the foreground feature vector and the background feature vector. The foreground feature vector and the background feature vector are constructed into an image feature representation matrix m represents the number of regions. In this embodiment, m = 128, d 1 refers to the dimension of the image feature vector corresponding to each region. In this embodiment, d 1 = 512.

[0063] As a preferred implementation manner of this embodiment, the text information extraction module is constructed based on the Bi-GRU algorithm, as shown in the following formula:

[0064]

[0065] Where: x i represents the i-th word in the text data of the pole tower patrol path; n represents the total number of words in the text data of the pole tower patrol path. In this embodiment, n = 15; represents the corresponding feature vector obtained by inputting each word in the text data of the pole tower patrol path into the Bi-GRU algorithm in the order from front to back; represents the corresponding feature vector obtained by inputting each word in the text data of the pole tower patrol path into the Bi-GRU algorithm in the order from back to front; en Represents the final feature vector of the i-th word in the text data of the pole tower patrol path;

[0066] Construct the final feature vectors of all words in the text data of the pole tower patrol path into a text feature representation matrix d 2 Represents the dimension of the text feature vector corresponding to each word. In this embodiment, d 2 = 512.

[0067] As a preferred implementation manner of this embodiment, the step of constructing the enhanced image feature representation matrix is:

[0068] Calculate the attention weights of each feature vector in the image feature representation matrix to all feature vectors in the text feature representation matrix through the collaborative attention module, and construct them into an attention weight matrix, as shown in the following formula:

[0069] S = H v W(H x ) T

[0070] A x = softmax(S) ∈ R m×n

[0071] Where: S represents the similarity matrix between the image feature representation matrix H v and the text feature representation matrix H x The greater the similarity, the closer the semantic information between the text feature and the image feature; W represents the weight matrix, T represents the transpose operation; A x represents the attention weight matrix; R represents an m×n real number matrix; m represents the total number of regions divided by the overall image feature;

[0072] Multiply the attention weight matrix by the text feature representation matrix to obtain a new text feature representation matrix, as shown in the following formula:

[0073]

[0074] Where: represents the new text feature representation matrix, H x ∈ R m×d2 ;

[0075] Concatenate the new text feature representation matrix with the image feature representation matrix to obtain the enhanced image feature representation matrix, as shown in the following formula:

[0076]

[0077] Where: Concat represents the concatenation operation; represents the enhanced image feature representation matrix, There are m feature vectors in total, each of which contains information of both image and text modalities.

[0078] In this embodiment, the transmission tower inspection line data set includes 6040 inspection line images, which are divided into three parts: training, verification, and testing in a ratio of 8:1.5:0.5. The defect types are divided into five categories, namely: inspection line weeds, road collapse, road cracks, tower base weeds, and fallen trees.

[0079] In this embodiment, the defect target detection model is built using the Pytorch framework, using the Stochastic Gradient Descent (SGD) optimizer, setting the learning rate to 0.001, inputting 15 text-image pairs in each iteration, and setting the number of iterations to 300k.

[0080] The trained defect target detection model is deployed to mobile devices for real-time field detection of line patrol defects. When a defect is detected, the real-time positioning position of the GPS module will be recorded to facilitate future line patrol maintenance.

[0081] Embodiment 2:

[0082] A transmission tower line inspection defect detection system, comprising a data acquisition module, an image feature extraction module, a text information extraction module, a collaborative attention module and a target detection module;

[0083] The data acquisition module is used to collect the pole tower patrol line image and pre-process the pole tower patrol line image, collect the pole tower patrol line text data and pre-process the pole tower patrol line text data;

[0084] The image feature extraction module is used to extract features from the preprocessed pole tower patrol line image to obtain an image feature representation matrix of the pole tower patrol line image;

[0085] The text information extraction module is used to extract features from the pre-processed pole tower patrol text data to obtain a text feature representation matrix of the pole tower patrol text data;

[0086] The collaborative attention module is used to enhance the image feature representation matrix based on the text feature representation matrix to obtain an enhanced image feature representation matrix;

[0087] The target detection module is used to perform defect detection through the enhanced image feature representation matrix to obtain defect detection results of the transmission tower patrol line.

[0088] The system is used to implement the method in Example 1, which will not be described in detail here.

[0089] Embodiment 3:

[0090] This embodiment provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the method described in any embodiment of the present invention is implemented.

[0091] Embodiment 4:

[0092] This embodiment provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, the method described in any embodiment of the present invention is implemented.

[0093] In the embodiments of the present application, "at least one" means one or more, and "a plurality" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent the cases of A existing alone, A and B existing simultaneously, and B existing alone. Where A and B may be singular or plural. The character " / " generally represents an "or" relationship between the associated objects before and after. "At least one of the following" and its similar expressions refer to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, and c may represent: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, and c may be single or multiple.

[0094] Those of ordinary skill in the art can realize that the units and algorithm steps described in the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0095] Those skilled in the art can clearly understand that for the convenience and conciseness of description, the specific working processes of the systems, devices, and units described above can refer to the corresponding processes in the foregoing method embodiments and will not be described herein again.

[0096] In several embodiments provided by the present application, if any function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present application. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memory (hereinafter referred to as ROM), random access memory (hereinafter referred to as RAM), magnetic disks, or optical discs that can store program codes.

[0097] The above are only the embodiments of the present invention, and do not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made by using the specification and drawings of the present invention, or directly or indirectly applied in other related technical fields, shall be equally included in the patent protection scope of the present invention.

Claims

1. A method for detecting defects in a transmission tower inspection line, characterized in that: The following steps are involved: Collecting pole tower patrol line images and preprocessing the pole tower patrol line images, collecting pole tower patrol line text data and preprocessing the pole tower patrol line text data; Constructing a defect target detection model, wherein the defect target detection model includes an image feature extraction module, a text information extraction module, a collaborative attention module and a target detection module; The image feature extraction module is used to extract features of the preprocessed pole tower patrol line image to obtain an image feature representation matrix of the pole tower patrol line image; The text information extraction module is used to extract features from the pre-processed pole tower patrol text data to obtain a text feature representation matrix of the pole tower patrol text data; The image feature representation matrix is ​​enhanced based on the text feature representation matrix through the collaborative attention module to obtain an enhanced image feature representation matrix; The enhanced image feature representation matrix is ​​input into the target detection module for defect detection, and the defect detection results of the transmission tower patrol line are obtained.

2. A method for detecting defects in a transmission tower inspection line according to claim 1, characterized in that: The pole tower patrol line text data includes the remark information of the pole tower patrol line and the historical patrol date of the pole tower patrol line.

3. A method for detecting defects in a transmission tower inspection line according to claim 1, characterized in that: The defect target detection model is built based on the improved Faster R-CNN model.

4. A method for detecting defects in a transmission tower inspection line according to claim 3, characterized in that: The image feature extraction module is constructed based on the ResNet101 network and the RPN network; The preprocessed tower inspection line image is passed through the ResNet101 network to extract the overall features, and then the overall features of the image are divided into several regions, including the foreground region and the background region. The foreground region and the background region are input into the RPN network to extract the foreground features and the background features. The coordinates of the corresponding areas of the foreground features and the background features are mapped to the overall features of the image to obtain the foreground feature vector and the background feature vector, and the foreground feature vector and the background feature vector are constructed as an image feature representation matrix.

5. A method for detecting defects in a transmission tower inspection line according to claim 3, characterized in that: The text information extraction module is constructed based on the Bi-GRU algorithm, as shown in the following formula: Where: x i represents the i-th word in the tower patrol line text data; n represents the total number of words in the tower patrol line text data; Indicates the corresponding feature vector obtained by inputting each word in the tower inspection line text data into the Bi-GRU algorithm in order from front to back; It represents the corresponding feature vector obtained by inputting each word in the tower inspection line text data into the Bi-GRU algorithm from back to front; e i Represents the final feature vector of the i-th word in the tower inspection line text data; The final feature vectors of all words in the tower inspection text data are constructed as a text feature representation matrix.

6. A method for detecting defects in a transmission tower inspection line according to claim 3, characterized in that: The enhanced image feature representation matrix construction steps are: The collaborative attention module calculates the attention weight of each feature vector in the image feature representation matrix to all feature vectors in the text feature representation matrix and constructs an attention weight matrix, as shown in the following formula: S=H v W(H x ) T A x =softmax(S)∈R m×n Where: S represents the image feature representation matrix H v And the text feature representation matrix H x The similarity matrix between them; W represents the weight matrix; T represents the transposition operation; A x represents the attention weight matrix; R represents an m×n real number matrix; m represents the total number of regions divided by the overall features of the image; Multiply the attention weight matrix by the text feature representation matrix to obtain a new text feature representation matrix, as shown in the following formula: in: Represents a new text feature representation matrix; The new text feature representation matrix is ​​concatenated with the image feature representation matrix to obtain the enhanced image feature representation matrix, as shown in the following formula: Among them: Concat represents the concatenation operation; Represents the enhanced image feature representation matrix.

7. A transmission tower line inspection defect detection system, characterized in that: It includes data acquisition module, image feature extraction module, text information extraction module, collaborative attention module and target detection module; The data acquisition module is used to collect the pole tower patrol line image and pre-process the pole tower patrol line image, collect the pole tower patrol line text data and pre-process the pole tower patrol line text data; The image feature extraction module is used to extract features from the preprocessed pole tower patrol line image to obtain an image feature representation matrix of the pole tower patrol line image; The text information extraction module is used to extract features from the pre-processed pole tower patrol text data to obtain a text feature representation matrix of the pole tower patrol text data; The collaborative attention module is used to enhance the image feature representation matrix based on the text feature representation matrix to obtain an enhanced image feature representation matrix; The target detection module is used to perform defect detection through the enhanced image feature representation matrix to obtain defect detection results of the transmission tower patrol line.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images