Multi-target detection method for power transmission line based on scene knowledge integrated network

Abstract

This invention relates to the field of power technology and provides a multi-target detection method for transmission lines based on a scene knowledge integration network. The method includes: acquiring an image of the hardware to be detected; extracting original scene features and region proposal features from the image using a Faster-R-CNN network; inputting the region proposal features into a pre-trained multi-layer convolutional network to obtain hardware structure information; processing the hardware structure information through a scene semantic feature pool to obtain enhanced features; fusing the enhanced features with the original scene features to obtain fused features; and detecting the hardware in the image based on the fused features. This invention can improve the detection accuracy of occluded hardware and small target hardware.

Description

Technical Field

[0001] This invention belongs to the field of power technology, and in particular relates to a multi-target detection method for transmission lines based on a scene knowledge integration network. Background Technology

[0002] Transmission lines, as one of the most important infrastructures of my country's energy network, have a crucial impact on the power grid's stable operation. Hardware, as a key component of transmission lines, can be categorized into protective hardware, splicing hardware, connecting hardware, tension clamps, and suspension clamps, among others. All of these hardware play a vital role in maintaining the stable operation of transmission lines. However, because transmission lines are often located in harsh environments such as snow-capped mountains, plains, or deserts, the hardware is constantly exposed to the elements, suffering from rain, snow erosion, wind, and sun exposure. Therefore, it is highly susceptible to corrosion, misalignment, and damage. Regular inspections of transmission line hardware to reduce faults are essential for the normal operation of transmission lines.

[0003] With the continuous maturation of drone inspection technology and its widespread application on power transmission lines, the offline burden on maintenance personnel has been effectively reduced. However, a large number of aerial images obtained by drones still require online manual visual inspection by maintenance personnel, affecting inspection efficiency. Therefore, it is necessary to introduce intelligent inspection technology for power transmission line components. As one of the important inspection targets of power transmission line components, the location and identification of power transmission line hardware can greatly improve the efficiency of power maintenance and repair, reduce the burden on maintenance personnel, and is also a prerequisite for hardware fault diagnosis. However, in actual multi-hardware inspection tasks, due to the different connection methods and shooting angles between hardware, there are often varying degrees of obstruction between hardware, resulting in overlap between hardware marking frames. For example, hardware such as connecting plates, hanging plates, or U-shaped rings within the structure may be obstructed by shielding rings, resulting in missing features in the obstructed hardware area. Meanwhile, the shielding ring marking frame may include other types of hardware, and the shielding ring area may contain noise. Furthermore, due to the limited line-of-sight during drone photography and the small size of the fittings themselves, the bounding box for the fittings is limited, resulting in less information contained in the regional features. Small target fittings such as mounting plates and U-shaped rings occupy a very small proportion of the overall image. Both of these factors inevitably lead to a decrease in the quality of the generated region of interest features, affecting the detection performance. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a multi-target detection method for transmission lines based on a scene knowledge integration network, so as to improve the detection accuracy of obstructed fittings and small target fittings.

[0005] A first aspect of this invention provides a method for multi-target detection of transmission lines based on a scene knowledge integration network, comprising:

[0006] The system acquires images of the hardware to be detected and extracts the original scene features and region proposal features of the images using a Faster-R-CNN network.

[0007] The region proposal features are input into a pre-trained multi-layer convolutional network to obtain the hardware structure information;

[0008] Enhanced features are obtained by processing the structural information of hardware through a scene semantic feature pool;

[0009] The enhanced features are fused with the original scene features to obtain the fused features;

[0010] Based on fusion features, the hardware in the image of the hardware to be detected is detected.

[0011] In conjunction with the first aspect, in one possible implementation of the first aspect, the original scene features and region proposal features of the hardware image are extracted through a Faster-RCNN network, including:

[0012] Extracting basic features from hardware images using a convolutional neural network;

[0013] The basic features are input into the RPN network to obtain the region of interest;

[0014] After pooling the basic features by incorporating the regions of interest into the global region, the results are fed into a fully connected layer to obtain region proposal features and original scene features.

[0015] In conjunction with the first aspect, in one possible implementation of the first aspect, the multi-layer convolutional network is trained in the following way:

[0016] Obtain the training set, which contains multiple hardware images;

[0017] The original scene features and region proposal features of each hardware image in the training set are extracted using the Faster-R-CNN network;

[0018] The region proposal features of each hardware image are used as input to a multi-layer convolutional network, and a scene-hardware co-occurrence matrix is ​​constructed to supervise the output of the multi-layer convolutional network and train the connection weights of the multi-layer convolutional network.

[0019] In conjunction with the first aspect, in one possible implementation of the first aspect, the scene-fitting co-occurrence matrix is ​​pre-constructed in the following way:

[0020] Based on the combined use of various hardware fittings, multiple hardware fitting scenarios are defined;

[0021] Based on historical data of hardware images, the probability of each type of hardware appearing in each hardware scene is statistically analyzed to obtain the scene-hardware co-occurrence matrix.

[0022] In conjunction with the first aspect, in one possible implementation of the first aspect, the scene semantic feature pool is predetermined in the following way:

[0023] Filter the suggested regional features for each hardware image;

[0024] The original scene features and filtered region suggestion features of each hardware image are input into the gated loop unit to filter the original scene features of each hardware image.

[0025] Based on the original scene features of each filtered hardware image, a scene classifier is trained in a supervised learning manner, and the weights of the scene classifier are extracted as a scene semantic feature pool.

[0026] In conjunction with the first aspect, in one possible implementation of the first aspect, a scene classifier is trained in a supervised learning manner based on the original scene features of each filtered hardware image, including:

[0027] Add scene annotations to each hardware image;

[0028] The original scene features of each filtered hardware image are used as input to the scene classifier. The output of the scene classifier is supervised by scene annotations, and the weights of the scene classifier are trained.

[0029] In conjunction with the first aspect, in one possible implementation of the first aspect, the detection of hardware in the hardware image to be detected is based on the fusion features, including: inputting the fusion features into a preset classification regression model to obtain the category of each hardware in the hardware image to be detected.

[0030] A second aspect of this invention provides a multi-target detection device for power transmission lines based on a scene knowledge integration network, comprising:

[0031] The acquisition module is used to acquire the captured image of the hardware to be detected, and extract the original scene features and region proposal features of the image of the hardware to be detected through the Faster-R-CNN network;

[0032] The first processing module is used to input the region proposal features into a pre-trained multi-layer convolutional network to obtain the hardware structure information;

[0033] The second processing module is used to process the hardware structure information through the scene semantic feature pool to obtain enhanced features;

[0034] The fusion module is used to fuse the enhanced features with the original scene features to obtain fused features;

[0035] The detection module is used to detect hardware in the image of the hardware to be detected based on fused features.

[0036] A third aspect of the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method as described in the first aspect or any implementation thereof.

[0037] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method as described in the first aspect or any implementation thereof.

[0038] The beneficial effects of the embodiments of the present invention compared with the prior art are as follows:

[0039] This invention defines multiple hardware scenes based on the aggregation of hardware. The original scene features and region proposal features of the hardware images to be detected are extracted using a Faster-R-CNN network. The region proposal features are then input into a pre-trained multi-layer convolutional network to obtain hardware structure information and the relationship between the structure and the scene. The hardware structure information is processed through a scene semantic feature pool to obtain enhanced features, thereby emphasizing global contextual information and fine-grained visual information of the hardware. Finally, the enhanced features are fused with the original scene features to assist the classifier and regressor of multi-hardware target detection in detecting hardware in the hardware images to be detected, effectively improving the detection performance of occluded hardware and small target hardware. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 This is a schematic diagram of hardware scene classification provided in an embodiment of the present invention;

[0042] Figure 2 This is a schematic diagram illustrating the implementation process of the multi-target detection method for power transmission lines based on a scene knowledge integration network provided in this embodiment of the invention.

[0043] Figure 3 This is a schematic diagram of the detection process of the scene knowledge integration network provided in an embodiment of the present invention;

[0044] Figure 4 This is a schematic diagram of the training process of the scene knowledge integration network provided in an embodiment of the present invention;

[0045] Figure 5 This is a schematic diagram of the processing procedure of the gated loop unit provided in an embodiment of the present invention;

[0046] Figure 6 This is a schematic diagram of the scene-fitting co-occurrence matrix provided in an embodiment of the present invention;

[0047] Figure 7 This is a schematic diagram of the processing procedure of the scene knowledge module provided in an embodiment of the present invention;

[0048] Figure 8 This is a schematic diagram of the structure of the multi-target detection device for power transmission lines based on a scene knowledge integration network provided in an embodiment of the present invention;

[0049] Figure 9 This is a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0050] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of the invention. However, those skilled in the art will understand that the invention can be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of the invention with unnecessary detail.

[0051] To illustrate the technical solution described in this invention, specific embodiments are described below.

[0052] This invention combines scene knowledge with multi-fitting target recognition of power transmission lines, proposing a multi-target detection method for power transmission lines based on a scene knowledge integration network.

[0053] First, let's introduce the knowledge of the scenarios:

[0054] Visual concepts in most natural images are composed of elements such as scene, object, and relationship. Psychological studies show that scene information and object relationships significantly influence the human process of object recognition. By mimicking the human object recognition process, general scene information has been shown in studies to effectively improve target detection. When a person cannot directly identify a target in an image (e.g., a boat), they tend to use scene information (e.g., the sea) to infer the target. From a common-sense perspective, a target at sea is more likely to be a boat than a car or other means of transportation. However, in the identification of multiple hardware components on power transmission lines, the aforementioned scene information will be of limited help. This is because scenes of forests, deserts, or skies have no necessary connection to the presence of any particular hardware component.

[0055] Therefore, in this embodiment, the scene information in the multi-fitting detection task is defined as an organic whole comprised of fittings and fitting objects. For example, the scene in the image is an anti-galling structure scene. Based on this scene information, experienced maintenance workers can infer the occluded fittings within the green frame based on some visual features and structural characteristics of the fittings. Thus, the scene information in the multi-fitting detection task, defined in this way, should include global context information, fine-grained visual information of the fittings, and structural information between fittings. Unlike previous target detection tasks, this embodiment mainly targets the occlusion and small target problems in multi-fitting detection tasks, focusing on utilizing the domain knowledge contained in the scene information in multi-fitting detection to assist in improving the deep learning model.

[0056] Using this application as an example, 14 types of hardware targets are analyzed: pre-stretched suspension clamp (PT), bag-type suspension clamp (BT), compression tension clamp (CT), wedge-type tension clamp (WT), hanging plate (HB), U-shaped hanging ring (UT), connecting plate (YP), parallel groove clamp (PG), vibration damper (SH), spacer bar (SP), equalizing ring (GR), shielding ring (SR), counterweight (WE), and adjusting plate (AB). By exploring professional knowledge in the power field and combining it with the habits of maintenance workers in identifying hardware, 11 hardware scenarios are defined to assist the network in multi-hardware target detection. See also Figure 1 Scenario 1 shows bag-type suspension clamps clustered around the connecting plate; Scenario 2 shows bag-type suspension clamps clustered around the counterweight; Scenario 3 shows the interconnection of the adjusting plate, hanging plate, and U-shaped hanging ring; Scenario 4 shows an equalizing and shielding structure composed of a shielding ring, equalizing ring, connecting plate, and scratches; Scenario 5 shows a single-unit suspension structure composed of a hanging plate, hanging ring, and connecting plate; Scenario 6 shows a connecting structure composed of parallel groove clamps, tension clamps, and hanging rings; Scenario 7 shows a double-unit suspension structure composed of hanging rings, hanging plates, bag-type suspension clamps, and connecting plates; Scenario 8 shows the connection between parallel groove clamps and the vibration damper; Scenario 9 shows a single-unit suspension structure composed of compression suspension clamps, hanging rings, hanging plates, and connecting plates; Scenario 10 shows a connecting structure of tension suspension clamps, connecting plates, and hanging rings; Scenario 11 shows a connecting structure of connecting plates, equalizing rings, and suspension clamps.

[0057] Please see also Figure 2 and Figure 3 As shown, the multi-target detection method for transmission lines based on a scene knowledge integration network provided in this embodiment specifically includes the following steps:

[0058] Step S101: Acquire the captured image of the hardware to be detected, and extract the original scene features and region proposal features of the image of the hardware to be detected through the Faster-R-CNN network.

[0059] In this embodiment, images of the hardware to be inspected can be taken at the inspection site using a drone.

[0060] In this step, the original scene features and region proposal features of the hardware image are extracted using the Faster-RCNN network, which can be described in detail as follows:

[0061] Step 1: Extract basic features from the hardware image using a convolutional neural network;

[0062] Step 2: Input the basic features into the RPN network to obtain the region of interest;

[0063] Step 3: After pooling the basic features by incorporating the regions of interest into the global region, the results are fed into the fully connected layer to obtain region proposal features and original scene features.

[0064] The explanations for steps one through three above are as follows:

[0065] The image of the hardware to be detected is used as the input to the network, and a convolutional neural network such as ResNet101 is used as the backbone network for feature extraction to obtain the basic features F. CNN =Γ CNN (Z i ||θ CNN Z i Input an image, Γ CNN (·||θ CNN ) represents the feature extraction process, θ CNN These are the learnable weights in the process.

[0066] The basic features are further input into the RPN (Region Proposal Network). RPN (·||θ RPN Obtain the region of interest K in ) rois K rois =Γ RPN (F CNN ||θ RPN Similarly, θ RPN These are learnable weights.

[0067] The generation of original scene features requires the pre-construction of a global region. Therefore, the region of interest, after incorporating the global region, can be represented as: Where, N η This represents the number of regions of interest; the extra one is the global region. This represents the coordinates of the top-left corner of the region. Then it is the coordinate of the lower right corner.

[0068] Then take K r ′ ois For F CNN After performing pooling operations and passing the data through a fully connected layer, we can obtain the region proposal features and the original scene features: Where Γtail (·||θ tail ) represents the pooling layer and the fully connected layer, θ tail This is where the learnable weights are. Then, for F... tail By segmenting the data, we can obtain the region proposal features. and original scene features Where C γ Indicates the feature dimension.

[0069] Step S102: Input the region proposal features into a pre-trained multi-layer convolutional network to obtain the hardware structure information.

[0070] In this embodiment, by convolving the region proposal features using a pre-trained multi-layer convolutional network, the hardware structure information F of the region of interest can be obtained. sk =Convs sk (F prop ).in For hardware structure information, i.e., the predicted scene vector, Convs sk (~) represents a multi-layer convolutional layer.

[0071] Step S103: Process the hardware structure information through the scene semantic feature pool to obtain enhanced features.

[0072] In this embodiment, the hardware structure information obtained through the multi-layer convolutional network relies on statistical results, neglecting the unique appearance and combination characteristics of each image (i.e., fine-grained visual information and global contextual information). Therefore, this embodiment uses a scene semantic feature pool to process the hardware structure information, thereby learning the fine-grained visual information and global contextual information F. rs =ψ(F sk )·(W ca ) T Where ψ(~) represents the softmax activation function, and (~) T Indicates matrix transpose. To enhance features. This is a scene semantic feature pool.

[0073] Step S104: The enhanced features are fused with the original scene features to obtain fused features.

[0074] First, for the enhanced feature F rs Perform feature mining to obtain F en :

[0075] F en =F rs ·W en

[0076] in, To transform the weight matrix, C e It is a dimension of scene information features.

[0077] Then, the original scene features F are utilized in a cascade manner. prop With F en The joint features that incorporate scene information are obtained. The fused features include fine-grained visual information of the fittings, global context information, and structural information between fittings, and the influence of image specificity is taken into account.

[0078] Step S105: Based on the fusion features, detect the hardware in the hardware image to be detected.

[0079] In this embodiment, the fused features include fine-grained visual information of the fittings, global contextual information, and structural information between fittings, and take into account the influence of image specificity. By inputting the fused features into the classification and regression detector, the reasoning of occluded fittings and small target fittings can be assisted.

[0080] As can be seen, this embodiment of the invention defines multiple hardware scenes based on the aggregation of hardware. The original scene features and region proposal features of the hardware images to be detected are extracted using a Faster-R-CNN network. The region proposal features are input into a pre-trained multi-layer convolutional network to obtain hardware structure information and the relationship between the structure and the scene. The hardware structure information is processed through a scene semantic feature pool to obtain enhanced features, thereby emphasizing global contextual information and fine-grained visual information of the hardware. Finally, the enhanced features are fused with the original scene features to assist the classifier and regressor of multi-hardware target detection in detecting hardware in the hardware images to be detected, effectively improving the detection effect of occluded hardware and small target hardware.

[0081] The following combination Figure 4 The training process of the scene knowledge integration network is introduced.

[0082] In this embodiment, images of hardware targets taken by UAVs during power transmission line inspections are used as the dataset. The training set contains 1,330 images with 16,358 hardware targets, while the test set contains 318 images with 2,767 hardware targets.

[0083] Similar to step S101, the processed hardware images from the training set are first used as input to the network. A ResNet101 convolutional neural network is used as the backbone network for feature extraction, and the original scene features are obtained using the RPN network and the ROIAlign network. and regional suggested features

[0084] During the training process, a scene filtering module and a scene knowledge module were defined.

[0085] (1) Scene filtering module.

[0086] It is mainly used to obtain the scene semantic feature pool, that is:

[0087] Filter the suggested regional features for each hardware image;

[0088] The original scene features and filtered region suggestion features of each hardware image are input into the gated loop unit to filter the original scene features of each hardware image.

[0089] Based on the original scene features of each filtered hardware image, a scene classifier is trained in a supervised learning manner, and the weights of the scene classifier are extracted as a scene semantic feature pool.

[0090] Specifically as follows:

[0091] While the original scene features contain global contextual information and fine-grained visual information about the fittings, this information contains a significant amount of noise due to the small proportion of effective targets like fittings and the large proportion of invalid areas such as the background in drone aerial images. This embodiment uses a scene filtering module to filter the scene information, removing redundant noise and emphasizing the fine-grained visual information of the fittings and the global contextual information formed by the combination of fittings.

[0092]

[0093] Where → indicates filtering, that is, in F prop Select N from the features μ Each component feature F fore This includes, as far as possible, suggested features for hardware fittings regions. From an individual perspective, a single valid hardware fitting region suggestion feature represents fine-grained visual information of the scene; from a holistic perspective, the combination of multiple valid hardware fitting region suggestion features constitutes the global contextual information of the scene. Therefore, a storage unit is needed to sequentially input individual hardware fitting region suggestion features to memorize valid fine-grained visual information about hardware fittings. Thus, over the entire time-series period, the storage unit memorizes combinations of multiple valid hardware fitting scene features, which are then endowed with global contextual information. And F fore A small portion of the data may contain background features, most of which are invalid information. Therefore, this storage unit also needs to be able to forget invalid information.

[0094] Therefore, this module uses a GRU (Gated Recurrent Unit), which has a simple structure, is easy to assemble, and achieves excellent long-term memory capacity due to its dual-gating mechanism. Furthermore, its update gate design allows it to forget useless information. Figure 5As shown, the original scene features are filtered using the specific principle of the gated loop unit:

[0095] First, construct the input in sequence form: Where t represents time t, and the original scene features F are used. sc As the initial state, the update gate u at time t... t for:

[0096]

[0097] Where σ(~) represents the Sigmoid activation function. Represents the state at time t-1, [,] denotes matrix concatenation, W u These are learnable parameters. The update gate at time t controls memory retention, thus allowing us to further deduce the scene state at time t.

[0098]

[0099] Where * represents the Hadama volume, and The formula is as follows:

[0100]

[0101] in This represents the tanh activation function, while r t The reset gate determines how the newly input information is combined with previously saved memories, and its formula is as follows:

[0102]

[0103] After the above process, the gate u can be updated. t and reset door r t Coordinated control, after t=N μ After a time series, the filtered scene features are obtained.

[0104] To constrain the memory direction of the GRU (i.e., emphasizing effective global contextual information and fine-grained visual information of the fittings, while ignoring invalid background information), the scene filtering module is designed to be motivated to learn using supervised loss, as follows:

[0105] First, construct the scene label space. The input image Z during training is... i Constituting dataset D train Then in D train Construct an n s The label space Q is of class +1. Specifically, the i-th image Z i Besides its original n fIn addition to the target detection and labeling for metal-like devices, it was also assigned a... The labels are used to represent the scene labels of the graph. When the graph contains a certain scene, then it is labeled in Y. i The value for the corresponding dimension is 1, otherwise it is 0; if the graph does not contain a scene, then The remaining dimensions are 0.

[0106] Next, a scene classifier is constructed to map the filtered scene features to the scene label space:

[0107]

[0108] in These are the weights for the scene classifier. The following loss function is used to supervise... The distribution of Y in the scene label space is close to that of Y. i :

[0109]

[0110] Where N s =n s +1, N b This represents the number of images contained in a batch.

[0111] Therefore, in order to and Y i The distribution of elements is similar, enabling the completion of scene classification tasks. During backpropagation, the GRU is constrained to retain effective information and eliminate invalid noise, i.e., emphasizing effective global contextual information and fine-grained visual information of hardware, while ignoring invalid background information, as mentioned earlier. Furthermore, the scene classifier weights will also learn this information during the mapping process. Therefore, this embodiment extracts the classifier weights to form a scene semantic feature pool, which is similar to the memory in the human brain when recalling a certain category. Its advantage is that information can be propagated in memory on the semantic concepts of all scene categories (not just the scene categories appearing in the image).

[0112] (2) Scene knowledge module.

[0113] The main function of the scene knowledge module is to propagate structural information between hardware components to the scene's semantic features. During training, the scene knowledge module primarily trains a multi-layer convolutional network.

[0114] Multi-layer convolutional networks are trained in the following way:

[0115] Obtain the training set, which contains multiple hardware images;

[0116] The original scene features and region proposal features of each hardware image in the training set are extracted using the Faster-R-CNN network;

[0117] The region proposal features of each hardware image are used as input to a multi-layer convolutional network, and a scene-hardware co-occurrence matrix is ​​constructed to supervise the output of the multi-layer convolutional network and train the connection weights of the multi-layer convolutional network.

[0118] The scene-fitting co-occurrence matrix is ​​pre-constructed in the following way:

[0119] Based on the combined use of various hardware fittings, multiple hardware fitting scenarios are defined;

[0120] Based on historical data of hardware images, the probability of each type of hardware appearing in each hardware scene is statistically analyzed to obtain the scene-hardware co-occurrence matrix.

[0121] First, let's introduce the scenario - the hardware co-occurrence matrix:

[0122] The structural information between hardware fittings can be extracted in different ways: from the perspective of instance level, it can be the co-occurrence matrix or spatial relationship matrix between hardware fittings; from the perspective of scene level, it can be based on what kind of hardware fitting combinations may be included in a scene, which includes the macroscopic structural information between hardware fittings. Furthermore, this scene-level structural information can be quantitatively represented by the scene-hardware co-occurrence matrix.

[0123] Therefore, this embodiment uses conditional probability as a basis and data-driven methods to model the relationship between the scene and the fitting co-occurrence matrix. The specific construction method is as follows:

[0124] The conditional probability formula used to construct the scene-tool co-occurrence matrix is ​​as follows:

[0125]

[0126] This represents the probability of the i-th tool appearing when the j-th scenario occurs. There are a total of n scenarios. s +1 category: Scenes without a defined scenario (consisting of scattered hardware items that cannot be categorized into any of the defined scenarios) are also considered a category, and there are a total of n hardware items. f Class. Among them:

[0127]

[0128]

[0129] Where N ij N represents the number of scenarios in the dataset where the i-th type of hardware exists, where N is the number of such scenarios. j N represents the number of times the j-th type of scene appears in the dataset. all This represents the total number of occurrences of all scenarios across all classes in the dataset.

[0130] Substituting the above formula, we get:

[0131]

[0132] Therefore, it is necessary to calculate N. ij and N j After statistically analyzing both, a scene-fitting co-occurrence matrix will be constructed. like Figure 6 As shown.

[0133] To learn the structural information between hardware components, a scene knowledge module is proposed. This module uses a prior knowledge graph—the scene-hardware co-occurrence matrix—as external supervision, rather than fixed connections, to learn the structural information between regions of interest. The specific process is as follows: Figure 7 As shown:

[0134] During the training process, features of the region of interest There exists a corresponding real category. Through its in M s-f The mapping on can be obtained This display mapping, which utilizes the true category of features in the region of interest, ensures the extraction of information and provides a strong guarantee for subsequent supervised learning of hardware structure information.

[0135] Construct multiple convolutional layers to learn the hardware structure information of the region of interest:

[0136] F sk =Convs sk (F prop )

[0137] in For the predicted scene vector, Convs sk (~) represents a multi-layer convolutional layer.

[0138] By letting the predicted scene vector F sk Real-world scene vectors obtained from proximity scene-knowledge matrix graphs In this task, the scene knowledge module will learn the structural information between hardware contained in the scene-knowledge matrix graph, thereby obtaining the connection weights of the multi-layer convolutional network.

[0139] The loss function used in this supervision process is shown in the following formula:

[0140]

[0141] The learned hardware structure information relies on statistical results and lacks compatibility with each image. Therefore, it is necessary to propagate the specific information contained in the scene semantic feature pool of the scene filtering module to the results of the scene knowledge module, thereby obtaining enhanced features for each region of interest. These enhanced features will contain global contextual information and fine-grained visual information of the hardware from the scene filtering module, as well as the hardware structure information learned from the scene knowledge module, which conforms to the definition of scene information in this paper. Further feature mining is then performed using these enhanced features, which are then cascaded and fused to the original region of interest features. When the visual features of the hardware in the original region of interest features are missing or of poor quality, the scene information in the additional enhanced features can assist the classifier and regressor in inferring the hardware category and its location. This process is described in steps S104-S105 and will not be repeated here.

[0142] Based on the above, this embodiment first uses the processed aerial image of hardware as input to the network, and uses a ResNet101 convolutional neural network as the backbone network for feature extraction. It then utilizes a Region Proposal Network (RPN) and a Region Align Network (ROIAlign) to obtain original scene features and region proposal features. The original scene features are processed by a scene filtering module to learn global contextual information and fine-grained visual information of the hardware in a supervised manner, and the learned information is propagated to the scene semantic features. The region proposal features are processed by a scene knowledge module, using a data-driven scene-hardware co-occurrence matrix to supervise the learning of the structural information between hardware and the scene, and the learned information is propagated to the scene semantic features. Finally, the scene semantic features are used for further feature mining, and then cascaded and fused with the original region proposal features to assist the classifier and regressor in multi-hardware target detection. The scene information in the scene semantic features will assist in reasoning about occluded targets and small hardware targets, thereby improving the multi-hardware target detection effect.

[0143] The main inventive points of this embodiment include:

[0144] (1) Explore professional knowledge in the field of power and combine it with the habits of maintenance workers to identify hardware. Define 11 hardware scenarios to assist the deep learning model in the detection of multiple hardware targets in transmission lines.

[0145] (2) A scene filtering module is proposed. This module uses a supervised approach to encode fine-grained visual information of hardware and some background visual information in the image into scene features using GRU. The filtered scene features are further mapped to the scene label space for scene classification. Therefore, the scene semantic features constructed using the scene classifier weights will also emphasize global context information and fine-grained visual information of hardware.

[0146] (3) A scene knowledge module is proposed, which uses a scene-fitting co-occurrence matrix to quantify the structural information between fittings, and then learns this information in a supervised manner. The information from the scene semantic features is then propagated to the learning results of the scene knowledge module to obtain enhanced features for the region of interest. These enhanced features conform to the definition of scene information in this paper and are compatible with the specificity of objects in the image. Finally, the enhanced features are used for feature mining and cascade fusion to assist in the detection of multiple fitting targets, effectively improving the detection performance of occluded fittings and small-target fittings.

[0147] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0148] See Figure 8 As shown, this embodiment provides a multi-target detection device 80 for power transmission lines based on a scene knowledge integration network, including:

[0149] The acquisition module 81 is used to acquire the captured image of the hardware to be detected, and extract the original scene features and region proposal features of the image of the hardware to be detected through the Faster-R-CNN network;

[0150] The first processing module 82 is used to input the region proposal features into a pre-trained multi-layer convolutional network to obtain the hardware structure information;

[0151] The second processing module 83 is used to process the hardware structure information through the scene semantic feature pool to obtain enhanced features;

[0152] The fusion module 84 is used to fuse the enhanced features with the original scene features to obtain fused features;

[0153] The detection module 85 is used to detect hardware in the image of the hardware to be detected based on fusion features.

[0154] As one possible implementation, module 81 is specifically used for:

[0155] Extracting basic features from hardware images using a convolutional neural network;

[0156] The basic features are input into the RPN network to obtain the region of interest;

[0157] After pooling the basic features by incorporating the regions of interest into the global region, the results are fed into a fully connected layer to obtain region proposal features and original scene features.

[0158] As one possible implementation, multi-layer convolutional networks are trained in the following way:

[0159] Obtain the training set, which contains multiple hardware images;

[0160] The original scene features and region proposal features of each hardware image in the training set are extracted using the Faster-R-CNN network;

[0161] The region proposal features of each hardware image are used as input to a multi-layer convolutional network, and a scene-hardware co-occurrence matrix is ​​constructed to supervise the output of the multi-layer convolutional network and train the connection weights of the multi-layer convolutional network.

[0162] As one possible implementation, the scene-fitting co-occurrence matrix is ​​pre-constructed in the following way:

[0163] Based on the combined use of various hardware fittings, multiple hardware fitting scenarios are defined;

[0164] Based on historical data of hardware images, the probability of each type of hardware appearing in each hardware scene is statistically analyzed to obtain the scene-hardware co-occurrence matrix.

[0165] As one possible implementation, the scene semantic feature pool is pre-determined in the following way:

[0166] Filter the suggested regional features for each hardware image;

[0167] The original scene features and filtered region suggestion features of each hardware image are input into the gated loop unit to filter the original scene features of each hardware image.

[0168] Based on the original scene features of each filtered hardware image, a scene classifier is trained in a supervised learning manner, and the weights of the scene classifier are extracted as a scene semantic feature pool.

[0169] As one possible implementation, a scene classifier is trained in a supervised learning manner based on the original scene features of each filtered hardware image, including:

[0170] Add scene annotations to each hardware image;

[0171] The original scene features of each filtered hardware image are used as input to the scene classifier. The output of the scene classifier is supervised by scene annotations, and the weights of the scene classifier are trained.

[0172] As one possible implementation, the detection module 85 is specifically used to: input the fused features into a preset classification regression model to obtain the category of each piece of hardware in the image of the hardware to be detected.

[0173] Figure 9 This is a schematic diagram of an electronic device 90 provided in an embodiment of the present invention. Figure 9As shown, the electronic device 90 of this embodiment includes: a processor 91, a memory 92, and a computer program 93 stored in the memory 92 and executable on the processor 91, such as a multi-target detection program for transmission lines based on a scene knowledge integration network. When the processor 91 executes the computer program 93, it implements the steps in the various embodiments of the multi-target detection method for transmission lines based on a scene knowledge integration network described above, for example... Figure 2 The steps S101 to S105 are shown. Alternatively, when the processor 91 executes the computer program 93, it implements the functions of each module in the above-described device embodiments, for example... Figure 8 The functions of modules 81 to 85 are shown.

[0174] For example, computer program 93 may be divided into one or more modules / units, one or more of which are stored in memory 92 and executed by processor 91 to complete the present invention. One or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of computer program 93 in electronic device 90.

[0175] Electronic device 90 can be a desktop computer, laptop, handheld computer, cloud server, or other computing device. Electronic device 90 may include, but is not limited to, a processor 91 and a memory 92. Those skilled in the art will understand that... Figure 9 This is merely an example of electronic device 90 and does not constitute a limitation on electronic device 90. It may include more or fewer components than shown, or combine certain components, or different components. For example, electronic device 90 may also include input / output devices, network access devices, buses, etc.

[0176] The processor 91 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0177] The memory 92 can be an internal storage unit of the electronic device 90, such as a hard disk or RAM of the electronic device 90. The memory 92 can also be an external storage device of the electronic device 90, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the electronic device 90. Furthermore, the memory 92 can include both internal and external storage units of the electronic device 90. The memory 92 is used to store computer programs and other programs and data required by the electronic device 90. The memory 92 can also be used to temporarily store data that has been output or will be output.

[0178] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0179] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0180] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0181] In the embodiments provided by this invention, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0182] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0183] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0184] If integrated modules / units are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0185] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A power transmission line multi-target detection method based on a scene knowledge integrated network, characterized in that, include: The system acquires images of the hardware to be detected and extracts the original scene features and region proposal features of the images using a Faster-R-CNN network. The region proposal features are input into a multi-layer convolutional network of a pre-trained scene knowledge module. The scene-fitting co-occurrence matrix generated by a data-driven approach is used to supervise the learning of the relationship between the structural information of fittings and the scene, thereby obtaining the fittings structural information. The hardware structure information is processed through the scene semantic feature pool of the scene filtering module to learn the fine-grained visual information and global context information of the hardware, thereby obtaining enhanced features; The enhanced features are fused with the original scene features to obtain fused features; Based on the fusion features, the hardware in the image of the hardware to be detected is detected; The original scene features and region proposal features of the hardware images were extracted using the Faster-RCNN network, including: Extracting basic features from hardware images using a convolutional neural network; The basic features are input into the RPN network to obtain the region of interest; After pooling the basic features by incorporating the regions of interest into the global region, the results are fed into a fully connected layer to obtain region proposal features and original scene features. The multi-layer convolutional network is trained in the following way: Obtain a training set, which contains multiple metal fitting images; The original scene features and region proposal features of each hardware image in the training set are extracted using the Faster-R-CNN network; The region proposal features of each hardware image are used as input to a multi-layer convolutional network, and a scene-hardware co-occurrence matrix is ​​constructed to supervise the output of the multi-layer convolutional network and train the connection weights of the multi-layer convolutional network. The scenario-fitting co-occurrence matrix is ​​pre-constructed in the following manner: Based on the combined use of various hardware fittings, multiple hardware fitting scenarios are defined; Based on historical data of hardware images, the probability of each type of hardware appearing when each hardware scene occurs is statistically analyzed to obtain the scene-hardware co-occurrence matrix.

2. The method of claim 1, wherein, The scene semantic feature pool is predetermined in the following way: Filter the suggested regional features for each hardware image; The original scene features and filtered region suggestion features of each hardware image are input into the gated loop unit to filter the original scene features of each hardware image. Based on the original scene features of each filtered hardware image, a scene classifier is trained in a supervised learning manner, and the weights of the scene classifier are extracted as a scene semantic feature pool.

3. The method of claim 2, wherein, Based on the original scene features of each filtered hardware image, a scene classifier is trained using supervised learning, including: Add scene annotations to each hardware image; The original scene features of each filtered hardware image are used as input to the scene classifier. The output of the scene classifier is supervised based on the scene annotations, and the weights of the scene classifier are trained.

4. The method according to any one of claims 1 to 3, characterized in that, Based on the fusion features, the detection of hardware in the hardware image to be detected includes: inputting the fusion features into a preset classification regression model to obtain the category of each hardware in the hardware image to be detected.

5. A power transmission line multi-object detection device based on a scenario knowledge integrated network, characterized in that, include: The acquisition module is used to acquire the captured image of the hardware to be detected, and extract the original scene features and region proposal features of the image of the hardware to be detected through the Faster-R-CNN network; The first processing module is used to input the region proposal features into the multi-layer convolutional network of the pre-trained scene knowledge module, and use the scene-fitting co-occurrence matrix generated by the data-driven method to supervise the learning of the relationship between the structural information of fittings and the scene, so as to obtain the fitting structural information. The second processing module is used to process the hardware structure information through the scene semantic feature pool of the scene filtering module, learn the fine-grained visual information and global context information of the hardware, and obtain enhanced features. The fusion module is used to fuse the enhanced features with the original scene features to obtain fused features; The detection module is used to detect hardware in the image of the hardware to be detected based on the fused features; The acquisition module is specifically used for: extracting basic features of the hardware image through a convolutional neural network; inputting the basic features into the RPN network to obtain the region of interest; and performing pooling operation on the basic features through the region of interest incorporated into the global region, and then inputting them into the fully connected layer to obtain region proposal features and original scene features. The multi-layer convolutional network is trained in the following way: Obtain a training set, which contains multiple metal fitting images; The original scene features and region proposal features of each hardware image in the training set are extracted using the Faster-R-CNN network; The region proposal features of each hardware image are used as input to a multi-layer convolutional network, and a scene-hardware co-occurrence matrix is ​​constructed to supervise the output of the multi-layer convolutional network and train the connection weights of the multi-layer convolutional network. The scenario-fitting co-occurrence matrix is ​​pre-constructed in the following manner: Based on the combined use of various hardware fittings, multiple hardware fitting scenarios are defined; Based on historical data of hardware images, the probability of each type of hardware appearing when each hardware scene occurs is statistically analyzed to obtain the scene-hardware co-occurrence matrix.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 4.

7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 4.

Images