A method and system for identifying multiple hardware components of power transmission lines by integrating implicit knowledge reasoning

Abstract

The present invention provides a method and system for identifying multiple hardware fittings for power transmission lines that integrates implicit knowledge reasoning. The method belongs to the field of image recognition technology and includes the following steps: obtaining an image of hardware fittings to be identified; performing feature extraction on the image of hardware fittings to be identified, determining scene features, multiple region suggestion box features, and the categories of each region suggestion box feature; performing scene enhancement on the region suggestion box features based on the scene features to obtain scene enhancement features; performing spatial enhancement on the region suggestion box features based on the categories of the region suggestion box features to obtain spatial enhancement features; and determining the location and type of hardware fittings in the image of hardware fittings to be identified based on the features of each region suggestion box and the scene enhancement features and spatial enhancement features corresponding to each region suggestion box feature. Through scene enhancement and spatial enhancement, the implicit relationships between hardware fittings are deeply explored, the visual features of obscured hardware fittings are enhanced, and the accuracy of hardware fitting identification is improved.

Description

Technical Field

[0001] The present invention relates to the field of image recognition, and in particular to a method and system for identifying multiple hardware fittings of a power transmission line integrating implicit knowledge reasoning. Background Art

[0002] Transmission line inspection is crucial for ensuring the safe, uninterrupted, and reliable operation of power systems. Hardware is a widely used component in transmission lines, supporting, securing, protecting, and connecting various line components, while maintaining line stability. Condition monitoring of transmission line hardware is a crucial and challenging task. Early detection of hardware defects can reduce power outages, prevent significant economic losses, and mitigate negative customer impacts on the power transmission system.

[0003] Currently, drones are commonly used to inspect power transmission lines, achieving more automated and efficient inspections. During drone inspections, a large number of aerial images and videos of hardware components are generated as photographic records. To achieve automated inspection and intelligent diagnosis of hardware components, it is crucial to accurately detect the hardware components from the aerial images, enabling subsequent operations and maintenance tasks such as fault diagnosis, camera tracking, and data management.

[0004] To identify power transmission line components in aerial images, existing detection methods use image processing to extract color, shape, and texture features from aerial images, thereby distinguishing power components from complex backgrounds. However, this typically requires manual feature extraction, and its performance depends heavily on the complexity of the background and the image processing results.

[0005] With the development of deep learning, methods based on CNNs (Convolutional Neural Networks) have been gradually applied to power transmission line component detection and have achieved considerable progress. Current CNN-based object detection methods can be summarized as region proposal-based methods (two-stage detectors) and regression-based methods (one-stage detectors). In two-stage methods, the first stage generates region proposals based on feature maps extracted by the CNN using a region proposal network (RPN); the second stage performs classification and bounding box regression on these region proposals. One-stage methods, on the other hand, combine the two stages into a single one, using regression by taking an input image and learning class probabilities and bounding box coordinates. Although deep learning-based object detection methods have been widely studied, they focus solely on information near the target region proposals in the image, propagating the target's visual features throughout the network while ignoring the complex topological structure between the targets. Consequently, the performance of these methods is somewhat limited.

[0006] Therefore, there are two main challenges in the hardware dataset: 1) Complex background. In the hardware aerial photography dataset, due to the wide distribution of hardware, the background of the hardware targets is also complex and diverse. The distribution background of the hardware includes fields, woods, and buildings. However, more often, the background is the pole tower itself. Since the material of the hardware is similar to that of the pole tower, both are iron or aluminum metal, and both appear silver-gray in color, the hardware targets are more likely to be hidden in the complex background in the aerial image and difficult to identify. 2) Dense targets. Since transmission lines carry the task of transmitting high voltage and ultra-high voltage, strong magnetic fields are generated around the transmission lines. Close-range shooting will interfere with the communication and navigation systems of the drone. Therefore, the hardware targets in the drone aerial images that can only be shot from a distance are small in size and densely distributed, making it easy for the image analysis algorithm to miss and misdetect.

[0007] Previous work proposed a new method for hardware target detection by integrating co-occurrence and spatial external knowledge into deep learning objectives. However, since this method requires the introduction of a large amount of external knowledge and the statistics and analysis of the original data set to obtain the characteristics of the data set, this method still has problems such as low generalization performance, susceptibility to the distribution of data in the training set, and the need for a lot of additional manual data preprocessing in statistics and analysis. Summary of the Invention

[0008] The purpose of the present invention is to provide a method and system for identifying multiple hardware fittings of a power transmission line by integrating implicit knowledge reasoning, which can improve the accuracy of hardware identification.

[0009] To achieve the above object, the present invention provides the following solutions:

[0010] A method for identifying multiple hardware components of power transmission lines that integrates implicit knowledge reasoning includes:

[0011] Obtaining an image of hardware to be identified;

[0012] Performing feature extraction on the hardware image to be identified, determining scene features, multiple region suggestion box features, and initial categories of the region suggestion box features in the hardware image to be identified;

[0013] For any region suggestion box feature, performing scene enhancement on the region suggestion box feature according to the scene feature to obtain a corresponding scene enhancement feature;

[0014] According to the initial category of the region proposal box feature, spatially enhance the region proposal box feature to obtain a corresponding spatially enhanced feature;

[0015] The hardware position and hardware type in the hardware image to be identified are determined based on the features of each region suggestion frame and the scene enhancement features and space enhancement features corresponding to the features of each region suggestion frame.

[0016] Optionally, the extracting features of the hardware image to be identified to determine scene features, multiple region suggestion box features, and initial categories of the region suggestion box features in the hardware image to be identified specifically includes:

[0017] Extracting features of the hardware image to be identified through a convolutional layer to obtain a basic feature map;

[0018] Determine multiple initial region proposal boxes and scene features in the basic feature map through a pre-trained region proposal network;

[0019] For any initial region proposal box, use the pre-trained Faster-RCNN detector to detect the initial region proposal box and determine the initial category and regression box of the initial region proposal box;

[0020] Resampling the regression frame of the initial region proposal frame to obtain an optimized region proposal frame corresponding to the initial region proposal frame;

[0021] According to each optimized region proposal box, a pooling operation is performed on the basic feature map to obtain multiple region proposal box features; the category of each region proposal box feature is the initial category of the corresponding initial region proposal box.

[0022] Optionally, performing scene enhancement on the region suggestion frame feature according to the scene feature to obtain a corresponding scene enhancement feature specifically includes:

[0023] Filtering the background in the region proposal frame feature to obtain a foreground frame feature;

[0024] Determining structural scene features based on the foreground frame features and the scene features based on a gated loop mechanism;

[0025] Based on a gated loop mechanism, the scene enhancement feature corresponding to the region proposal box is determined according to the structural scene feature and the region proposal box feature.

[0026] Optionally, spatially enhancing the region suggestion feature according to the initial category of the region suggestion feature to obtain a corresponding spatially enhanced feature specifically includes:

[0027] Calculating a spatial geometric relationship between the region suggestion frame feature and a target region suggestion frame feature; the target region suggestion frame feature is any region suggestion frame feature other than the region suggestion frame feature in the hardware image to be identified;

[0028] Calculating the spatial visual relationship between the region suggestion box feature and the target region suggestion box feature;

[0029] determining an implicit spatial relationship between the region suggestion frame feature and any target region suggestion frame feature based on a spatial geometric relationship and a spatial visual relationship between the region suggestion frame feature and each target region suggestion frame feature;

[0030] The implicit spatial relationship between the region proposal box features and each target region proposal box feature is sparsely represented to obtain a sparse spatial network;

[0031] Determining, according to the initial category of the region suggestion frame feature, a category appearance feature corresponding to the region suggestion frame feature;

[0032] Based on a graph convolutional network, a spatial enhancement feature corresponding to the region proposal feature is determined according to the category appearance feature corresponding to the region proposal feature and the sparse spatial network.

[0033] Optionally, the following formula is used to determine the spatial geometric relationship between the i-th region proposal box feature and the j-th region proposal box feature:

[0034]

[0035]

[0036] in, The spatial geometric relationship between the i-th region proposal box feature and the j-th region proposal box feature, W g is the predetermined spatial relationship weight, is the relative geometric position vector between the feature of the i-th region proposal box and the feature of the j-th region proposal box, w i The width of the proposed box feature for the i-th region, h i is the height of the feature proposal box for the i-th region, s i is the area of ​​the feature of the proposed box in the i-th region, w j The width of the proposed box feature for the jth region, h j The height of the proposed box feature for the jth region, s j is the area of ​​the feature of the proposed box in the jth region, W is the width of the hardware image to be identified, H is the height of the hardware image to be identified, S is the area of ​​the hardware image to be identified, (x i ,y i ) is the center point coordinate of the i-th region proposal feature, (x j ,y j ) is the center point coordinate of the j-th region proposal feature, i≠j.

[0037] Optionally, calculating the spatial visual relationship between the region suggestion box feature and the target region suggestion box feature specifically includes:

[0038] Performing a linear transformation on the region proposal box feature to obtain a latent space feature of the region proposal box feature;

[0039] Performing a nonlinear transformation on the target region suggestion box feature to obtain a latent space feature of the target region suggestion box feature;

[0040] The spatial visual relationship between the latent spatial feature of the region proposal box feature and the latent spatial feature of the target region proposal box feature is calculated.

[0041] Optionally, the following formula is used to determine the spatial visual relationship between the i-th region proposal box feature and the j-th region proposal box feature:

[0042]

[0043] in, is the spatial visual relationship between the feature of the proposed box of the i-th region and the feature of the proposed box of the j-th region, z i The latent space feature of the proposed box feature for the i-th region, z j The latent space features of the proposed box feature for the j-th region, is the transpose operation of the latent space features of the j-th region proposal box feature, || || is the L2 norm, i≠j.

[0044] Optionally, the following formula is used to determine the implicit spatial relationship between the i-th region proposal feature and the j-th region proposal feature:

[0045]

[0046] Among them, e ij is the implicit spatial relationship between the feature of the i-th region proposal box and the feature of the j-th region proposal box, The spatial geometric relationship between the i-th region proposal box feature and the j-th region proposal box feature, is the spatial visual relationship between the i-th region proposal box feature and the j-th region proposal box feature, m = 1, 2, ... M, M is the total number of region proposal box features in the hardware image to be identified, i ≠ j, m ≠ j.

[0047] Optionally, determining the hardware position and hardware type in the hardware image to be identified based on the features of each region suggestion frame, the scene enhancement features corresponding to the features of each region suggestion frame, and the space enhancement features corresponding to the features of each region suggestion frame specifically includes:

[0048] For any region proposal box feature, perform a full connection operation on the region proposal box feature to obtain a classification feature;

[0049] Performing a convolution operation on the region proposal box features to obtain regression features;

[0050] Cascading the classification feature with the scene enhancement feature corresponding to the region proposal box feature to obtain a classification cascade feature;

[0051] Cascading the regression feature with the spatial enhancement feature corresponding to the region proposal feature to obtain a positioning cascade feature;

[0052] Determining the type of hardware in the region suggestion frame feature according to the classification cascade feature;

[0053] The hardware position in the region suggestion frame feature is determined according to the positioning cascade feature.

[0054] To achieve the above object, the present invention also provides the following solution:

[0055] A transmission line multi-fitting identification system integrating implicit knowledge reasoning, including:

[0056] An image acquisition unit, used to acquire an image of a hardware to be identified;

[0057] a feature extraction unit connected to the image acquisition unit, configured to extract features from the hardware image to be identified, and determine scene features, multiple region suggestion box features, and initial categories of the region suggestion box features in the hardware image to be identified;

[0058] a scene enhancement unit connected to the feature extraction unit, configured to perform scene enhancement on any region suggestion box feature according to the scene feature to obtain a corresponding scene enhancement feature;

[0059] a spatial enhancement unit connected to the feature extraction unit, configured to perform spatial enhancement on any region proposal feature according to an initial category of the region proposal feature to obtain a corresponding spatial enhancement feature;

[0060] The hardware identification unit is connected to the feature extraction unit, the scene enhancement unit and the space enhancement unit, and is used to determine the hardware position and hardware type in the hardware image to be identified based on the feature of each area suggestion frame and the scene enhancement feature and space enhancement feature corresponding to each area suggestion frame feature.

[0061] According to the specific embodiment provided by the present invention, the present invention discloses the following technical effects: first, feature extraction is performed on the hardware image to be identified, and scene features, multiple region suggestion box features, and the initial category of each region suggestion box feature in the hardware image to be identified are determined; then, scene enhancement is performed on the region suggestion box features based on the scene features; and spatial enhancement is performed on the region suggestion box features based on the initial category of the region suggestion box features; finally, the hardware position and hardware type in the hardware image to be identified are determined based on each region suggestion box feature, the scene enhancement features, and the spatial enhancement features corresponding to each region suggestion box feature. Through scene enhancement and spatial enhancement, the implicit relationship between hardware is deeply explored, the visual features of obscured hardware are enhanced, and the problem of difficulty in detecting multiple hardware due to dense hardware targets and mutual obstruction in a complex background is solved, thereby improving the accuracy of hardware identification. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0063] Figure 1 This is a flow chart of the method for identifying multiple hardware fittings for power transmission lines that incorporates implicit knowledge reasoning according to the present invention;

[0064] Figure 2 This is a schematic diagram of the overall structure of the multi-metal decoupling detection model;

[0065] Figure 3 Schematic diagram of the scene reasoning module;

[0066] Figure 4 It is a schematic diagram of spatial relationship;

[0067] Figure 5 Schematic diagram of classification for decoupling detection module;

[0068] Figure 6 This is a schematic diagram of the module structure of the transmission line multi-fitting identification system that integrates implicit knowledge reasoning in the present invention.

[0069] Explanation of symbols:

[0070] Image acquisition unit-1, feature extraction unit-2, scene enhancement unit-3, space enhancement unit-4, hardware identification unit-5. DETAILED DESCRIPTION

[0071] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0072] The purpose of the present invention is to provide a method and system for identifying multiple hardware fittings in power transmission lines that integrates implicit knowledge reasoning. Through scene enhancement and spatial enhancement, the implicit relationship between hardware fittings is deeply explored, the visual characteristics of obscured hardware fittings are enhanced, and the problem of difficulty in detecting multiple hardware fittings caused by dense hardware targets and mutual obstruction in complex backgrounds is solved, thereby improving the accuracy of hardware identification.

[0073] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0074] like Figure 1 As shown, the transmission line multi-metal fittings identification method integrating implicit knowledge reasoning of the present invention includes:

[0075] S1: Obtain an image of the hardware to be identified. Specifically, use a drone to capture images of the hardware at a transmission line inspection site.

[0076] S2: extracting features from the hardware image to be identified, and determining scene features, multiple region suggestion box features, and initial categories of the region suggestion box features in the hardware image to be identified.

[0077] S3: For any region suggestion box feature, perform scene enhancement on the region suggestion box feature according to the scene feature to obtain a corresponding scene enhancement feature.

[0078] S4: According to the initial category of the region proposal box feature, spatially enhance the region proposal box feature to obtain a corresponding spatially enhanced feature.

[0079] S5: Determine the position and type of hardware in the hardware image to be identified based on the features of each region suggestion frame and the scene enhancement features and space enhancement features corresponding to the features of each region suggestion frame.

[0080] Steps S2-S5 of the present invention are processed by a pre-trained multi-metal decoupling detection model. Figure 2As shown, the multi-fitting decoupling detection model includes a feature extraction module, a scene reasoning module, a spatial reasoning module, and a decoupling detection module. The scene reasoning module is connected to the feature extraction module, the spatial reasoning module is connected to the feature extraction module, and the decoupling detection module is connected to the scene reasoning module and the spatial reasoning module. The feature extraction module, scene reasoning module, spatial reasoning module, and decoupling detection module are trained using a sample set to obtain a trained multi-fitting decoupling detection model. The sample set includes multiple sample images and the fitting position labels and fitting category labels in each sample image. That is, the multi-fitting decoupling detection model includes a training phase and a recognition phase. The steps of the present invention are the recognition phase.

[0081] Among them, the feature extraction module includes a convolutional neural network, a region proposal network, a Faster-RCNN detector, a resampling model and a box decoder connected in sequence. The scene reasoning module includes a scene perception GRU (gated recurrent unit) and a scene reasoning GRU. The spatial reasoning module includes a spatial geometric relationship determination unit, a spatial visual relationship determination unit, an implicit spatial relationship determination unit and two layers of GCN (Graph Convolutional Networks). The decoupled detection module includes a fully connected layer, a convolutional layer, two cascade layers, a positioning layer and a classification layer.

[0082] Furthermore, step S2 includes:

[0083] S21: extracting features of the hardware image to be identified through a convolutional layer to obtain a basic feature map. Specifically, extracting features of the hardware image to be identified through a convolutional neural network.

[0084] S22: Determine a plurality of initial region proposal frames and scene features in the basic feature map. Specifically, determine the region proposal frames and scene features in the basic feature map through a region proposal network.

[0085] S23: For any initial region proposal frame, detect the initial region proposal frame and determine the initial category and regression frame of the initial region proposal frame. Specifically, the initial region proposal frame is detected by using a pre-trained Faster-RCNN detector. During the training phase of the multi-metal decoupling detection model, the classification weights of the Faster-RCNN detector are continuously adjusted to obtain the optimal classification weights to obtain a trained Faster-RCNN detector. Each initial region proposal frame shares the same classification weight.

[0086] S24: Resampling the regression frame of the initial region proposal frame to obtain an optimized region proposal frame corresponding to the initial region proposal frame.

[0087] Specifically, the categories and regression boxes are first resampled through the Resampling model. Based on the baseline model, a resampling process similar to the Cascade-RCNN model is first implemented. Then, the coordinates of the regression boxes are resampled as RoIs through the Boxdecoder. Specifically, the bbox detection box generated in the previous level is used as the initial region proposal box, resulting in relatively accurate region proposal boxes. This method can alleviate the error propagation problem existing in the pipeline model to a certain extent through subsequent scene enhancement and spatial enhancement, enabling better and faster model convergence during training.

[0088] S25: Based on each optimized region proposal, a pooling operation is performed on the basic feature map to obtain a plurality of region proposal features. The category of each region proposal feature is the category of the corresponding initial region proposal.

[0089] Step S3 specifically includes:

[0090] S31: Filter the background in the region proposal frame feature to obtain a foreground frame feature.

[0091] S32: Based on the gated recurrent mechanism, the structural scene features are determined according to the foreground frame features and the scene features. Specifically, the foreground frame features serve as the input of the scene-aware GRU (gated recurrent unit), and the scene features serve as the initial hidden layer vector of the scene-aware GRU.

[0092] like Figure 3 As shown in Figure 1, GRU consists of two gates, the reset gate and the update gate. The update gate controls the previous information that will be brought to the current layer, while the reset gate determines the amount of past information to be forgotten, that is, retaining the output of the past at a certain ratio. Update gate z t and reset gate r t The equation is as follows:

[0093] z t =σ(W z x t +U z h t-1 );

[0094] r t =σ(W r x t +U r h t-1 );

[0095] Among them, x t is the input of the foreground frame feature at time t. t When it is input to GRU, it is multiplied by the weight W zThe scene feature h generated by the previous hidden layer t-1 Multiply by its weight U z , then add the two results together and use a sigmoid function σ as the activation function to generate a probability value between 0 and 1. The closer the value is to 1, the more it means retaining the hidden state of the previous time step. The closer the value is to 0, the more it means resetting the corresponding hidden state element to 0. At the initial moment, h t is the scene feature, and thus it is continuously updated. t and reset gate r t The value of the process, the weight W z , U z , W r , U r The vectors are learned during the training phase.

[0096]

[0097] in, is the candidate hidden state information, W o and U o are the weights of GRU. Using the reset gate r t Stores control foreground frame features x t The fused information. Also use sigmoid function as activation function, Is the element-wise product function of two vectors. The hidden layer output h at time t is calculated by the following formula: t , according to the input x t The sequence state is iterated from the first time point to the last time point, and the final output updated hidden layer features are calculated as the output of GRU, as follows:

[0098]

[0099] Finally, the output of GRU obtains the structural scene feature f′ sc .

[0100] S33: Based on the gated loop mechanism, determine the scene enhancement features corresponding to the region proposal box according to the structural scene features and the region proposal box features. Specifically, the structural scene features serve as the input of the scene reasoning GRU, and the region proposal box features serve as the initial hidden layer features of the scene reasoning GRU.

[0101] In order to achieve iterative reasoning of object details from scene information, M region proposal box features are used as the initial hidden layer state of the scene reasoning GRU, and the structural scene features are used as the input of the scene reasoning GRU model. The GRU can choose to ignore some object states that are not related to the scene environment, or use the scene environment information to enhance some object states. Finally, the output hidden layer vector of the scene reasoning GRU model obtains the scene enhancement feature of the region proposal box feature.

[0102] Step S4 specifically includes:

[0103] S41: Calculating a spatial geometric relationship between the region suggestion frame feature and a target region suggestion frame feature. The target region suggestion frame feature is any region suggestion frame feature other than the region suggestion frame feature in the hardware image to be identified.

[0104] Specifically, to mine implicit spatial knowledge, we design a set of relationship variables to represent the spatial relationship between two region proposal features. We take the relative geometric features of the region proposal features as input and then project them into a subspace. The following formula is used to determine the spatial geometric relationship between the i-th region proposal feature and the j-th region proposal feature:

[0105]

[0106]

[0107] in, The spatial geometric relationship between the i-th region proposal box feature and the j-th region proposal box feature, W g is the predetermined spatial relationship weight, is the relative geometric position vector between the feature of the i-th region proposal box and the feature of the j-th region proposal box, w i The width of the proposed box feature for the i-th region, h i The height of the proposed box feature of the i-th region, s i is the area of ​​the feature of the proposed box in the i-th region, w j The width of the proposed box feature for the jth region, h j The height of the proposed box feature for the jth region, s j is the area of ​​the feature of the proposed box in the jth region, W is the width of the hardware image to be identified, H is the height of the hardware image to be identified, S is the area of ​​the hardware image to be identified, (x i ,y i ) is the center point coordinate of the i-th region proposal feature, (x j ,y j ) is the center point coordinate of the j-th region proposal feature, i≠j. In this embodiment, W gThe optimal spatial relationship weights determined during the training phase.

[0108] By setting the relative geometric position vector to 12 dimensions, this method not only describes the relative distance between the i-th region proposal feature and the j-th region proposal feature, as well as the relative orientation between the region proposals, but also embeds information about the mutual occlusion between hardware annotation frames, thus assisting the model in processing images with densely packed objects. Furthermore, due to the uncertainties of drone inspection photography environments, the relative position of hardware is easily affected by the shooting angle. By incorporating the dimensional characteristics of the input image itself, the geometric relationship features are insulated from changes in image scale and movement.

[0109] S42: Calculate the spatial visual relationship between the region proposal feature and the target region proposal feature. The hardware image to be identified contains a large number of region proposal features with occlusion relationships. The more severe the occlusion between the region proposal features, the more significant the overlap between the two region proposal features on the image. This results in a higher visual feature similarity between the region proposal features. The degree of overlap between the region proposal features can be inferred from this visual feature similarity, thereby mining implicit spatial information. Therefore, introducing RoIs visual features can assist in the prediction of dense targets.

[0110] Specifically, a nonlinear transformation is performed on the two region proposal features to obtain the corresponding latent space features of the two region proposal features, and the spatial visual relationship between the two latent space features is calculated.

[0111] The following formula is used to determine the spatial visual relationship between the i-th region proposal feature and the j-th region proposal feature:

[0112]

[0113] in, is the spatial visual relationship between the feature of the proposed box of the i-th region and the feature of the proposed box of the j-th region, z i The latent space feature of the proposed box feature for the i-th region, z j The latent space features of the proposed box feature for the j-th region, is the transpose operation of the latent space features of the j-th region proposal box feature, || || is the L2 norm, i≠j.

[0114] Specifically, the region proposal box feature is used as input. First, the nonlinear transformation z i =φ(f i ) transforms the region proposal box features into latent space features, and the set of all region latent space features is Among them, f i Propose box features for the i-th region, z iis the i-th latent space feature, E is the dimension of the latent space feature, φ() is a nonlinear function, and a fully connected layer with a Relu activation function is used as a nonlinear transformation function. The spatial visual relationship is R v =ZZ T .

[0115] S43: Determine an implicit spatial relationship between the region suggestion frame feature and any target region suggestion frame feature based on the spatial geometric relationship and spatial visual relationship between the region suggestion frame feature and each target region suggestion frame feature.

[0116] Specifically, we use a weighted combination method to fuse the spatial geometric relationship and the spatial visual relationship to obtain the implicit spatial relationship between the region proposals. The following formula is used to determine the implicit spatial relationship between the i-th region proposal feature and the j-th region proposal feature:

[0117]

[0118] Among them, e ij is the implicit spatial relationship between the feature of the i-th region proposal box and the feature of the j-th region proposal box, The spatial geometric relationship between the i-th region proposal box feature and the j-th region proposal box feature, is the spatial visual relationship between the i-th region proposal box feature and the j-th region proposal box feature, m = 1, 2, ... M, M is the total number of region proposal box features in the hardware image to be identified, i ≠ j, m ≠ j.

[0119] The implicit spatial relationship weight is calculated mainly through the spatial geometric relationship weight and the spatial visual relationship weight. The denominator of the fraction is the sum of the implicit spatial relationship weights of the i-th region proposal box feature and all the region proposal box features in the set. The purpose is to normalize the spatial relationship weight of the i-th region proposal box feature. The numerator is the spatial relationship between the i-th region proposal box feature and the j-th region proposal box feature. The purpose is to normalize the spatial relationship. Figure 4 Shown is a schematic diagram of the determination of spatial relationships.

[0120] S44: Sparsifying the implicit spatial relationships between the region proposal features and each target region proposal feature to obtain a sparse spatial network. Specifically, a sparse method is used to filter the implicit spatial relationships. For each region proposal feature, only the k region proposal features with the greatest relationships are retained, thereby selecting more relevant points as its neighboring nodes, thereby effectively sparsifying the adjacency matrix.

[0121] S45: Determine, according to the initial category of the region suggestion frame feature, a category appearance feature corresponding to the region suggestion frame feature.

[0122] By the formula X = M × W c Get graph node features W c is the classification weight of the Faster-RCNN detector.

[0123] Specifically, in order to achieve the mapping between the region suggestion box and the category, the features of the corresponding region suggestion box node are obtained. The region suggestion box features are combined with the category appearance features by using soft connection, and the classification probability distribution is used. To map, N is the classification probability, which is the classification result of the Faster-RCNN detector on the region proposal box feature.

[0124] Different from the region proposal box visual features, the category appearance features can be regarded as a high-level visual representation of a specific category of objects. The weights are used as the visual embedding of each category. Formally, let is the classification weight in the Faster-RCNN detector, where C is the number of categories and D b The dimension of the visual feature of the region proposal box is obtained by copying the classification weight W of the Faster-RCNN detector in step S2. c , we can get the category visual embedding, that is, the category appearance features.

[0125] During training, W c The visual embedding becomes more accurate over time. In addition, the multi-tool disentangled detection model can be trained in an end-to-end manner, avoiding averaging or clustering all data.

[0126] S46: Based on the graph convolutional network, determine the spatial enhancement feature corresponding to the region proposal feature according to the category appearance feature corresponding to the region proposal feature and the sparse spatial network. In this embodiment, a two-layer graph convolutional network is used to determine the spatial enhancement feature corresponding to the region proposal feature.

[0127] Step S5 specifically includes:

[0128] S51: For any region proposal box feature, perform a full connection operation on the region proposal box feature to obtain a classification feature.

[0129] S52: Perform a convolution operation on the region proposal box feature to obtain a regression feature.

[0130] The region proposal box features pass through the fully connected layer and the residual convolution layer respectively to generate classification features and regression features.

[0131] Because classification and regression have different sensitivities to features, a mismatch between the optimal category score and the optimal bounding box can easily occur. Furthermore, the scene reasoning module integrates scene information to facilitate the reasoning and discrimination of hardware categories, while the spatial reasoning module exploits implicit spatial information to facilitate the localization and regression of proposal boxes in dense regions. Therefore, the two modules have different focuses on feature representation and processing, as well as on classification and regression tasks.

[0132] S53: Cascading the classification feature with the scene enhancement feature corresponding to the region proposal feature to obtain a classification cascade feature.

[0133] S54: Cascading the regression feature with the spatial enhancement feature corresponding to the region proposal feature to obtain a positioning cascade feature.

[0134] S55: Determine the hardware type in the region suggestion frame feature according to the classification cascade feature.

[0135] S56: Determine the hardware position in the region suggestion frame feature according to the positioning cascade feature.

[0136] S57: Determine the hardware position and hardware type in the hardware image to be identified based on the hardware type and hardware position in the feature of each area suggestion frame. Figure 5 Shown is a schematic diagram of the localization and classification process.

[0137] In response to the demand for multi-metal fitting target detection and the problems of complex background and dense targets encountered in the metal fitting detection process, the present invention proposes a multi-metal fitting decoupling detection model that integrates implicit knowledge reasoning. By deeply exploring the implicit relationship between metal fitting targets and using multiple reasoning methods to enhance the visual features of occluded metal fittings, the instantiation expression of implicit knowledge is realized, and the decoupling detection module is used to separate the classification and regression tasks for training. It can accurately identify the position and category of metal fittings in complex environmental backgrounds or when metal fittings are densely occluded.

[0138] This invention effectively solves the problem of difficulty in detecting multiple hardware fittings due to dense and mutual occlusion of hardware fittings in complex backgrounds. The detection effect on the multi-fitting data set of transmission line is better than other advanced models. Compared with the original baseline model, the accuracy rate is improved by 10.8%. Qualitative experiments show that the algorithm can solve the problem of target occlusion and realize the detection of dense multiple targets, providing new ideas for the intelligent inspection technology of transmission line equipment components.

[0139] In order to better understand the solution of the present invention, the training process of the multi-metal fitting decoupling detection model of the present invention is further described below.

[0140] First, we constructed a dataset using sample images captured by drones during transmission line inspections. We used the labelimg tool to annotate the locations and categories of hardware for training and testing. In this example, we selected 14 types of hardware for the experiment, with 1,092 training and 363 test image samples, respectively, for a total of 9,098 object boxes.

[0141] Each sample image is then used as input. Basic features are extracted through a convolutional neural network. Region proposal boxes are obtained from the basic features through a region proposal network. The region proposal boxes are passed through Faster-RCN to obtain coarse-grained categories and regression boxes. The coarse-grained categories and regression boxes are resampled through the Resampling model. The coordinates of the regression boxes are resampled as RoIs regions through Boxdecoder. The basic features are pooled again according to the new RoI regions to obtain the scene features f sc and region proposal feature set in D b is the dimension of the visual feature of the region proposal box, M is the number of region proposal box features, and f i Propose box features for the i-th region.

[0142] The region proposal box features are then input into the scene perception reasoning module to obtain the scene enhancement features of the corresponding region proposal box. In order to make the training easier to converge, the same positive and negative sample sampling strategy as Faster-RCNN is used. The IOU with the real box is greater than 0.5 as the foreground, and the IOU with the real box is less than 0.1 as the background. Therefore, the M region proposal box features are divided into two categories: foreground and background. Since the scene implicit information to be extracted is related to the characteristics of the target, the foreground box filtering operation is used to filter the background information and retain the foreground box features f fg , infer the corresponding scene information through effective foreground features and extract the corresponding scene features. Filter the foreground frame of the original region proposal frame features to obtain the foreground frame features that only contain the hardware. fg .

[0143] In order to improve the expressive power of structural scene features, scene features and foreground frame features are introduced into GRU respectively to obtain structural scene features f′ sc In order to learn scene enhancement features, the structural scene feature f′ sc The feature f of the region proposal is copied as the input of the scene reasoning GRU through the broadcast mechanism. i as the initial hidden layer features.

[0144] The spatial reasoning module then simulates the spatial relationship between the features of the two region proposals. The spatial relationship reasoning module then uses the category appearance features and the sparse spatial relationship as input to generate spatially enhanced features. Spatial information is extracted and mined, and implicit spatial knowledge is expressed using a graph neural network. The resulting spatially enhanced visual features assist in the localization and regression of the target box. The features of each node obtained through graph convolution are the spatially enhanced features.

[0145] In order to effectively propagate implicit spatial information and enable the model to adaptively acquire spatial information for different image targets, the graph convolutional network is introduced to perform graph reasoning learning on spatial knowledge. For the GCN model, it focuses more on learning the structure of the topological graph, where the input of the GCN module is the feature vector of the region proposal box. The implicit spatial information is propagated and learned through GCN to obtain the spatial knowledge enhancement features of the corresponding region proposal box.

[0146] For graph convolutional networks, given a graph signal The normalized Laplace matrix can be expressed as L = I K -D -1 / 2 AD -1 / 2 , where A is the adjacency matrix, D is the degree matrix, K is the dimension of the graph signal x, and I K is a unit matrix with dimension K. Graph convolution is equivalent to the product of the graph signal x and the filter kernel g in the spectral domain, and Fourier transform is performed on this basis. The matrix-vector form of graph convolution can be expressed as follows:

[0147] x*g θ =UG θ (Λ)U T x=G θ (UΛU T )x=G θ (L)x;

[0148] Among them, x*g θ is the convolution formula, g θ is the Laplace function, Λ is the eigenvalue matrix, U is the degree matrix, and its column vector is L=UΛU T The eigenvalue after eigendecomposition, G θ (L) is a diagonal matrix, and the object elements are parameterized by the function eigenvalue θ. On this basis, according to the Chebyshev expansion of the graph Laplace, the formula can be obtained:

[0149] x*g θ ≈θ(I K +D -1 / 2 AD -1 / 2 )x;

[0150] Among them, the above formula represents the first-order approximate form of graph convolution, and the general form of the graph convolution network can be obtained as follows:

[0151]

[0152] in, is the normalized adjacency matrix with self-connection, X l and X l+1 are the input and output of graph convolution, W l is the learned weight matrix. The graph convolutional network propagates information between nodes through the adjacency matrix, mining the implicit spatial relationship information between target boxes. The input of the graph convolutional network is the final spatial enhancement feature.

[0153] The present invention does not improve the loss function, and the loss function used is the loss function of Faster-RCNN itself. During the training process, the weights of each part of the model are adjusted according to the loss function (the classification weights W of the Faster-RCNN classifier are 1 and 2). c , spatial relationship weight W g , the weight matrix W in the graph convolutional network l , the weight vector W of GRU z , U z , W r , U r ) until the loss function converges and a trained multi-metal decoupling detection model is obtained.

[0154] In order to mine the combined structural knowledge between hardware fittings, the scene perception method in the scene reasoning module is used to mine the scene information in each image, and the inference network is further used to effectively fuse the scene information; in order to mine the spatial position association knowledge between hardware fittings, the spatial perception network in the spatial reasoning module is used to extract the spatial visual feature relationship and geometric feature relationship, and the graph convolutional network is further used to realize spatial knowledge reasoning; finally, the decoupling module is used to train and learn the hardware inspection task in a non-coupled manner to make the final inspection decision.

[0155] like Figure 6 As shown, the transmission line multi-fitting recognition system integrating implicit knowledge reasoning of the present invention includes: an image acquisition unit 1, a feature extraction unit 2, a scene enhancement unit 3, a space enhancement unit 4 and a fitting recognition unit 5.

[0156] The image acquisition unit 1 is used to acquire the image of the hardware to be identified.

[0157] The feature extraction unit 2 is connected to the image acquisition unit 1, and is used to extract features from the hardware image to be identified, and determine scene features, multiple region suggestion box features, and categories of each region suggestion box feature in the hardware image to be identified.

[0158] The scene enhancement unit 3 is connected to the feature extraction unit 2, and is used to perform scene enhancement on any region suggestion frame feature according to the scene feature to obtain a scene enhancement feature corresponding to the region suggestion frame feature.

[0159] The spatial enhancement unit 4 is connected to the feature extraction unit 2, and is used to perform spatial enhancement on any region suggestion frame feature according to the initial category of the region suggestion frame feature to obtain a corresponding spatial enhancement feature.

[0160] The hardware identification unit 5 is connected to the feature extraction unit 2, the scene enhancement unit 3 and the space enhancement unit 4. The hardware identification unit 5 is used to determine the hardware position and hardware type in the hardware image to be identified based on the features of each area suggestion frame, the scene enhancement features and the space enhancement features corresponding to the features of each area suggestion frame.

[0161] Compared with the prior art, the transmission line multi-fitting identification system integrating implicit knowledge reasoning of the present invention has the same beneficial effects as the above-mentioned transmission line multi-fitting identification method integrating implicit knowledge reasoning, which will not be repeated here.

[0162] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0163] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The above examples are only intended to help understand the method and core concept of the present invention. At the same time, those skilled in the art will find that the specific implementation methods and application scopes may vary based on the concept of the present invention. In summary, the contents of this specification should not be construed as limiting the present invention.

Claims

1. A method for identifying multiple hardware components of a power transmission line by integrating implicit knowledge reasoning, characterized in that: The method for identifying multiple hardware components of a transmission line by integrating implicit knowledge reasoning includes: Obtaining an image of hardware to be identified; Performing feature extraction on the hardware image to be identified, determining scene features, multiple region suggestion box features, and initial categories of the region suggestion box features in the hardware image to be identified; For any region proposal box feature, the background in the region proposal box feature is filtered to obtain the foreground box feature; the foreground box feature is used as the input of the scene perception gated recurrent unit, and the scene feature is used as the initial hidden layer vector of the scene perception gated recurrent unit to obtain the structural scene feature; the structural scene feature is used as the input of the scene reasoning gated recurrent unit, and the region proposal box feature is used as the initial hidden layer feature of the scene reasoning gated recurrent unit. The scene reasoning gated recurrent unit chooses to ignore some object states that are not related to the scene environment, or uses the scene environment information to enhance some object states. Finally, the output hidden layer vector of the scene reasoning gated recurrent unit obtains the scene enhancement feature of the region proposal box feature; Calculate the spatial geometric relationship between the region suggestion frame feature and the target region suggestion frame feature; the target region suggestion frame feature is any region suggestion frame feature in the hardware image to be identified except the region suggestion frame feature; use the following formula to determine the spatial geometric relationship between the i-th region suggestion frame feature and the j-th region suggestion frame feature: in, The spatial geometric relationship between the i-th region proposal box feature and the j-th region proposal box feature, W g is the predetermined spatial relationship weight, is the relative geometric position vector between the feature of the i-th region proposal box and the feature of the j-th region proposal box, w i The width of the proposed box feature for the i-th region, h i is the height of the feature proposal box for the i-th region, s i is the area of ​​the feature of the proposed box in the i-th region, w j The width of the proposed box feature for the jth region, h j The height of the proposed box feature for the jth region, s j is the area of ​​the feature of the proposed box in the jth region, W is the width of the hardware image to be identified, H is the height of the hardware image to be identified, S is the area of ​​the hardware image to be identified, (x i ,y i ) is the center point coordinate of the i-th region proposal feature, (x j ,y j ) is the center point coordinate of the j-th region proposal feature, i≠j; Calculating the spatial visual relationship between the region suggestion box feature and the target region suggestion box feature; determining an implicit spatial relationship between the region suggestion frame feature and any target region suggestion frame feature based on a spatial geometric relationship and a spatial visual relationship between the region suggestion frame feature and each target region suggestion frame feature; The implicit spatial relationship between the region proposal box features and each target region proposal box feature is sparsely represented to obtain a sparse spatial network; Determining, according to the initial category of the region suggestion frame feature, a category appearance feature corresponding to the region suggestion frame feature; Determine, based on a graph convolutional network, the spatial enhancement features corresponding to the region proposal features according to the category appearance features corresponding to the region proposal features and the sparse spatial network; The hardware position and hardware type in the hardware image to be identified are determined based on the features of each region suggestion frame and the scene enhancement features and space enhancement features corresponding to the features of each region suggestion frame.

2. The method for identifying multiple hardware components of a power transmission line by integrating implicit knowledge reasoning according to claim 1 is characterized in that: The feature extraction of the hardware image to be identified to determine the scene features, multiple region suggestion box features, and the initial category of each region suggestion box feature in the hardware image to be identified specifically includes: Extracting features of the hardware image to be identified through a convolutional layer to obtain a basic feature map; Determining a plurality of initial region proposal boxes and scene features in the basic feature map; For any initial region proposal frame, detect the initial region proposal frame and determine the initial category and regression frame of the initial region proposal frame; Resampling the regression frame of the initial region proposal frame to obtain an optimized region proposal frame corresponding to the initial region proposal frame; According to each optimized region proposal box, a pooling operation is performed on the basic feature map to obtain multiple region proposal box features; the category of each region proposal box feature is the initial category of the corresponding initial region proposal box.

3. The method for identifying multiple hardware components of a power transmission line by integrating implicit knowledge reasoning according to claim 1 is characterized in that: The calculating the spatial visual relationship between the region suggestion frame feature and the target region suggestion frame feature specifically includes: Performing a linear transformation on the region proposal box feature to obtain a latent space feature of the region proposal box feature; Performing a nonlinear transformation on the target region suggestion box feature to obtain a latent space feature of the target region suggestion box feature; The spatial visual relationship between the latent spatial feature of the region proposal box feature and the latent spatial feature of the target region proposal box feature is calculated.

4. The method for identifying multiple hardware components of a power transmission line by integrating implicit knowledge reasoning according to claim 3 is characterized in that: The following formula is used to determine the spatial visual relationship between the i-th region proposal feature and the j-th region proposal feature: in, is the spatial visual relationship between the feature of the proposed box of the i-th region and the feature of the proposed box of the j-th region, z i The latent space feature of the proposed box feature for the i-th region, z j The latent space features of the proposed box feature for the j-th region, is the transpose operation of the latent space features of the j-th region proposal box feature, |||| is the L2 norm, i≠j.

5. The method for identifying multiple hardware components of a power transmission line by integrating implicit knowledge reasoning according to claim 1 is characterized in that: The following formula is used to determine the implicit spatial relationship between the i-th region proposal feature and the j-th region proposal feature: Among them, e ij is the implicit spatial relationship between the feature of the i-th region proposal box and the feature of the j-th region proposal box, The spatial geometric relationship between the i-th region proposal box feature and the j-th region proposal box feature, is the spatial visual relationship between the i-th region proposal box feature and the j-th region proposal box feature, m = 1, 2, ... M, M is the total number of region proposal box features in the hardware image to be identified, i ≠ j, m ≠ j.

6. The method for identifying multiple hardware components of a power transmission line by integrating implicit knowledge reasoning according to claim 1 is characterized in that: The determining of the hardware position and hardware type in the hardware image to be identified based on the features of each region suggestion frame and the scene enhancement features and space enhancement features corresponding to the features of each region suggestion frame specifically includes: For any region proposal box feature, perform a full connection operation on the region proposal box feature to obtain a classification feature; Performing a convolution operation on the region proposal box features to obtain regression features; Cascading the classification feature with the scene enhancement feature corresponding to the region proposal box feature to obtain a classification cascade feature; Cascading the regression feature with the spatial enhancement feature corresponding to the region proposal feature to obtain a positioning cascade feature; Determining the type of hardware in the region suggestion frame feature according to the classification cascade feature; The hardware position in the region suggestion frame feature is determined according to the positioning cascade feature.

7. A transmission line multi-fitting identification system integrating implicit knowledge reasoning, applied to the transmission line multi-fitting identification method integrating implicit knowledge reasoning according to any one of claims 1-6, characterized in that: The transmission line multi-fitting identification system integrating implicit knowledge reasoning includes: An image acquisition unit, used to acquire an image of a hardware to be identified; a feature extraction unit connected to the image acquisition unit, configured to extract features from the hardware image to be identified, and determine scene features, multiple region suggestion box features, and initial categories of the region suggestion box features in the hardware image to be identified; a scene enhancement unit connected to the feature extraction unit, configured to perform scene enhancement on any region suggestion box feature according to the scene feature to obtain a corresponding scene enhancement feature; a spatial enhancement unit connected to the feature extraction unit, configured to perform spatial enhancement on any region proposal feature according to an initial category of the region proposal feature to obtain a corresponding spatial enhancement feature; The hardware identification unit is connected to the feature extraction unit, the scene enhancement unit and the space enhancement unit, and is used to determine the hardware position and hardware type in the hardware image to be identified based on the feature of each area suggestion frame and the scene enhancement feature and space enhancement feature corresponding to each area suggestion frame feature.

Images