Anti-vibration hammer detection method and system based on prior knowledge and data fusion driving

Abstract

The present application relates to the technical field of anti-chug hammer defect detection, and proposes an anti-chug hammer detection method and system based on prior knowledge and data fusion driving, which comprises: performing multi-scale feature extraction on the anti-chug hammer image to obtain positioning features and classification features; extracting positioning prior knowledge representing the positioning position area of the anti-chug hammer to construct an anti-chug hammer spatial prior distribution heat map; fusing the positioning features and the anti-chug hammer spatial prior distribution heat map to obtain positioning prediction features; constructing a defect class co-occurrence matrix, splicing the classification features and the defect class co-occurrence matrix through a gating mechanism to obtain an adjacency matrix, solving the class prediction features by using graph convolution, and generating an anti-chug hammer prediction result. The present application innovatively fuses spatial prior knowledge and graph convolution network, and performs intelligent detection of anti-chug hammer defects based on anti-chug hammer spatial distribution data and class co-occurrence relationship of defect classes, thereby improving the accuracy, robustness and on-site adaptability of defect recognition.

Description

Technical Field

[0001] This invention relates to the technical field of vibration damper defect detection, specifically to a vibration damper detection method and system driven by prior knowledge and data fusion. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] In power transmission and distribution line systems, vibration dampers are widely used as a critical protective device in overhead transmission lines to suppress conductor vibration and galloping caused by wind. Their main function is to increase the damping of the conductor system through their own mass and structural characteristics, reducing the amplitude and frequency of wind-induced vibrations, thereby effectively preventing fatigue damage, strand breakage, and other faults in the conductors under long-term vibration. The effective operation of vibration dampers is of great significance for ensuring the safe, stable, and efficient operation of the power grid system. Especially in areas with frequent wind loads, their performance directly affects the line lifespan and maintenance costs. However, vibration dampers are exposed to the outdoor environment for extended periods, often facing erosion from complex climatic conditions such as strong winds, rain, snow, salt spray, and temperature differences. They are prone to defects such as loosening, wear, cracks, corrosion, and breakage. If these defects are not detected and addressed in time, they can easily lead to the failure of vibration damping performance, potentially causing serious power accidents.

[0004] Currently, drones equipped with camera systems are widely used in power transmission line inspections to acquire high-altitude images and remotely monitor the condition of components such as vibration dampers. However, due to the complex backgrounds, variable lighting, and small size of defects in the inspection images, existing image recognition and defect detection methods still face many challenges. Firstly, vibration dampers are situated in natural background environments (such as trees, blue skies, poles, and fittings), and their textures and colors may be highly similar to those of defect targets, easily leading to false positives in traditional detection algorithms. Traditional processing methods struggle to effectively distinguish between background noise and actual defects, reducing the robustness of detection. Secondly, vibration damper defects typically exhibit small target features, with small dimensions and blurred details. Different defect categories are highly similar in appearance; for example, the edge features of looseness and wear may be difficult to distinguish, leading to frequent misclassification issues in one-step detection. On the other hand, in actual line inspection images, the textures and lighting variations of complex backgrounds can easily be confused with the defect features of vibration dampers, making it difficult for algorithms to accurately distinguish between targets and background during feature extraction, increasing the risk of false positives and false negatives. The high similarity and coupling between different defect types leads to blurred category boundaries, making it easy for algorithms to misclassify categories, especially in small object detection, where the difficulty of distinguishing subtle features is further aggravated. Summary of the Invention

[0005] To address the aforementioned issues, this invention proposes a vibration damper detection method and system driven by prior knowledge and data fusion. It innovatively integrates spatial prior knowledge with graph convolutional networks to intelligently detect vibration damper defects based on the spatial distribution data of vibration dampers and the co-occurrence relationship of defect categories, thereby improving the accuracy, robustness, and field adaptability of defect identification.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] One or more embodiments provide a vibration damper detection method based on prior knowledge and data fusion, including the following steps:

[0008] Images of the vibration damper are acquired, multi-scale feature extraction is performed, and the extracted features are fused to obtain localization features and classification features including shallow features, medium features, and deep features.

[0009] For the acquired vibration damper images, edge feature extraction and probabilistic Hough transform operations are fused to detect the conductor region, and prior knowledge representing the location of the vibration damper is extracted to construct a spatial prior distribution heatmap of the vibration damper.

[0010] The localization features are convolved, pooled and stitched together with the spatial prior distribution heatmap of the vibration damper. After fusion, feature selective enhancement is performed to obtain the localization prediction features.

[0011] Obtain prior knowledge of defect classification, calculate co-occurrence probabilities, and construct a defect category co-occurrence matrix;

[0012] The adjacency matrix is ​​obtained by concatenating the classification features and the defect category co-occurrence matrix through a gating mechanism. The node feature matrix is ​​obtained by convolving the classification features. The category prediction features are solved by graph convolution. The localization prediction features and the category prediction features are fused to generate the vibration damper prediction results.

[0013] One or more embodiments provide a vibration damper detection system driven by prior knowledge and data fusion, including an image acquisition device and a processor;

[0014] Image acquisition device, used to acquire images of the vibration damper;

[0015] The processor is configured to perform the steps in the above-described vibration damper detection method based on prior knowledge and data fusion.

[0016] One or more embodiments provide a vibration damper detection system based on prior knowledge and data fusion, including:

[0017] The backbone network and the neck network are configured to acquire images of the vibration damper, perform multi-scale feature extraction, and fuse the extracted features to obtain localization features and classification features including shallow features, medium features and deep features.

[0018] The prior knowledge extraction module is configured to extract prior knowledge representing the location of the vibration damper from the acquired vibration damper image by fusing edge feature extraction and probabilistic Hough transform operation to detect the guide wire region and construct a spatial prior distribution heatmap of the vibration damper.

[0019] The prior distribution localization module is configured to perform convolution, pooling and stitching fusion on the localization features and the spatial prior distribution heatmap of the vibration damper. After fusion, feature selective enhancement is performed to obtain localization prediction features.

[0020] The prior distribution localization module is configured to acquire prior knowledge of defect classification, calculate co-occurrence probabilities, and construct a defect category co-occurrence matrix.

[0021] The co-occurrence matrix-guided graph classification module concatenates the co-occurrence matrices of classification features and defect categories through a gating mechanism to obtain an adjacency matrix. It then performs convolution on the classification features to obtain a node feature matrix and uses graph convolution to solve for the category prediction features. Finally, it fuses the location prediction features and the category prediction features to generate the vibration damper prediction results.

[0022] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0023] This invention introduces a multi-scale fusion strategy in the image processing stage, significantly improving the ability to express features at different levels and ensuring precise identification of small target defects. In the localization stage, the introduction of spatial prior heatmaps effectively enhances the model's robustness in complex backgrounds, significantly reducing false positive and false negative rates. By combining probabilistic Hough transform and edge extraction mechanisms, accurate detection of conductor regions is achieved, thereby more precisely constraining the spatial distribution of vibration dampers. The introduction of a defect category co-occurrence matrix and graph convolution mechanism effectively captures the intrinsic connections and distinguishing features between defect categories, improving the accuracy of multi-category defect identification. The overall system combines a data-driven and knowledge-guided dual strategy, enhancing the model's generalization ability while improving the expression of fine-grained defect features. It is suitable for intelligent image analysis of power transmission line inspections in complex high-altitude environments, possessing strong practicality and promotional value.

[0024] The advantages of the present invention, as well as its additional advantages, will be described in detail in the following specific embodiments. Attached Figure Description

[0025] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute a limitation thereof.

[0026] Figure 1 This is a deep learning network structure diagram of the vibration damper detection method driven by prior knowledge and data fusion in Embodiment 1 of the present invention.

[0027] Figure 2 This is a flowchart of the vibration damper detection method based on prior knowledge and data fusion driven by Embodiment 1 of the present invention;

[0028] Figure 3 This is a structural diagram of the skip connection feature extraction network of Embodiment 1 of the present invention;

[0029] Figure 4 This is a structural diagram of the C2f module in Embodiment 1 of the present invention;

[0030] Figure 5 This is a structural diagram of the spatial pyramid pooling SPPF module of Embodiment 1 of the present invention;

[0031] Figure 6 This is a structural diagram of the feature pyramid module of Embodiment 1 of the present invention;

[0032] Figure 7 This is a schematic diagram of the processing procedure of the positioning prior knowledge extraction module in Embodiment 1 of the present invention;

[0033] Figure 8 This is a schematic diagram of the processing procedure of the fusion prior distribution localization module in Embodiment 1 of the present invention;

[0034] Figure 9 This is a schematic diagram of the processing procedure of the co-occurrence matrix guided graph classification module in Embodiment 1 of the present invention;

[0035] Figure 10 This is a dataset from Embodiment 1 of the present invention, showing the results of defect detection of a classic scenario vibration damper. Detailed Implementation

[0036] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0037] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0038] It should be noted that the terminology used herein is for describing particular embodiments only and is not intended to limit the exemplary embodiments of the present invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof. It should be noted that, without conflict, the various embodiments and features within those embodiments can be combined with each other. The embodiments will now be described in detail with reference to the accompanying drawings.

[0039] Example 1

[0040] In one or more of the technical solutions disclosed in the embodiments, such as Figures 1 to 10 As shown, a vibration damper detection method based on prior knowledge and data fusion includes the following steps:

[0041] Step 1: Acquire images of the vibration damper, perform multi-scale feature extraction, and fuse the extracted features to obtain localization features including shallow, medium, and deep features. and classification features ;

[0042] Step 2: For the acquired vibration damper image, the edge feature extraction and probabilistic Hough transform operation are fused to detect the conductor region, and the prior knowledge of the positioning of the vibration damper is extracted to construct a spatial prior distribution heat map of the vibration damper.

[0043] Step 3: Locate the features The features are convolved, pooled, and stitched together with the prior spatial distribution heatmap of the vibration damper. After fusion, feature selective enhancement is performed to obtain the localization prediction features. ;

[0044] Step 4: Obtain prior knowledge of defect classification, calculate co-occurrence probabilities, and construct a defect category co-occurrence matrix;

[0045] Step 5: Classification features The co-occurrence matrix of the defect category is concatenated using a gating mechanism to obtain the adjacency matrix, which is used for classification features. Convolution is performed to obtain the node feature matrix, and graph convolution is used to solve for the category prediction features; the location prediction features and the category prediction features are fused to generate the vibration damper prediction results.

[0046] This implementation method is based on a graph neural network structure guided by multi-source feature fusion and prior knowledge. First, multi-scale feature extraction is performed on the vibration damper image to ensure the capture of spatial and semantic information at different levels. Feature fusion integrates shallow detail texture information, mid-level structural information, and deep semantic information to generate localization features for target localization and classification features for defect discrimination. To enhance the localization capability of the vibration damper in complex image backgrounds, edge feature extraction technology and probabilistic Hough transform are used to jointly identify the guide wire region, thereby obtaining spatial localization priors. This information is expressed in the form of a heatmap. The heatmap and localization features are convolved, pooled, and concatenated to enhance structural perception of the target while preserving spatial information, improving feature discrimination ability, and obtaining more accurate localization prediction features. For defect classification, prior knowledge of defect classification is used to mine the co-occurrence relationships between different defect types, constructing a defect category co-occurrence matrix. This matrix is ​​then fused with the classification features through a gating mechanism to generate an adjacency matrix, guiding the propagation of node information in the graph neural network. This method employs graph convolution operations to represent features of each category and learn classification boundaries, forming category prediction features. These features are then fused with localization prediction features to output the detection and defect identification results of the vibration damper. This approach comprehensively considers the regional characteristics of the vibration damper and the relationships between different defect categories. By using prior knowledge of the vibration damper to assist the deep learning model in learning the defect characteristics of the vibration damper, it significantly improves the accuracy and reliability of vibration damper defect detection.

[0047] This embodiment introduces a multi-scale fusion strategy in the image processing stage, significantly improving the ability to express features at different levels and ensuring precise identification of small target defects. In the localization stage, the introduction of spatial prior heatmaps effectively enhances the model's robustness in complex backgrounds, significantly reducing false positive and false negative rates. By combining probabilistic Hough transform and edge extraction mechanisms, accurate detection of conductor regions is achieved, thereby more precisely constraining the spatial distribution of vibration dampers. The introduction of a defect category co-occurrence matrix and graph convolution mechanism effectively captures the intrinsic connections and distinguishing features between defect categories, improving the accuracy of multi-category defect identification. The overall system combines a data-driven and knowledge-guided dual strategy, enhancing the model's generalization ability while improving the expression of fine-grained defect features. It is suitable for intelligent image analysis of transmission line inspections in complex environments and possesses strong practicality and promotional value.

[0048] In step 1, the vibration damper image is acquired and multi-scale feature extraction is performed. The extracted features are then fused using feature pooling to obtain localization features and classification features, including shallow, medium, and deep features. The method for extracting localization features and classification features includes the following steps.

[0049] Step 11: Perform convolution, skip connections, and pooling operations on the acquired vibration damper images to obtain initial shallow features at different scales. Mid-layer characteristics and deep features ;

[0050] Step 12: Obtain the initial shallow features Mid-layer characteristics and deep features Feature fusion is performed using top-down downsampling and bottom-up upsampling to obtain the final shallow features. Mid-layer characteristics and deep features The obtained features are then spliced ​​and fused to obtain localization features and classification features.

[0051] Specifically, step 1 is achieved through the construction of a skip connection feature extraction network and a feature pyramid module, such as... Figures 3 to 5 As shown, a skip connection feature extraction network is used to implement step 11. It includes multiple feature extraction layers, each of which includes a convolutional layer (Conv) and a C2f module connected in sequence. The last feature extraction layer is connected to a spatial pyramid pooling module (SPPF). The feature extraction layers correspond to different feature depths. In this embodiment, the outputs of the feature extraction layers and the spatial pyramid pooling module (SPPF) are selected as the initial shallow features. Mid-layer characteristics and deep features ;

[0052] In one specific embodiment, the constructed C2f module performs feature extraction to capture multi-layered local detail information in the vibration damper image, such as... Figure 4 As shown, the C2f module includes a first CR module, a splitting module, multiple jump connection modules, a splicing module, and a second CR module connected in sequence.

[0053] The first CR module and the second CR module adopt the same structure, including a convolutional layer Conv and an activation layer ReLU;

[0054] Specifically, the C2f module performs feature extraction, and the corresponding formula is:

[0055] (1);

[0056] (2);

[0057] (3);

[0058] (4);

[0059] (5);

[0060] in, This represents the input of the C2f module. This represents the output of the C2f module. This represents the features of the input deep extraction branches. This represents the multi-level features obtained through stepwise extraction of deep feature extraction blocks. This represents the convolution operation. Represents the ReLU activation function. This represents a combination of convolution operation and ReLU activation function. Indicates a jump connection block x C2f , This represents the input to the jump connection block.

[0061] In one embodiment of this invention, the Spatial Pyramid Pooling Module (SPPF) is used to extract feature information at different scales, such as... Figure 5 As shown, it includes an input convolutional layer Conv, multiple max pooling layers Maxpool, a concatenation layer, and an output convolutional layer Conv.

[0062] Specifically, the formula for feature pooling fusion in the Spatial Pyramid Pooling Module (SPPF) is as follows:

[0063] (6);

[0064] (7);

[0065] in, This represents the input of the SPPF module. This indicates the output of the SPPF module. This represents the convolution operation. This indicates a max pooling operation.

[0066] like Figure 3 The skip connection feature extraction network shown processes the input image data through convolutional layers, a C2f module, and a spatial pyramid pooling (SPPF) module to obtain shallow features. Mid-layer characteristics and deep features The formula is as follows:

[0067] (8);

[0068] (9);

[0069] (10);

[0070] in, This represents the input image data. This represents the convolution operation. This indicates the feature extraction operation of the C2f module. This represents the spatial pyramid pooling operation.

[0071] The feature pyramid module is used to implement step 12, such as... Figure 6 As shown, it includes upsampling units arranged from bottom to top and downsampling units arranged from top to bottom; the upsampling unit includes an upsampling module Upsample, a stitching and fusion module Concat, and a C2f module; the downsampling unit includes a convolutional layer Conv, a stitching and fusion module Concat, and a C2f module connected in sequence.

[0072] like Figure 6 As shown, the final shallow features obtained Mid-layer characteristics and deep features The features are obtained by splicing and merging the individual parts. and classification features ;

[0073] In step 2, a method for constructing a spatial prior distribution heatmap of the vibration damper is used, which integrates edge feature extraction and probabilistic Hough transform operations to detect the conductor region and extract prior knowledge representing the location of the vibration damper. This step can be implemented in the constructed prior knowledge extraction module. The method for constructing the spatial prior distribution heatmap of the vibration damper includes the following steps:

[0074] Step 21: Convert the input vibration damper image to grayscale, apply Gaussian filtering, and obtain the edge feature map of the image using Canny edge extraction. The formula is as follows:

[0075] (11);

[0076] (12);

[0077] (13);

[0078] (14);

[0079] (15);

[0080] (16);

[0081] in, These represent the three color channels of an image. It is a grayscale image. Represents the Gaussian kernel. Represents the standard deviation and controls the width of the Gaussian function. This represents the grayscale image after Gaussian filtering. and These represent the gradients in the horizontal and vertical directions, respectively. and These represent the gradient magnitude and direction, respectively.

[0082] In this embodiment, the RGB color image is converted to a grayscale image to simplify color information and reduce computational complexity. A Gaussian filter with a normalized kernel is used to smooth the image and suppress noise while preserving edge structure. The Gaussian filter mitigates high-frequency noise caused by environmental factors, ensuring that the wire edges remain prominent in subsequent detection.

[0083] Step 22: Identify edges in the edge feature map using an intensity threshold comparison method, separate the region within the edge from the foreground region, and obtain the filtered texture map;

[0084] The metallic surface of the conductor has a distinct intensity transition compared to a natural or cluttered background, allowing texture extraction methods to effectively separate the foreground region using an appropriate intensity threshold.

[0085] Step 221: For the edge feature map, identify strong edges, weak edges and non-edge regions according to the set intensity threshold;

[0086] Specifically, according to the two set thresholds ( , The gradient magnitudes obtained in step S34 are divided into three categories:

[0087] Strong edge: The gradient magnitude is greater than the high threshold;

[0088] Weak edge: The gradient magnitude is between the high and low thresholds.

[0089] Non-edge: The gradient magnitude is less than the low threshold.

[0090] Step 222: Connect weak and strong edges using hysteresis thresholding. Based on the connected edges, distinguish the foreground region and perform filtering to obtain the filtered texture map. The effect is as follows: Figure 7 The image shown in the bottom right corner;

[0091] Specifically, starting from strong edge pixels, weak edge pixels are checked along the gradient direction; if a weak edge pixel is connected to a strong edge pixel, it is retained as an edge; otherwise, it is marked as a non-edge.

[0092] Step 23: Perform probabilistic Hough transform on the filtered texture map to obtain the wire region map;

[0093] The wires exhibit a linear distribution with strong continuity and a slender, straight geometry, often spanning a large portion of the image. Extended linear structures can be detected using geometric transformation techniques.

[0094] In this embodiment, the texture image filtered in step S35 is subjected to probabilistic Hough transform to detect the wire region. The specific process is as follows:

[0095] Step 231: Based on probabilistic Hough transform detection, the filtered texture map is detected to obtain candidate straight lines;

[0096] Step 2311: Convert the edge points in the filtered texture image to polar coordinate space. ;

[0097] Represent the straight line using polar coordinates and discretize the parameter space into a grid, specifying the angle parameters along the line direction. Discretize by step size to obtain An angle represents the shortest distance from the straight line to the origin. Discretize by step size to obtain One distance;

[0098] Step 2312: Create a two-dimensional voting array The initial value is 0;

[0099] Step 2313: For each edge point in the filtered texture image Calculate all its possible The combination involves accumulating votes at the corresponding positions in the two-dimensional voting array to find the local voting peak (i.e., the number of votes exceeds the threshold) as the candidate line.

[0100] Specifically, a point is randomly selected from the set of edge points. Calculate all possible values ​​for that point. Combine and update the accumulator for voting. When a certain When the number of votes for a combination exceeds a threshold, it is considered that there may be a candidate straight line.

[0101] Step 232: Perform connectivity screening on each candidate line obtained, and use it as a valid conductor segment to obtain the detected conductor region map, a new map in a single channel;

[0102] Specifically, the search for continuous edge points along the candidate line direction and the checking of whether the edge points near the candidate line are continuously distributed. The connectivity screening indicators include the total length of continuous points and the break interval at the break.

[0103] Total length of consecutive points The following conditions must be met:

[0104] (17);

[0105] in, and These are the beginning and end points of a connected segment of wire. As a set value, formula (17) means that a complete wire is considered to be a connected pixel in an image when the distance between connected pixels exceeds a certain minimum length.

[0106] If there is a break in the line segment, the break interval Must meet:

[0107] (18);

[0108] in, and For the two ends of the break, the formula (18) indicates that for a conductor segment that is allowed to break, the interval between the break points must not exceed a certain range;

[0109] Step 24: For the obtained traverse region map, calculate the Euclidean distance from each pixel (i,j) to the nearest traverse point as the value of pixel (i,j), and generate a spatial prior distribution heatmap of the vibration damper. The image reflects the possible locations of the vibration dampers;

[0110] For the validated set of continuous points, calculate the distance of each pixel to the nearest straight-line pixel. The calculation formula is:

[0111] (19);

[0112] Where (i,j) represents the coordinates of the conductor point, and (x,y) represents the pixel currently being calculated in the heatmap;

[0113] The spatial distribution of vibration dampers is typically achieved by installing them along the conductor. This is to generate a heatmap of the prior spatial distribution of the vibration dampers. It provides information. By calculating the Euclidean distance from each pixel to the detected wire and mapping these distances to intensity values, it highlights areas near the wire that may be the location of vibration dampers, improving the positioning efficiency in automated inspections.

[0114] In this embodiment, a spatial prior distribution heatmap of the vibration damper is constructed based on the topology of the transmission tower-conductor-vibration damper. This provides crucial prior knowledge for the positioning of vibration dampers. Subsequently, in deep learning models, prior region information is combined with attention-based mechanisms to guide the network to focus on the foreground region, reduce interference from complex backgrounds, and enable rapid positioning of vibration dampers and defect analysis.

[0115] Step 3, achieved by constructing a fusion prior distribution positioning module, utilizes the spatial prior distribution heatmap of the vibration damper obtained in Step 2. Guide the deep learning model to focus on the shallow features obtained in step 1 Mid-layer characteristics and deep features Positioning features after splicing The key areas were identified, and location prediction features were obtained. .

[0116] In step 3, the location features are... The features are convolved, pooled, and stitched together with the prior spatial distribution heatmap of the vibration damper. After fusion, feature-selective enhancement is performed to obtain the enhanced localization prediction features. The method includes the following steps:

[0117] Step 31: Locate the features After convolution, pooling is performed to obtain features after global average pooling and max pooling. and ;

[0118] Step 32: Perform convolution and average pooling operations on the spatial prior distribution heatmap of the vibration damper to obtain the adaptive feature map. Features after global average pooling and max pooling and The features are spliced ​​and merged to obtain the spliced ​​features. ;

[0119] Specifically, the spatial prior distribution heat map of the vibration damper... The feature map is transformed into an adaptive feature map through 3×3 convolution and average pooling. and features after global average pooling and max pooling and The data is then spliced ​​together, and the spatial prior distribution heat map of the vibration damper is then used. By fusing the localization features obtained through network learning, the advantages of prior knowledge and learned features are combined. The above process is as follows: Figure 8 As shown, the formula is as follows:

[0120] (20);

[0121] (twenty one);

[0122] (twenty two);

[0123] (twenty three);

[0124] (twenty four);

[0125] in, Indicates the features after splicing. This represents the convolution operation. This indicates the average pooling operation. This indicates a max pooling operation.

[0126] Step 33: Perform convolution processing on the concatenated features and combine them with the features based on localization. Features after convolution Perform the Hadamard product operation to obtain the localization prediction features after feature selective enhancement. ;

[0127] The Hadamard product, also known as element-wise multiplication, is the new matrix obtained by multiplying corresponding elements of two matrices.

[0128] Specifically, the spliced ​​features are processed using a 7×7 large-scale convolutional kernel. , obtain features It can capture long-range spatial context relationships.

[0129] In this embodiment, structural guidance is provided based on the layout information of the conductors contained in the prior spatial distribution heatmap of the vibration damper, while the image features extracted by the model capture the data-driven pattern. The Sigmoid function is used to... Normalization is performed, and a Hadamard product operation is conducted with the original feature map to achieve feature selective enhancement. Subsequently, a 3×3 convolution is used to further extract localization prediction features. By combining domain constraints with adaptive representation, the accuracy, robustness, and generalization ability of localization under diverse UAV line-following image conditions are improved; the formula is as follows:

[0130] (25);

[0131] (26);

[0132] (27);

[0133] in, This indicates a convolution operation with a 7×7 kernel. This represents the Hadamard product operation. This indicates a convolution operation with a kernel size of 3×3.

[0134] While hard-coding the distribution mask of the vibration dampers directly into the loss provides good interpretability, it suffers from poor robustness and lacks flexibility. Furthermore, the spatial prior distribution heatmap of the vibration dampers... While providing weak supervision for initial localization, it cannot directly capture the distribution of vibration dampers. In the above implementation of this embodiment, cross-regional dependencies are modeled based on a mechanism similar to spatial attention, dynamically weighting features and focusing on regions that may contain vibration dampers. Cross-regional dependencies refer to two geographically distant but physically related regions in the image; in this embodiment, this is the relationship between the guide wire and the vibration damper. This prioritizes relevant spatial context, mitigates interference from complex backgrounds, and aligns with the physical distribution of vibration dampers, thereby improving the accuracy of vibration damper localization.

[0135] Vibration damper defects can be categorized into normal, single-missing, double-missing, bending, and offset, and these categories are inherently interrelated. For example, the evolution from normal to single-missing, and then to double-missing reflects a clear gradient in defect severity. Similarly, bending and offset defects may occur simultaneously due to shared environmental stresses, such as strong winds or structural fatigue. Furthermore, because vibration dampers in a single image are exposed to the same environmental conditions, such as wind, temperature, or structural stress, the defects they exhibit tend to be concentrated in a specific area of ​​the scene.

[0136] Step 4, the method for constructing the defect category co-occurrence matrix, is implemented by constructing a classification prior knowledge extraction module, including the following steps:

[0137] Step 41: Obtain prior knowledge of defect classification, calculate the co-occurrence probability of any two defect categories occurring simultaneously, and construct a conditional probability matrix;

[0138] Optionally, prior knowledge for defect classification can be historical inspection data and manually labeled samples;

[0139] In vibration damper defect detection, the co-occurrence matrix captures the conditional probability of defect categories co-occurring in the same scene, reflecting potential relationships driven by common environmental or structural factors. By statistically analyzing the number of times each defect category co-occurs with other defect categories in the same image, the potential connections between defect categories in the vibration damper data are revealed. Based on the co-occurrence relationship, a conditional probability matrix is ​​obtained, where each element represents the probability of two defect categories co-occurring within the total number of occurrences of a particular defect category. The calculation formula is as follows:

[0140] (28);

[0141] in, Indicates category and categories The number of times they appear together Indicates category Number of times it appears Indicates the total number of categories. The set smoothing factor.

[0142] Step 42: Construct a defect category co-occurrence matrix by treating the defect categories as rows and columns, and using the co-occurrence probabilities in the conditional probability matrix between defect categories as the matrix values. ;like Figure 9 The matrix shown in the upper left corner;

[0143] Complex inter-category and contextual relationships of defects pose a significant challenge to standard convolutional neural networks (CNNs), which primarily focus on local feature extraction and struggle to model complex dependencies due to their limited relational reasoning capabilities. To overcome these challenges, this embodiment employs graph convolutional networks (GCNs) to model the topological and semantic relationships between defect categories. GCNs represent data as graphs, where nodes correspond to defect categories and edges encode their interdependencies. By leveraging node representation learning, GCNs aggregate information from neighboring nodes, enabling the model to capture both local and global relational patterns.

[0144] In step 5, the classification features and the co-occurrence matrix of defect categories are concatenated using a gating mechanism to obtain an adjacency matrix. Convolution is then used to extract the node feature matrix for each defect category. Graph convolution is used to solve for the category prediction features. This is achieved by constructing a co-occurrence matrix to guide the graph classification module, including the following steps:

[0145] Step 51: Classification features Perform convolution, and use the features obtained from the convolution as the activation coefficients;

[0146] (29);

[0147] (30);

[0148] in, This represents the convolution operation. This represents the Sigmoid function. , Here, is a learnable parameter, and gate is the gating coefficient.

[0149] Step 52: Based on the obtained gating coefficients, combine the convolutional classification features with the defect category co-occurrence matrix. The adjacency matrix is ​​obtained by merging. ;

[0150] The model obtains shallow features from step S33. Mid-layer characteristics and deep features Corresponding classification section The learnable weighted adjacency matrix, which automatically infers connection weights between nodes, further enhances the model's ability to capture local and global relationships. It dynamically infers implicit relationships between defect categories, such as the co-occurrence probability of bending and offset defects under specific environmental conditions. Furthermore, it utilizes a dynamically gated and balanced defect category co-occurrence matrix. With classification features To ensure the robustness and context-awareness of defect classification, the fused adjacency matrix... The solution formula is as follows:

[0151] (31);

[0152] This fused adjacency matrix also incorporates data-driven local information. Co-occurrence relationships driven by prior knowledge (defect category co-occurrence matrix) It has a stronger sense of context and improves the correlation between information between categories;

[0153] Step 53: Construct a graph structure node feature matrix, including classification features. Performing a convolution operation yields a node feature matrix, which serves as the node input features for graph convolution. ;

[0154] (32);

[0155] Step 54: Construct graph convolution to propagate contextual features. The feature representation of each node is propagated and aggregated through the adjacency matrix to finally obtain the category prediction features. ;

[0156] Optionally, a two-layer graph convolutional network (GCN) is constructed to propagate contextual features. The graph structure nodes in the first layer aggregate features from direct neighbor nodes; the graph structure nodes in the second layer aggregate information from higher-order neighbors to obtain the final inter-class context-aware features.

[0157] Specifically, a two-layer GCN is used to map features to the category prediction features of each anchor box. In the first layer, each node uses and Features are aggregated from its immediate neighbors to capture local relationships, such as the direct correlation between single and dual defects due to shared severity progression. In the second layer, GCN further aggregates features from the neighbors of neighbors to capture higher-order relationships. This iterative process optimizes the node representation, enabling the model to capture complex patterns. By modeling these interactions, GCN ensures that the classification of each defect category is influenced by its contextual and relational dependencies with other categories, significantly improving classification accuracy and robustness. The formula for GCN is as follows:

[0158] (33);

[0159] in, Represent the ( The node feature matrix of the input layer That is, the original feature matrix; For the ( The learnable weight matrix of the layer. For activation functions;

[0160] This represents an adjacency matrix with added self-loops, i.e.:

[0161] ;

[0162] represent The degree matrix, i.e.:

[0163] ;

[0164] Finally, a prediction map is generated using location prediction features and category prediction features, including the vibration damper anchor frame, the defect category of the vibration damper within the anchor frame, and its probability, such as... Figure 10 As shown in the figure, different colors are used to represent different color categories in this embodiment, and the category probability is marked in the box.

[0165] In this embodiment, the location and category of the vibration damper are predicted to achieve defect detection. This method comprehensively considers the regional characteristics of the vibration damper and the relationship between various defect categories. By using prior knowledge of the vibration damper to assist a deep learning model in learning the defect characteristics of the vibration damper, the accuracy and reliability of vibration damper defect detection are significantly improved.

[0166] Furthermore, steps 1 to 5 above are implemented through a constructed deep learning network, the network structure of which is as follows: Figure 1 As shown, it includes a prior knowledge extraction module, a backbone network and a neck network, and a detection head;

[0167] The prior knowledge extraction module includes a location prior knowledge extraction module and a classification prior knowledge extraction module;

[0168] The backbone network and neck network include a skip connection feature extraction network and a feature pyramid module;

[0169] The detection head includes a fusion prior distribution localization module and a co-occurrence matrix guided graph classification module;

[0170] Furthermore, the method for training the constructed deep learning model includes the following steps:

[0171] Step S1: Conduct inspections of power transmission and distribution lines to collect images of vibration dampers, and use Labelimg to label the location and defect categories of the vibration dampers to construct a vibration damper defect dataset.

[0172] Optionally, drones can be used to inspect power transmission and distribution lines and collect images of vibration dampers.

[0173] Step S2: Divide the anti-vibration hammer defect dataset into a training dataset and a test dataset according to a certain ratio.

[0174] Step S3: Input the anti-vibration hammer defect training dataset into the deep learning model for training. Execute the process from Step 1 to Step 5 to obtain the localization prediction features and category prediction features to generate the anti-vibration hammer prediction results. Through multiple iterations of learning and model parameter optimization, obtain the optimal network model.

[0175] Step S4: Use the vibration damper defect test dataset to test the optimal training model and calculate the model performance evaluation index.

[0176] In step S3, during the training process, the overall loss function is constructed as a weighted sum of classification loss, bounding box loss, and distribution focusing loss;

[0177] Based on the obtained location prediction features and category prediction features Positive samples are matched against labeled images using TaskAligned, and classification loss, bounding box loss, and distribution focusing loss are calculated.

[0178] Optional, classification loss A binary cross-entropy loss with sigmoid activation is applied independently to each category. The classification loss optimizes the model's predictive ability for the target category, ensuring the model can correctly distinguish which defect category the target belongs to. The formula is as follows:

[0179] (34);

[0180] in, The number of positive samples. No. Does the true label of each sample belong to class? , Model predicts the first Each sample belongs to category The probability of.

[0181] Bounding box loss The full intersection-union ratio (CIU) loss between the predicted and target bounding boxes is used to optimize the difference between the model-predicted bounding box coordinates and the ground truth bounding boxes, ensuring more accurate localization of the vibration damper target. The formula is as follows:

[0182] (35);

[0183] (36);

[0184] (37);

[0185] (38);

[0186] in, The positive sample size is represented by IoU, which is the standard crossover ratio. To predict the Euclidean distance between the center point of the bounding box and the center point of the ground truth bounding box, and The center coordinates of the predicted bounding box and the ground truth bounding box. The diagonal length of the minimum closed area covering both frames. These are the weighting coefficients. This is a measure of aspect ratio consistency.

[0187] The distributed focusing loss employs a binary cross-entropy supervised model to predict the discrete values ​​of the location coordinates. By modeling the bounding box coordinates as a discrete probability distribution, the accuracy of the predicted box coordinates is optimized, enabling fine adjustments to the bounding box coordinates. The formula is as follows:

[0188] (39);

[0189] in, The number of positive samples. To represent the coordinates of the bounding box (center point) , ,width ,high ), and These are the two discrete intervals that are closest to the true coordinates. It is an interval The target probability, It is an interval The predicted probability.

[0190] Based on the calculated classification loss, bounding box loss, and distribution focusing loss, the model parameters are continuously adjusted through multiple iterations of training to optimize model performance and ultimately obtain the optimal training parameters.

[0191] In step S4, the vibration damper defect test dataset is used to test the obtained optimal training model. The evaluation metrics include the model's precision, recall, F1 score, mAP@0.5, mAP@[0.5:0.95], number of parameters (M), and inference time to evaluate the network performance.

[0192] Among them, the number of parameters (M) is an indicator of the complexity of the model, representing the total number of all trainable weights (such as convolutional kernels, fully connected matrices, embedding vectors, etc.) in the model, and the unit is usually Million (M).

[0193] Precision (P) measures the proportion of samples that the model predicts as positive, out of all actual positive samples, evaluating the accuracy of the model's predictions. Recall (R) measures the proportion of correctly identified positive samples out of all actual positive samples, evaluating the model's ability to cover defects. The calculation formulas are as follows:

[0194] (40);

[0195] (41);

[0196] in, This represents the number of samples that were correctly predicted as positive. This represents the number of negative samples that were incorrectly predicted as positive. This indicates the number of positive samples that were incorrectly predicted as negative.

[0197] F1 is the harmonic mean of P and R, and is a core comprehensive indicator for evaluating the performance of classification and detection models. The formula is as follows:

[0198] (42);

[0199] Mean accuracy (mAP) is used to evaluate the model's detection performance across all categories, and its calculation formula is as follows:

[0200] (43);

[0201] mAP@0.5 refers to the mAP calculated when the crossover ratio (CUP) threshold is set to 0.5.

[0202] (44);

[0203] mAP@[0.5:0.95] represents the average of all calculated mAP values ​​when the IoU threshold changes from 0.5 to 0.95 in increments of 0.05. It is used to evaluate the detection robustness of the model under different levels of strictness, and the formula is as follows:

[0204] (45);

[0205] To demonstrate the testing results, a simulation experiment was conducted, and the results are as follows: Figure 10 In the inspection image scenario shown, the vibration damper detection method based on prior knowledge and data fusion proposed in this embodiment exhibits excellent detection accuracy and stability. For example... Figure 10 As shown, this method can effectively identify vibration damper components in power lines under different weather conditions, shooting angles, and complex backgrounds, and accurately label their location and category confidence levels. Different colored boxes represent defect categories and locations, and the numerical values ​​within the boxes represent probabilities. For example... Figure 10 (a) Under dim lighting conditions, such as Figure 10 (b) can be accurately identified even under long-distance shooting conditions;

[0206] like Figure 10 In (c) and (d), high-precision identification of vibration damper targets at different scales and degrees of occlusion in the images was achieved, demonstrating the effectiveness of the multi-scale feature extraction and spatial prior fusion strategy.

[0207] like Figure 10 In (e) and (f), under complex background interference, such as greenhouses, farmland, and buildings, the detection box accurately covers the vibration damper area, significantly reducing false positives and false negatives, verifying the auxiliary role of the localization heatmap in spatial constraints. The classification of defect targets is reasonable, and the category boundaries are clearly distinguishable, indicating that the graph convolution module successfully models the semantic co-occurrence relationship between defect types, effectively improving classification accuracy. In multi-target dense scenes, the model still maintains stable detection performance.

[0208] Example 2

[0209] Based on Embodiment 1, this embodiment provides a vibration damper detection system driven by prior knowledge and data fusion, including an image acquisition device and a processor;

[0210] An image acquisition device is used to acquire images of the vibration damper; an image acquisition device based on a drone can be used, employing a camera mounted on the drone for image acquisition.

[0211] The processor is configured to execute the steps in the vibration damper detection method based on prior knowledge and data fusion as described in Example 1.

[0212] Example 3

[0213] Based on Example 1, this example provides a vibration damper detection system driven by prior knowledge and data fusion, including:

[0214] The backbone network and the neck network are configured to acquire images of the vibration damper, perform multi-scale feature extraction, and fuse the extracted features to obtain localization features and classification features including shallow features, medium features and deep features.

[0215] The prior knowledge extraction module is configured to extract prior knowledge representing the location of the vibration damper from the acquired vibration damper image by fusing edge feature extraction and probabilistic Hough transform operation to detect the guide wire region and construct a spatial prior distribution heatmap of the vibration damper.

[0216] The prior distribution localization module is configured to perform convolution, pooling and stitching fusion on the localization features and the spatial prior distribution heatmap of the vibration damper. After fusion, feature selective enhancement is performed to obtain localization prediction features.

[0217] The prior distribution localization module is configured to acquire prior knowledge of defect classification, calculate co-occurrence probabilities, and construct a defect category co-occurrence matrix.

[0218] The co-occurrence matrix-guided graph classification module concatenates the co-occurrence matrices of classification features and defect categories through a gating mechanism to obtain an adjacency matrix. It then performs convolution on the classification features to obtain a node feature matrix and uses graph convolution to solve for the category prediction features. Finally, it fuses the location prediction features and the category prediction features to generate the vibration damper prediction results.

[0219] Furthermore, the co-occurrence matrix guided graph classification module is configured to perform the following steps:

[0220] Classification features Perform convolution, and use the features obtained from the convolution as the activation coefficients;

[0221] Based on the obtained gating coefficients, the co-occurrence matrix of the convolutional classification features and defect categories is... The adjacency matrix is ​​obtained by merging. ;

[0222] Construct a graph structure node feature matrix, and classify the features. Performing a convolution operation yields a node feature matrix, which serves as the node input features for graph convolution. ;

[0223] A graph convolution is constructed to propagate contextual features. The feature representation of each node is propagated and aggregated through the adjacency matrix, ultimately yielding the category prediction features. .

[0224] It should be noted that each module in this embodiment corresponds one-to-one with each step in embodiment 1, and their specific implementation process is the same, so it will not be repeated here.

[0225] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0226] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A method for detecting a vibration damper based on prior knowledge and data fusion driving, characterized in that, The method comprises the following steps: An anti-vibration hammer image is acquired, multi-scale feature extraction is performed, and the extracted features are fused to obtain positioning features and classification features including shallow features, middle features and deep features; For the acquired anti-vibration hammer image, edge feature extraction and probability Hough transform operation are fused to detect the wire region, extract the positioning prior knowledge representing the anti-vibration hammer positioning position region, and construct an anti-vibration hammer spatial prior distribution heat map; The method for constructing the anti-vibration hammer spatial prior distribution heat map comprises the following steps: The input anti-vibration hammer image is subjected to grayscale, Gaussian filtering and edge feature extraction through Canny edge extraction to obtain an edge feature map of the image; The edge feature map is subjected to edge recognition through intensity threshold comparison, the edge region is separated from the foreground region, and a filtered texture map is obtained; For the filtered texture map, a probability Hough transform is performed to obtain a wire region map; For the obtained wire region map, the Euclidean distance of each pixel point (i, j) to the nearest wire point is calculated as the value of the pixel point (i, j), and an anti-vibration hammer spatial prior distribution heat map is generated; The positioning features and the anti-vibration hammer spatial prior distribution heat map are convolved, pooled and spliced to obtain positioning prediction features after selective enhancement of the fused features; Defect classification prior knowledge is acquired to calculate the co-occurrence probability, and a defect category co-occurrence matrix is constructed; The classification features and the defect category co-occurrence matrix are spliced through a gating mechanism to obtain an adjacency matrix, the classification features are convolved to obtain a node feature matrix, and a category prediction feature is solved by graph convolution; The positioning prediction features and the category prediction features are fused to generate an anti-vibration hammer prediction result; The classification features and the defect category co-occurrence matrix are spliced through a gating mechanism to obtain an adjacency matrix, a node feature matrix of each defect category is extracted by convolution, and a category prediction feature is solved by graph convolution. Classification features performing convolution, and performing activation based on the features after the convolution as the gating coefficients; Based on the obtained gating coefficient, the classified feature after convolution and the defect category co-occurrence matrix are fused to obtain an adjacency matrix The fusion is performed to obtain an adjacency matrix ; Construct a graph structure node feature matrix, and classify features Perform a convolution operation to obtain a node feature matrix as the node input feature of graph convolution ; The architecture of GCN is shown in FIG.

1. The graph convolution is used to propagate and aggregate the features of each node through the adjacency matrix, and finally the class prediction features are obtained .

2. The method of claim 1, wherein the method is characterized by, The method for extracting positioning features and classification features comprises the following steps: The acquired anti-vibration hammer image is convolved, jump connected and pooled to obtain initial shallow features, middle features and deep features of different scales; The obtained initial shallow features, middle features and deep features are fused by top-down downsampling and bottom-up upsampling to obtain final shallow features, middle features and deep features; and the obtained features are spliced and fused to obtain positioning features and classification features.

3. The method of claim 1, wherein the method is characterized by, The method for convolving, pooling and splicing the positioning features and the anti-vibration hammer spatial prior distribution heat map, and selectively enhancing the fused features to obtain enhanced positioning prediction features, comprises the following steps: The positioning features are convolved, pooled and processed to obtain features after global average pooling and maximum pooling; The anti-vibration hammer spatial prior distribution heat map is convolved and average-pooled to obtain an adaptive feature mapping, which is spliced and fused with the features after global average pooling and maximum pooling to obtain spliced features; The spliced features are convolved and subjected to Hadamard product operation with the features based on the convolution of the positioning features to obtain positioning prediction features.

4. The method of claim 1, wherein the method is characterized by, The method for constructing the defect category co-occurrence matrix comprises the following steps: The prior knowledge of defect classification is acquired, the co-occurrence probability of simultaneous occurrence of any two defect categories is calculated, and a conditional probability matrix is constructed; The defect category is taken as row and column respectively, and the co-occurrence probability in the conditional probability matrix between defect categories is taken as the value of the matrix to construct a defect category co-occurrence matrix .

5. The method of claim 1, wherein the method is characterized by: The method comprises the following steps: constructing a graph convolution to perform context feature propagation, and propagating and aggregating the feature representation of each node through an adjacency matrix to finally obtain a class prediction feature The method comprises the following steps: constructing a two-layer graph convolution to perform context feature propagation, and aggregating features from direct neighbor nodes for the nodes of the first-layer graph structure. The nodes of the second layer of graph structure aggregate information from high-order neighbors to obtain final inter-class context-aware features.

6. The anti-chatter hammer detection system based on prior knowledge and data fusion driving, characterized in that: The image acquisition device and the processor are included. The image acquisition device is used to acquire the image of the anti-vibration hammer. The processor is configured to perform the steps of the prior knowledge and data fusion driven anti-vibration hammer detection method according to any one of claims 1-5.

7. A system for detecting a vibration damper based on prior knowledge and data fusion driving, characterized in that, It includes: The backbone network and the neck network are configured to acquire the anti-vibration hammer image, perform multi-scale feature extraction, and fuse the extracted features to obtain positioning features and classification features including shallow features, middle features and deep features; The positioning prior knowledge extraction module is configured to fuse edge feature extraction and probabilistic Hough transform operation to detect the wire region for the acquired anti-vibration hammer image, extract positioning prior knowledge representing the positioning position region of the anti-vibration hammer, and construct an anti-vibration hammer spatial prior distribution heat map; The method for constructing the anti-vibration hammer spatial prior distribution heat map includes the following steps: The input anti-vibration hammer image is grayed, Gaussian filtered, and the edge feature map of the image is obtained through Canny edge extraction; The edge feature map is recognized by intensity threshold comparison method, the edge region is separated from the foreground region, and the filtered texture map is obtained; The filtered texture map is subjected to probabilistic Hough transform detection to obtain a wire region map; For the obtained wire region map, the Euclidean distance from each pixel point (i, j) to the nearest wire point is calculated as the value of the pixel point (i, j), and an anti-vibration hammer spatial prior distribution heat map is generated; The fusion prior distribution positioning module is configured to convolve, pool and splice the positioning features and the anti-vibration hammer spatial prior distribution heat map, and perform selective feature enhancement after fusion to obtain positioning prediction features; The fusion prior distribution positioning module is configured to acquire defect classification prior knowledge, calculate co-occurrence probability, and construct a defect category co-occurrence matrix; The co-occurrence matrix guided graph classification module splices the classification features and the defect category co-occurrence matrix through a gating mechanism to obtain an adjacency matrix, convolves the classification features to obtain a node feature matrix, and solves the category prediction features by graph convolution; The positioning prediction features and the category prediction features are fused to generate an anti-vibration hammer prediction result. The method for splicing the classification features and the defect category co-occurrence matrix through a gating mechanism to obtain an adjacency matrix, extracting node feature matrices of each defect category by convolution, and solving category prediction features by graph convolution includes the following steps: On the classification features Convolution is performed, and activation is performed based on the features after convolution as the gating coefficients; Based on the obtained gating coefficient, the classified feature after convolution and the defect category co-occurrence matrix are fused to obtain an adjacency matrix The fusion is performed to obtain an adjacency matrix ; Construct a graph structure node feature matrix, and classify features Perform a convolution operation to obtain a node feature matrix as the node input feature of graph convolution ; The architecture of GCN is shown in FIG.

1. The graph convolution is used to propagate and aggregate the features of each node through the adjacency matrix, and finally the class prediction features are obtained .

Images