Power grid inspection method and device

Abstract

This invention provides a power grid inspection method and apparatus. The method includes: acquiring visible light and infrared images of power grid equipment; extracting features from the visible light and infrared images and performing frequency domain decomposition to obtain multimodal frequency domain features; constructing a two-layer hypergraph node set based on the multimodal frequency domain features, and constructing a hyperedge set based on the two-layer hypergraph node set to obtain a hypergraph; performing feature aggregation based on the hypergraph to obtain hypergraph embedding features, and stripping modal redundancy from the hypergraph embedding features to obtain alignment features; determining the node priority of each scanning node based on the hypergraph and alignment features, and generating a nonlinear scanning sequence based on the node priority; and performing feature scanning and extraction on the alignment features based on the nonlinear scanning sequence to obtain the power grid equipment defect detection results.

Description

Technical Field

[0001] This invention relates to the field of power grid equipment inspection technology, and in particular to a power grid inspection method and apparatus. Background Technology

[0002] As a core infrastructure for energy transmission, the safe and stable operation of the power grid is directly related to social production, daily life, and energy supply security. Power facilities are widely distributed in complex terrains such as mountains and rivers, making traditional manual inspection methods inadequate for the needs of modern power grid operation and maintenance. Unmanned aerial vehicle (UAV) inspection technology, with its advantages of high mobility, wide coverage, and high efficiency, has become an important development direction for the industry. Meanwhile, the iteration of computer vision and deep learning technologies has provided support for intelligent defect detection in UAV inspections. Recognition systems based on target detection algorithms are gradually replacing manual visual judgment and becoming the core means of intelligent inspection.

[0003] Existing technologies typically employ drones equipped with visible light cameras and infrared thermal imaging devices to acquire images of power grid equipment. Features are then extracted from the visible light and infrared images using convolutional neural networks, object detection networks, or other deep learning models. Multimodal information fusion is achieved through feature stitching, result weighting, or bounding box fusion. This type of method can, to a certain extent, utilize appearance information such as texture, edges, and color from visible light images, as well as temperature distribution and thermal anomaly information from infrared images, thereby enabling the identification of defects such as insulator damage, hardware corrosion, and joint overheating.

[0004] However, the above methods still have the following problems: On the one hand, visible light images and infrared images have different imaging mechanisms, and they differ significantly in spatial resolution, texture representation, thermal distribution characteristics, and noise types. Existing methods mostly extract features separately and then directly stitch or weightedly fuse them, making it difficult to effectively characterize the correlation between features of different modes, especially the connection between high-frequency detail features and low-frequency structural features within the same mode, as well as the correspondence between shared defect information between different modes. Therefore, in inspection scenarios with strong light, shadows, rain, fog, vegetation obstruction, or significant equipment thermal noise interference, the fused features are still prone to background redundancy, mode-specific noise, and false anomaly information, resulting in insufficient prominence of defect features such as microcracks, slight corrosion, and latent thermal anomalies.

[0005] On the other hand, existing detection methods based on sequential feature scanning or feature extraction typically scan image regions in a fixed or regular order, such as processing features sequentially according to row and column order, window sliding order, or preset region order. This approach does not fully utilize the correlation strength between multimodal features and struggles to dynamically adjust the scanning order based on defect correlation, local information content, and noise levels. This results in defect regions, edge regions, and background regions being processed approximately uniformly during feature scanning. For power grid inspection images with complex backgrounds, numerous low-information background regions, high-noise regions, and invalid redundant regions consume computational resources and weaken the expressive power of defect-related features during feature propagation, thus affecting the accuracy and robustness of defect detection in complex inspection scenarios. Summary of the Invention

[0006] This invention provides a power grid inspection method and apparatus to address the shortcomings of existing technologies that only perform simple feature fusion or fixed-sequence scanning on visible light and infrared images, which makes it difficult to effectively determine the correlation between multimodal features, resulting in insufficient detection accuracy and robustness for minute and weakly significant defects under complex backgrounds and noise interference.

[0007] This invention provides a power grid inspection method, comprising: acquiring visible light images and infrared images of power grid equipment; extracting features from the visible light images and performing frequency domain decomposition on the infrared images to obtain multimodal frequency domain features; constructing a two-layer hypergraph node set based on the multimodal frequency domain features, and constructing a hyperedge set based on the two-layer hypergraph node set to obtain a hypergraph; performing feature aggregation based on the hypergraph to obtain hypergraph embedding features, and performing modal redundancy stripping on the hypergraph embedding features to obtain alignment features; determining the node priority of each scanning node based on the hypergraph and the alignment features, and generating a nonlinear scanning sequence based on the node priority; and performing feature scanning and extraction on the alignment features based on the nonlinear scanning sequence to obtain power grid equipment defect detection results.

[0008] According to the power grid inspection method provided by the present invention, the multimodal frequency domain features include a single-mode high-frequency feature vector and a single-mode low-frequency feature vector corresponding to the visible light image, and a single-mode high-frequency feature vector and a single-mode low-frequency feature vector corresponding to the infrared image; the step of extracting features from the visible light image and the infrared image and performing frequency domain decomposition to obtain multimodal frequency domain features includes: performing enhanced wavelet transform on the visible light image and the infrared image respectively to obtain their respective single-mode high-frequency feature vectors; and performing fast Fourier transform on the visible light image and the infrared image respectively to obtain their respective single-mode low-frequency feature vectors.

[0009] According to the power grid inspection method provided by the present invention, the step of constructing a two-layer hypergraph node set based on the multimodal frequency domain features includes: constructing a single-mode frequency domain feature node set based on the single-mode high-frequency feature vector and single-mode low-frequency feature vector corresponding to the visible light image and the infrared image; calculating the cosine similarity of each single-mode high-frequency feature vector and single-mode low-frequency feature vector to obtain inter-modal attention weights; and concatenating the single-mode frequency domain feature nodes corresponding to different modes based on the inter-modal attention weights to construct a cross-modal shared feature node set.

[0010] According to the power grid inspection method provided by the present invention, the construction of the hyperedge set includes: determining the intra-modal association weight between single-modal high-frequency feature nodes and single-modal low-frequency feature nodes within the same modality based on information entropy, and constructing an intra-modal hyperedge based on the intra-modal association weight; determining cross-modal association weight based on the semantic relevance and spatial similarity between multiple cross-modal shared feature nodes, and constructing a cross-modal hyperedge based on the cross-modal association weight.

[0011] According to the power grid inspection method provided by the present invention, the step of performing feature aggregation based on the two-layer hypergraph node set and the hyperedge set to obtain hypergraph embedding features, and performing modal redundancy stripping on the hypergraph embedding features to obtain alignment features, includes: performing feature aggregation on the hypergraph through a hypergraph convolutional network to obtain hypergraph embedding features; performing factor unwrapping on the hypergraph embedding features to obtain cross-modal shared factors and modality-specific redundancy factors; and performing reconstruction decoding based on the cross-modal shared factors and the modality-specific redundancy factors to obtain the alignment features.

[0012] According to the power grid inspection method provided by the present invention, the step of determining the node priority of each scanning node based on the hyperedge set and the alignment feature, and generating a nonlinear scanning sequence based on the node priority, includes: determining the information weight corresponding to the alignment feature, and pruning the alignment feature based on the information weight to obtain a preliminary feature map; filtering the preliminary feature map to obtain an effective feature map; dividing the effective feature map into multiple scanning blocks, taking each scanning block as a scanning node, and determining the node features of the scanning node; determining the scanning priority of each scanning node according to the node information weight, hyperedge weight, and average noise weight of the scanning block corresponding to each scanning node; taking the current scanning node, the set of used scanning hyperedges, and the noise weight of the current scanning node as the path state, and determining the jump action space according to the adjacent scanning nodes reachable by the current scanning node through unused scanning hyperedges; and nonlinearly sorting the node features of each scanning node based on the path state, the jump action space, and the node priority of each scanning node to generate the nonlinear scanning sequence.

[0013] According to the power grid inspection method provided by the present invention, the step of performing feature scanning and extraction on the alignment features based on the nonlinear scanning sequence to obtain the power grid equipment defect detection result includes: performing feature scanning and extraction on the node features of each scanning node according to the nonlinear scanning sequence to obtain defect identification features; and determining the power grid equipment defect detection result based on the defect identification features.

[0014] The present invention also provides a power grid inspection device, comprising: an acquisition module for acquiring visible light images and infrared images of power grid equipment; a frequency domain decomposition module for extracting features from the visible light images and the infrared images and performing frequency domain decomposition to obtain multimodal frequency domain features; a hypergraph construction module for constructing a two-layer hypergraph node set based on the multimodal frequency domain features, and constructing a hyperedge set based on the two-layer hypergraph node set to obtain a hypergraph; a redundancy stripping module for performing feature aggregation based on the hypergraph to obtain hypergraph embedding features, and performing modal redundancy stripping on the hypergraph embedding features to obtain alignment features; a sequence determination module for determining the node priority of each scanning node based on the hypergraph and the alignment features, and generating a nonlinear scanning sequence based on the node priority; and a scanning module for performing feature scanning and extraction on the alignment features based on the nonlinear scanning sequence to obtain power grid equipment defect detection results.

[0015] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the power grid inspection method as described above.

[0016] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the power grid inspection method as described above.

[0017] Compared with the prior art, the present invention has the following beneficial effects: The power grid inspection method and apparatus provided by this invention reduces noise and modal differences in visible light and infrared images by utilizing frequency domain decomposition. Then, it establishes a correlation structure between multimodal frequency domain features through a two-layer hypergraph node set and hyperedge set, and performs feature aggregation and modal redundancy stripping based on this hypergraph to obtain aligned features more suitable for defect detection. Furthermore, it determines the priority of scanning nodes and generates a nonlinear scanning sequence based on the hypergraph and aligned features, enabling subsequent feature scans to prioritize defect-related regions rather than uniformly processing all regions in a fixed order. This reduces the impact of background redundancy and noise interference on the detection results, improving the detection accuracy and robustness of minor defects, weakly significant defects, and thermal anomalies in power grid equipment under complex inspection scenarios. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0019] Figure 1 This is a flowchart of the power grid inspection method of the present invention; Figure 2 This invention illustrates a flowchart of constructing a two-layer hypergraph node set based on multimodal frequency domain features, as an example of the present invention. Figure 3 The flowchart illustrates an example of the present invention of feature aggregation based on a two-layer hypergraph node set and hyperedge set to obtain hypergraph embedding features, and modal redundancy stripping of the hypergraph embedding features to obtain aligned features; Figure 4 The flowchart illustrates an example of the present invention of determining the node priority of each scan node based on the hyperedge set and alignment features, and generating a nonlinear scan sequence based on the node priority; Figure 5 A structural block diagram of an example power grid inspection device of the present invention is shown; Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0021] Figure 1 This is a flowchart of the power grid inspection method of the present invention.

[0022] like Figure 1 As shown, the method includes operations S110~S160.

[0023] During operation S110, visible light and infrared images of the power grid equipment are acquired.

[0024] According to embodiments of the present invention, visible light images and infrared images can be acquired by an image acquisition device mounted on power grid inspection equipment such as drones.

[0025] In operation S120, features are extracted from the visible light image and the infrared image respectively, and frequency domain decomposition is performed to obtain multimodal frequency domain features.

[0026] According to embodiments of the present invention, visible light images can be used to reflect information such as the appearance, surface texture, edge contour, color changes, and structural damage of power grid equipment, while infrared images can be used to reflect information such as the temperature distribution, localized heating, thermal anomaly diffusion areas, and latent fault signs of power grid equipment.

[0027] Therefore, by extracting features from visible light images and infrared images respectively, image features that can characterize the state of power grid equipment under different modes can be obtained for subsequent defect identification.

[0028] According to embodiments of the present invention, convolutional neural networks, visual feature extraction networks, attention networks, feature pyramid networks, or other image feature extraction models can be used to extract features from visible light images and infrared images respectively, thereby obtaining visible light image features and infrared image features. Through the above-mentioned image features, the information mixing problem caused by simply stitching visible light images and infrared images together can be avoided, which is beneficial to retaining the effective information of different modalities.

[0029] According to embodiments of the present invention, visible light image features and infrared image features can be decomposed in the frequency domain to divide the image features of each modality into different frequency components. For example, high-frequency components can typically be used to characterize detailed information such as edges, cracks, damage boundaries, fine textures, local abrupt changes, and noise disturbances of power grid equipment; low-frequency components can typically be used to characterize global information such as the main structure, overall outline, temperature distribution trend, and large-scale background changes of equipment such as towers, insulator strings, conductors, and fittings. By performing frequency domain analysis on images of each modality, detailed information, structural information, and thermal anomaly information that contribute to defect identification in visible light and infrared images can be expressed separately, and the impact of illumination changes, UAV jitter, thermal noise, background interference, and modal differences on subsequent fusion processing can be reduced.

[0030] Through the frequency domain decomposition process described above, the frequency domain characteristics of the visible light mode at different frequency levels and the frequency domain characteristics of the infrared mode at different frequency levels are combined to form a feature set, thereby obtaining multimodal frequency domain characteristics.

[0031] Before performing cross-modal fusion, feature extraction and frequency domain decomposition are performed on the visible light and infrared images respectively. This allows the effective defect information in different modes to be expressed in a clearer frequency hierarchy. On the one hand, this helps to preserve fine-grained defect information such as cracks, broken edges, and rust textures; on the other hand, it helps to preserve global information such as the overall structure of the equipment and the distribution of thermal anomalies, thus providing more reliable input features for subsequent hypergraph construction, feature aggregation, modal redundancy stripping, and nonlinear scanning.

[0032] In operation S130, a two-layer hypergraph node set is constructed based on multimodal frequency domain features, and a hyperedge set is constructed based on the two-layer hypergraph node set to obtain the hypergraph.

[0033] According to embodiments of the present invention, the two-layer hypergraph node set may include a single-modal frequency domain feature node set and a cross-modal shared feature node set. The single-modal frequency domain feature node set is mainly used to preserve the frequency domain features of each individual mode; the cross-modal shared feature node set is mainly used to express the common and complementary features between different modes.

[0034] For example, the frequency domain features corresponding to a visible light image can be converted into visible light frequency domain feature nodes, and the frequency domain features corresponding to an infrared image can be converted into infrared frequency domain feature nodes. Visible light frequency domain feature nodes can be used to characterize the edge contours, surface textures, structural morphology, and appearance defects of power grid equipment; infrared frequency domain feature nodes can be used to characterize the temperature distribution, localized heating areas, and thermal anomaly changes of power grid equipment. Thus, the single-mode frequency domain feature node set can retain defect-related information in different modes, avoiding the weakening of detail information caused by directly stitching visible light and infrared images during the initial fusion stage.

[0035] Furthermore, visible light frequency domain feature nodes and infrared frequency domain feature nodes can be fused based on the similarity, complementarity, or attention weights between different modal frequency domain features to construct cross-modal shared feature nodes. Cross-modal shared feature nodes allow images from different modalities to jointly point to a specific structural region or suspected defect area within the power grid equipment. For example, when an edge damage feature exists at a certain location in a visible light image, and a temperature anomaly feature exists at the corresponding location in an infrared image, the cross-modal related information at that location can be expressed as a cross-modal shared feature node.

[0036] For defect detection in power grid equipment, the determination of a defect region is often not solely determined by image features of a single modality, but may simultaneously be related to features such as visible light edge features, visible light texture features, infrared thermal anomaly features, and the location of the equipment structure. Therefore, hyperedges can be used to characterize the higher-order correlations between different feature nodes to more completely express the joint relationships between multiple modal features.

[0037] According to embodiments of the present invention, a hypergraph can be constructed based on a set of hyperedges, a set of two-layer hypergraph nodes, and the hyperedge weights corresponding to the hyperedge set. Since defects in power grid equipment typically involve multiple types of information, such as equipment structure, edge texture, temperature anomalies, and spatial location, using a hypergraph can simultaneously associate multiple related feature nodes through a single hyperedge. This is suitable for describing the synergistic relationships between multimodal defect features, thus providing a basis for subsequent identification of defects in power grid equipment.

[0038] In operation S140, feature aggregation is performed based on the hypergraph to obtain hypergraph embedding features, and modal redundancy is stripped from the hypergraph embedding features to obtain aligned features.

[0039] According to embodiments of the present invention, due to the different imaging mechanisms of visible light images and infrared images, mode-specific information may still exist after feature aggregation. For example, visible light images may contain illumination variations, shadows, background textures, and artifacts caused by drone jitter; infrared images may contain thermal noise, temperature drift, imaging resolution differences, and non-faulty hotspots. Although the above information is a characteristic of a certain mode, it is not necessarily related to defects in power grid equipment. If it is directly detected without processing, it may lead to false detections or missed detections. Therefore, mode redundancy stripping can be performed on the hypergraph embedding features to weaken mode-specific interference unrelated to defect identification. The resulting aligned features provide corresponding support for subsequently determining the priority of nonlinear scanning sequences.

[0040] In operation S150, based on the hypergraph and alignment features, the node priority of each scan node is determined, and a nonlinear scan sequence is generated based on the node priority.

[0041] According to embodiments of the present invention, the feature map corresponding to the alignment feature can be divided into multiple feature blocks, and each feature block can be used as a scanning node. Then, the node priority of each scanning node is determined by combining the hyperedge association relationship with each feature block in the hypermap, as well as the information content, defect response intensity, and noise level of each feature block itself. A nonlinear scanning sequence can be generated according to the node priority and the association relationship between scanning nodes. For example, scanning nodes corresponding to defect regions, equipment edges, abnormal temperature distributions, or highly correlated hyperedges can be given higher priority, while scanning nodes corresponding to background regions, low-information regions, or regions with strong noise can be given lower priority. In this way, defect-related regions are processed first in subsequent feature scanning processes, followed by edge transition regions and background regions, thereby improving feature extraction efficiency and defect detection accuracy.

[0042] In operation S160, based on the nonlinear scanning sequence, the aligned features are scanned and extracted to obtain the defect detection results of the power grid equipment.

[0043] According to an embodiment of the present invention, the scanning nodes in the aligned features can be rearranged according to a nonlinear scanning sequence to obtain serialized features, and then feature scanning can be performed on the serialized features. Since the nonlinear scanning sequence prioritizes the arrangement of defect-related regions, during the feature scanning process, defect-related features such as equipment cracks, damage, corrosion, foreign objects, and localized heating can be extracted first. At the same time, combined with the information of the subsequently arranged edge regions and background regions, the defect location, defect category, and defect confidence level can be identified, and finally the defect detection results of the power grid equipment can be obtained.

[0044] Through the above-described configuration, embodiments of the present invention first perform feature extraction and frequency domain decomposition on visible light and infrared images to preserve multimodal information such as equipment appearance details, structural contours, temperature distribution, and thermal anomalies. Then, a hypergraph is constructed based on the multimodal frequency domain features to express higher-order correlations between different frequency domain features within the same mode and between different modal features. Subsequently, through hypergraph feature aggregation and modal redundancy stripping, aligned features with less noise interference and aligned modal information are obtained. Finally, a nonlinear scanning sequence is generated based on the hypergraph and aligned features, allowing subsequent feature extraction processes to prioritize defect-related regions. This improves the fusion effect of visible light and infrared images, reduces the impact of background noise and modal redundancy on detection results, and enhances the accuracy, robustness, and processing efficiency of power grid inspection in complex inspection scenarios.

[0045] In one illustrative embodiment, the multimodal frequency domain features include a single-mode high-frequency feature vector and a single-mode low-frequency feature vector corresponding to a visible light image, and a single-mode high-frequency feature vector and a single-mode low-frequency feature vector corresponding to an infrared image.

[0046] Operation S120 includes: Enhanced wavelet transforms were performed on the visible light image and the infrared image respectively to obtain their corresponding single-mode high-frequency feature vectors.

[0047] According to an embodiment of the present invention, the input single-modal image can first be decomposed into wavelet decomposition, dividing the single-modal image into multiple high-frequency sub-bands along the horizontal, vertical, and diagonal directions. These multiple high-frequency sub-bands are then aggregated to obtain initial high-frequency features. The single-modal image can be a visible light image or an infrared image. Specifically, it can be represented as follows:

[0048] in, , , They represent the first High-frequency subbands in the horizontal, vertical, and diagonal directions obtained by layer wavelet decomposition. Indicates the wavelet decomposition level. This represents the initial high-frequency features after aggregation.

[0049] Furthermore, pixel-level convolution processing is performed on the initial high-frequency features, and global average pooling and global max pooling are combined to extract global high-frequency response information, resulting in high-frequency enhancement weights. Specifically, this can be represented as:

[0050] in, Represents pixel-level convolution. Indicates global average pooling. Indicates global max pooling. This represents the Sigmoid activation function. This indicates high-frequency enhanced weights.

[0051] Furthermore, high-frequency enhancement weights can be convolutionally fused to obtain high-frequency attention-guided features:

[0052] in, Indicates the kernel size as Convolution operation, Indicates the number of channels. This represents the high-frequency attention-guided features after fusion.

[0053] Subsequently, attention enhancement is applied to the initial high-frequency features based on high-frequency attention-guided features, and optimized high-frequency features are generated through residual connections:

[0054] in, Indicates high-frequency attention-guided features For initial high-frequency features Perform weighted enhancement, This represents the optimized high-frequency features.

[0055] Finally, the optimized high-frequency features are subjected to high-frequency filtering to suppress high-frequency noise and retain effective high-frequency information, resulting in a single-mode high-frequency feature vector. Specifically, this can be expressed as:

[0056] in, This indicates the optimized high-frequency features at the location eigenvalues ​​at that location This indicates the filtering threshold corresponding to that position. This represents the single-mode high-frequency characteristics after filtering. This filtering process removes invalid high-frequency disturbances caused by drone jitter, image noise, thermal imaging noise, etc., while retaining effective high-frequency components related to defects in power grid equipment.

[0057] For visible light images and infrared images, the above processing steps can be performed separately to obtain the single-mode high-frequency feature vectors corresponding to the visible light images and the infrared images, respectively. The single-mode high-frequency feature vectors corresponding to the visible light images are mainly used to characterize appearance details such as device edges, surface textures, cracks, damage, and corrosion; the single-mode high-frequency feature vectors corresponding to the infrared images are mainly used to characterize thermal anomaly details such as local temperature abrupt changes, thermal anomaly boundaries, and the edges of heating areas.

[0058] In addition, fast Fourier transforms can be performed on visible light images and infrared images respectively to obtain their corresponding single-mode low-frequency feature vectors.

[0059] Specifically, a Fast Fourier Transform is performed on the input single-modal image to transform it from the spatial domain to the frequency domain, thus obtaining frequency domain features:

[0060] in, This indicates the location of the input single-modal image. Pixel value or feature value at that location a and b These represent the row index and column index in the spatial field, respectively. H and W These represent the height and width of the image, respectively. Represents frequency domain characteristics, u and v These represent the vertical and horizontal frequency indices in the frequency domain, respectively. Subsequently, the frequency domain features are decomposed into real and imaginary features, and convolutional processing is performed on each to correct global offsets and artifacts in the low-frequency features.

[0061] in, and Representing frequency domain features respectively The real and imaginary features; This represents a convolution operation used to process real part features. This indicates a convolution operation used to process the imaginary part of the feature; This represents the real part output feature obtained after convolution processing. This represents the imaginary part output feature obtained after convolution processing; This represents the optimized frequency domain feature obtained by combining the real and imaginary output features.

[0062] Furthermore, a Gaussian low-pass filter function is used to perform low-pass filtering on the optimized frequency domain features:

[0063] in, This represents the Gaussian low-pass filter function. The scaling parameter represents the Gaussian low-pass filter function. This represents the frequency domain characteristics after Gaussian low-pass filtering.

[0064] Finally, the filtered frequency domain features are inversely transformed and vectorized to obtain the single-mode low-frequency feature vector. :

[0065] Therefore, it is possible to preserve the main structure, overall outline, spatial layout, temperature distribution trend and large-scale thermal anomaly information of the device in visible light or infrared images, and suppress the influence of local noise, imaging drift and irrelevant details on the expression of low-frequency features.

[0066] Figure 2 The flowchart illustrates an example of the present invention of constructing a two-layer hypergraph node set based on multimodal frequency domain features.

[0067] According to an embodiment of the present invention, the dual-layer hypergraph node set may include a single-mode frequency domain feature node set and a cross-mode shared feature node set.

[0068] like Figure 2 As shown, operation S130 includes operations S210~S230.

[0069] In operation S210, a set of single-mode frequency domain feature nodes is constructed based on the single-mode high-frequency feature vector and single-mode low-frequency feature vector corresponding to the visible light image and infrared image.

[0070] According to an embodiment of the present invention, each single-mode high-frequency feature vector and single-mode low-frequency feature vector obtained after frequency domain decomposition can be used as a single-mode frequency domain feature node to form a single-mode frequency domain feature node set.

[0071] For example, the high-frequency feature vectors and low-frequency feature vectors of each single mode can be expanded into vectors and represented as vectors of a uniform dimension, as follows:

[0072] in, Represents the set of feature nodes in the single-mode frequency domain. Indicates the first A single-mode frequency domain feature node.

[0073] In operation S220, cosine similarity is calculated for each single-modal high-frequency feature vector and single-modal low-frequency feature vector to obtain inter-modal attention weights.

[0074] According to an embodiment of the present invention, the high-frequency feature vector and low-frequency feature vector of a single mode corresponding to the same mode can be concatenated by channels to obtain the frequency domain combined feature corresponding to that mode; then, cosine similarity is calculated for the frequency domain combined features corresponding to different modes to determine the degree of correlation between different modes, and inter-modal attention weights are generated based on the degree of correlation. Specifically, this can be expressed as:

[0075] in, and They represent the first The single-mode high-frequency feature vector and single-mode low-frequency feature vector corresponding to each mode. and They represent the first The single-mode high-frequency feature vector and single-mode low-frequency feature vector corresponding to each mode. This indicates concatenation along the channel dimension. This indicates the calculation of cosine similarity. Indicates the first The first mode relative to the first Intermodal attention weights for each modality.

[0076] In operation S230, based on intermodal attention weights, the channel dimension of the single-modal frequency domain feature nodes corresponding to different modalities is spliced ​​to construct a cross-modal shared feature node set.

[0077] According to embodiments of the present invention, frequency domain combination features corresponding to different modalities can be weighted and fused based on inter-modal attention weights, and the fused features can be used as cross-modal shared feature nodes. Specifically, this can be expressed as follows:

[0078] in, Represents the intermodal attention weights. Indicates the first The frequency domain combination features corresponding to each mode. Therefore, the set of cross-modal shared feature nodes can be obtained:

[0079] in, Represents a set of feature nodes shared across modalities. Indicates the first Cross-modal shared feature nodes. Through the above method, based on the similarity and complementarity between different modalities, device structure information, defect region information, or abnormal response information jointly pointed to by visible light images and infrared images can be extracted, thus providing a cross-modal node foundation for subsequent hyperedge set construction and hypergraph feature aggregation.

[0080] In one illustrative embodiment, the process of constructing the hyperedge set includes: Based on the information entropy, determine the intramodal association weights between single-modal high-frequency feature nodes and single-modal low-frequency feature nodes within the same modality, and construct intramodal hyperedges based on the intramodal association weights.

[0081] According to an embodiment of the present invention, for the same mode, the information entropy of the single-mode high-frequency feature nodes and the single-mode low-frequency feature nodes corresponding to that mode can be calculated, and the intra-mode correlation weight can be determined based on their information entropy. Specifically, this can be expressed as:

[0082] in, Indicates the first The single-mode high-frequency feature nodes corresponding to the various modes Indicates the first The single-mode low-frequency feature nodes corresponding to the various modes The information entropy function is represented by... Indicates the first The intramodal association weights correspond to each modality. The larger the intramodal association weight, the stronger the information correlation between high-frequency features and low-frequency features in that modality.

[0083] Furthermore, based on intra-modal correlation weights, high-frequency feature nodes and low-frequency feature nodes of the same modality can be connected to form intra-modal hyperedges, thereby constructing a set of intra-modal hyperedges:

[0084] in, Indicates the first Each modal has an intramodal hyperedge.

[0085] Based on the semantic relevance and spatial similarity among multiple cross-modal shared feature nodes, cross-modal association weights are determined, and cross-modal hyperedges are constructed based on these weights.

[0086] According to embodiments of the present invention, semantic relevance and spatial similarity among multiple cross-modal shared feature nodes can be calculated, and cross-modal association weights can be determined based on the calculation results. Specifically, this can be expressed as follows:

[0087] in, and This indicates that any two cross-modal shared feature nodes, The semantic relevance score between the two is represented by the score. This indicates the spatial or structural similarity between the two. Indicates the first Cross-modal association weights. The larger the cross-modal association weight, the stronger the semantic consistency and spatial correspondence between the corresponding cross-modal shared feature nodes.

[0088] Furthermore, based on cross-modal association weights, multiple cross-modal shared feature nodes with association relationships can be connected into cross-modal hyperedges, thereby constructing a set of cross-modal hyperedges:

[0089] in, Indicates the first A cross-modal hyperedge.

[0090] This method can characterize the cross-modal correlation between different modal areas that are commonly pointed to by the device structure, suspected defect areas, or abnormal response areas.

[0091] Finally, the intramodal hyperedge set and the cross-modal hyperedge set can be combined to obtain the hyperedge set:

[0092] By combining the set of nodes in the two-layer hypergraph with the corresponding hyperedge weights, the hypergraph is obtained. G :

[0093] in, V Represents the set of nodes in a two-level hypergraph. E Denotes the set of superedges. W Represents the set of superedge weights. .

[0094] Figure 3 The flowchart illustrates an example of the present invention of feature aggregation based on a two-layer hypergraph node set and hyperedge set to obtain hypergraph embedding features, and modal redundancy stripping of the hypergraph embedding features to obtain aligned features.

[0095] like Figure 3 As shown, operation S140 includes operations S310 to S330.

[0096] In operation S310, the hypergraph features are aggregated through a hypergraph convolutional network to obtain hypergraph embedding features.

[0097] According to embodiments of the present invention, hypergraph feature encoding can be performed. Hypergraph features are aggregated using a hypergraph convolutional network (HGN) to obtain hypergraph embedding features, as detailed below:

[0098] Where Z represents the hypergraph embedding feature. This represents a hypergraph convolutional network. This represents the normalized adjacency matrix of the hypergraph. This represents the learnable parameters of the hypergraph convolutional network. This represents the activation function.

[0099] In operation S320, factor unwrapping is performed on the hypergraph embedding features to obtain cross-modal shared factors and modality-specific redundancy factors.

[0100] According to embodiments of the present invention, the hypergraph embedding feature Z can be factor-unwrapped and decomposed into cross-modal shared factors. and mode-specific redundancy factor Among them, cross-modal sharing factor Information used to characterize the equipment structure, defect shape, defect location, or abnormal region commonly indicated by different modes; mode-specific redundancy factor. This is used to characterize redundant information such as device noise, imaging artifacts, illumination interference, or thermal noise introduced by a single mode. Specifically, it can be achieved through the following loss constraint:

[0101] in, Indicates the total loss. Denotes KL divergence, This represents the untangling loss under the constraints of the hypergraph G. and Let represent the probability distributions of the modality-specific redundancy factor and the cross-modality shared factor, respectively. Indicates the reconstruction loss. This represents the adaptive balance coefficient.

[0102] In operation S330, reconstruction decoding is performed based on cross-modal shared factors and modality-specific redundancy factors to obtain aligned features.

[0103] According to embodiments of the present invention, cross-modal sharing factors can be... and mode-specific redundancy factor The input reconstruction decoder retains shared information related to power grid equipment defects across different modes through reconstruction constraints, while removing mode-specific redundant information that is weakly correlated with defect identification, thus obtaining alignment features. Specifically, this can be represented as:

[0104] in, Indicates alignment features, This indicates the reconstruction decoder.

[0105] Through the above operations, the alignment features can retain the device structure information, defect edge information, and abnormal area information that are pointed to by both the visible light image and the infrared image. At the same time, the influence of noise, artifacts, and background redundancy information in a single mode is reduced, thereby providing input for subsequent nonlinear scanning and defect detection.

[0106] Figure 4 The flowchart illustrates an example of the present invention of determining the node priority of each scan node based on the hyperedge set and alignment features, and generating a nonlinear scan sequence based on the node priority.

[0107] like Figure 4 As shown, operation S150 includes operations S410~S460.

[0108] In operation S410, the information weights corresponding to the alignment features are determined, and the alignment features are cropped based on the information weights to obtain a preliminary feature map.

[0109] According to embodiments of the present invention, the information richness, hypergraph association strength, and noise distribution at each position in the alignment feature can be comprehensively evaluated to obtain the regional information weight. Specifically, this can be expressed as:

[0110] in, Indicates position Corresponding regional information weights This represents the normalized local information entropy. This represents the normalized hypergraph association weights. Indicates the noise suppression weight. , , This represents the corresponding weighting coefficient; Indicates position The gray entropy of the neighborhood, This represents the hypergraph association weight corresponding to this location, which is determined based on the weights of the hyperedges connected to the feature nodes corresponding to this location's feature region. This represents the local noise variance corresponding to that location. , , , These represent the mean and standard deviation of the corresponding indicators, respectively. This represents the adjustment coefficient.

[0111] Subsequently, an effective mask can be determined based on the region information weights, and the alignment features can be cropped based on the effective mask. Specifically, an adaptive threshold can be set:

[0112] And determine the effective mask based on the adaptive threshold:

[0113] in, Indicates an adaptive threshold. This represents the threshold adjustment coefficient. Indicates position The corresponding effective mask value. Based on this effective mask, regions with high information weight can be retained, while low-light backgrounds, low-information-redundancy regions, or strong noise regions can be removed, thus obtaining a preliminary feature map.

[0114] In operation S420, the preliminary feature map is filtered to obtain an effective feature map.

[0115] According to an embodiment of the present invention, preliminary denoising processing can be performed on the alignment features based on the effective mask. For positions with an effective mask value of 1, the original alignment features are retained; for positions with an effective mask value of 0, the median filtering result is used for replacement. Specifically, this can be expressed as follows:

[0116] in, This indicates the position after initial denoising based on the effective mask. eigenvalues ​​at that location Indicates position Alignment features within the neighborhood The filtering result obtained by median filtering.

[0117] Subsequently, the features after initial denoising can be cropped based on an effective mask to obtain a cropped preliminary feature map. :

[0118] Furthermore, adaptive soft thresholding filtering can be applied to the preliminary feature map to suppress residual weak noise and obtain an effective feature map. :

[0119] in, Indicates position The corresponding adaptive soft threshold can be determined based on local noise estimation.

[0120] in, Indicates position The standard deviation of the noise estimates within the neighborhood. Therefore, while preserving the effective region related to the defect, background redundancy and local noise can be further suppressed to obtain an effective feature map.

[0121] In operation S430, the effective feature map is divided into multiple scan blocks, each scan block is used as a scan node, and the node features of the scan node are determined.

[0122] According to embodiments of the present invention, the effective feature map can be divided into multiple non-overlapping or partially overlapping scan blocks according to a preset size or a preset number. Each scan block corresponds to a local region in the effective feature map, and each scan block is used as a scan node. Specifically, if the effective feature map is divided into... By scanning blocks, we can obtain the set of scanned nodes. ,in, Indicates the first The scan block, that is, the th scan block One scanning node.

[0123] Furthermore, to enhance the discriminative power and noise resistance of node features, feature aggregation can be performed on each scan block to determine the node features of the corresponding scan node. For example, average pooling and max pooling can be performed on the scan block separately, and the results can be concatenated and then processed by convolution to obtain the node features of the scan node.

[0124] in, Indicates the first Node characteristics of each scanned node, Indicates the first The average response features are obtained by average pooling of each scan block. Indicates the first The peak response characteristics are obtained by max pooling each scan block. This indicates a splicing operation. This indicates convolution processing. By concatenating channels and then compressing the dimensions using 1×1 convolution, the representation capability of nodal feature power grid defect regions can be enhanced while suppressing the interference of local noise.

[0125] In operation S440, the scanning priority of each scanning node is determined based on the node information weight, super-edge weight, and average noise weight of the scanning block corresponding to each scanning node.

[0126] Specifically, for the first For each scanning node, we can first determine the average information weight of the scan block corresponding to that scanning node, the sum of the weights of the hyperedges in which that scanning node participates, and the average noise weight of the scan block corresponding to that scanning node; then, based on the above information, we can determine the scanning priority:

[0127] in, Indicates the first The scanning priority of each scanning node. Indicates the first Node characteristics of each scanned node, This indicates the average information weight of the scanned block corresponding to that scanned node. This represents the sum of the weights of the hyperedges that the scanned node participates in. Indicates the superedge The weight of the superedge. This represents the average noise weight of the scan block corresponding to that scan node. This represents the noise adjustment factor.

[0128] Therefore, scanning nodes with higher information content, stronger correlation with defect-related hyperedges, and lower noise can obtain higher scanning priority; while scanning nodes with lower information content, weaker hyperedge correlation, or higher noise can obtain lower scanning priority, so that subsequent scanning processes will focus on the core defect area, the edge area of ​​the equipment, or the abnormal response area.

[0129] In operation S450, the current scan node, the set of used scan superedges, and the noise weight of the current scan node are used as the path state, and the jump action space is determined based on the adjacent scan nodes that the current scan node can reach through unused scan superedges.

[0130] According to an embodiment of the present invention, in the process of generating a nonlinear scan sequence, the scan path planning can be represented as a path search process on a hypergraph. For the scan process at the current moment, the current scan node, the set of used scan hyperedges, and the noise weight of the current scan node can be used together as the path state, specifically represented as:

[0131] Where HGN represents the previous path state, Indicates the currently scanned node. This represents the set of scanned superedges that have already been used. This represents the noise weight corresponding to the current scan node.

[0132] Furthermore, it can be based on the current scanning node In the hypergraph, the connectivity relationships are determined, identifying adjacent scan nodes reachable through unused scan hyperedges, and these adjacent scan nodes are used as the jump action space. Specifically, this can be represented as:

[0133] in, This represents the jump action space corresponding to the currently scanned node. This indicates an adjacent scan node that can be used as the next hop. This represents the scan superedge connecting the current scan node and its adjacent scan node. Indicates the set of superedges to be scanned. This indicates that the scanned superedge set has been used.

[0134] By using the above method, the scanning path can start from the current scanning node at each step and select the next hop node from the adjacent scanning nodes connected by the unused scanning superedge, thereby avoiding repeated scanning and invalid jumps, and providing an action range for subsequent generation of nonlinear scanning sequences based on node priority.

[0135] In operation S460, based on path status, jump action space and node priority of each scan node, the node features of each scan node are non-linearly sorted to generate a non-linear scan sequence.

[0136] According to an embodiment of the present invention, the current path state, the node priorities of candidate scan nodes in the jump action space, and the weights of the scan hyperedges from the current scan node to the candidate scan nodes can be input into a hypergraph convolutional network to obtain the jump probability distribution from the current scan node to each candidate scan node. Specifically, this can be expressed as:

[0137] in, Indicates starting from the current scan node Jump to candidate scan node The jump probability, Indicates the scanning priority of the current scanning node. Indicates the scanning priority of candidate scan nodes. This represents the weight of the scan superedge between the current scan node and the candidate scan nodes. This represents a hypergraph convolutional network. This represents the hyperedge path activation function. Used to output the jump probability distribution of candidate nodes.

[0138] The hyperedge path activation function can be determined based on the hyperedge weight, candidate node priority, scan coverage, and candidate node noise.

[0139]

[0140] in, Indicates a hyperedge association term. Indicates node priority. Indicates the scan coverage item. Indicates the noise suppression term; Indicates the spatial distance attenuation coefficient. This represents the spatial distance between the current scanning node and the candidate scanning nodes. Indicates the number of nodes scanned. N This indicates the total number of nodes scanned. Represents the noise weight of candidate scan nodes. and This represents the balance coefficient.

[0141] Furthermore, based on the jump probability distribution, the next scan node can be selected from the jump action space and added to the scan sequence; the above jump probability calculation and next scan node selection process is repeated until a preset termination condition is met, such as all scan nodes have been sorted or the jump action space is empty. Thus, a nonlinear scan sequence can be obtained:

[0142] in, Represents a non-linear scanning sequence. Indicates the first t The scanned nodes are then sorted and concatenated according to the non-linear scan sequence to obtain serialized features:

[0143] in, This represents the serialized features arranged according to a non-linear scanning sequence. The number of channels representing node features. This represents the real number field. Using the above method, the scanning sequence can prioritize scanning nodes with strong hyperedge correlation, high node priority, and low noise, achieving nonlinear scanning that extends from the defect core region to the edge and background regions.

[0144] In one illustrative embodiment, operation S160 includes: Following a nonlinear scanning sequence, feature scanning and extraction are performed on the node features of each scanning node to obtain defect identification features. Based on these defect identification features, the defect detection results of the power grid equipment are determined.

[0145] According to an embodiment of the present invention, local feature extraction is performed on the node features of each scanning node according to the nonlinear scanning sequence. By guiding the sliding window attention and super-edge weight, fine-grained features such as power grid defects, tower crane edges, and glass insulator status are captured, thereby enhancing the ability to distinguish local details in occluded scenarios and suppressing boundary transition noise interference.

[0146] The specific process may include: sequentially acquiring the node features of each scanning node according to the nonlinear scanning sequence, and modeling the spatial positional relationship and channel response relationship between adjacent scanning nodes within a local window range to obtain local enhancement features.

[0147] Specifically, the relative position encoding can be determined first based on the spatial positions of the query node and the key node:

[0148] in, This represents a relative position encoding function. Indicates the spatial location of the queried node. Indicates the spatial location of the key node. This represents a multilayer perceptron. This indicates a splicing operation.

[0149] Furthermore, channel attention calibration can be performed on the input node features:

[0150] in, This represents the channel attention function. This represents the node features input according to a non-linear scan sequence. Indicates global average pooling. and Indicates learnable parameters, This represents the Sigmoid activation function. This indicates weighting by channel.

[0151] Subsequently, local attention aggregation can be performed within the local window to obtain locally enhanced features:

[0152] in, Indicates the first Node features within a local window , , These represent the projection matrices of the query, key, and value, respectively. Represents the dimension of the key vector. This represents the normalized hyperedge weight matrix corresponding to the scanned node within the i-th local window. Indicates the bias term. This indicates a local enhancement feature.

[0153] Finally, the local enhanced features and the original node features can be residually fused and normalized to obtain the local feature extraction results:

[0154] in, Presentation layer normalization processing, This represents the results of local feature extraction. Using the above method, while processing the features of the scanned nodes according to a nonlinear scanning sequence, the local spatial relationships between nodes can be preserved, and fine-grained defect features such as cracks, damage, corrosion, foreign object edges, and local thermal anomalies can be enhanced.

[0155] Furthermore, global association features can be extracted from the node features of each scanning node according to the nonlinear scanning sequence. This can include: dynamically updating the state based on the local feature extraction results according to the nonlinear scanning sequence to aggregate the global context information between different scanning nodes and obtain global association features.

[0156] Specifically, the features obtained through local feature extraction can be used as sequence input, and the state vector can be updated through a dynamic state-space model:

[0157] in, Indicates the first The state vector corresponding to each scan position This represents the state vector corresponding to the previous scan position. Indicating the first nonlinear scan sequence The input features corresponding to each scan node Represents the global average feature. , , Indicates learnable parameters, This represents the activation function. This represents the gated function.

[0158]

[0159] in, , and This represents the learnable parameters of the gate function. This represents the Sigmoid activation function.

[0160] Furthermore, a modal attention fusion mechanism can be introduced to perform global average pooling and mapping on the local enhancement features corresponding to different modalities, obtaining global features for each modality. Modal attention weights are then determined based on the correlation between the global features of each modality and the average global features of the modality. Finally, the global features of each modality are weighted and fused based on these modal attention weights, and combined with the state vector to obtain a fused state vector, ultimately outputting the global association features. Specifically, this can be represented as:

[0161] in, Indicates modal index; Indicates the first Local enhancement features corresponding to each mode; Indicates global average pooling; Indicates the first Feature mapping networks corresponding to various modalities; Indicates the first Global features corresponding to each modality; The average feature representing the global features of each modality; Represents the attention projection matrix; Indicates the first Modal attention weights corresponding to each modality; Indicates the first The state vector corresponding to each scan position; Represents the modal fusion projection matrix; This represents the state vector after fusing multimodal global information; This represents the feature sequence of scan nodes input according to a nonlinear scan sequence; This represents the fusion state vector corresponding to the last scan position; Indicates the sequence length; Indicates length is A vector of all 1s, used to... Extend to the length corresponding to the feature sequence of the scanned node; Indicates a splicing operation; Represents a multilayer perceptron; This indicates a shape rearrangement operation; This indicates a globally related feature.

[0162] After the above process, the node features of each scanning node can be fused at multiple scales. This can include generating feature representations at multiple scales based on global correlation features, and performing weighted fusion based on the correlation between features at different scales to obtain defect recognition features.

[0163]

[0164] in, Indicates the first Input features at each scale, Indicates the first channel after projection Individual scale features express Convolution processing.

[0165] Subsequently, global average pooling is performed on the features at each scale to obtain the corresponding scale description features:

[0166] in, Indicates the first Each scale corresponds to a scale description feature. Furthermore, based on the correlation between the scale description features and the inter-scale hyperedge weights, the inter-scale correlation weight matrix is ​​calculated:

[0167] in, Indicates the first The first scale and the first The correlation weight matrix between the scales and They represent the first The first scale describes the features and the second Transpose of features described by each scale and They represent the first The first scale and the first The first scale, the first Hyperedge weights between scales.

[0168] Then, based on the inter-scale correlation weight matrix, the features at each scale are weighted and fused:

[0169] in, Indicates the first The fusion features corresponding to each scale Indicates the first The scale feature was adjusted to the [number]th [number]. The dimensions corresponding to each scale and They represent the first The height and width of each scale feature and These represent the first channel after projection. The scale feature and the first Individual scale features.

[0170] Furthermore, the fused features at various scales can be enhanced through channel attention, convolution, and normalization:

[0171] in, Indicates the enhanced first Individual scale features This indicates the channel attention mechanism. express Convolution processing, Presentation layer normalization processing.

[0172] Finally, the enhanced features from multiple scales can be concatenated and processed through convolution and pixel rearrangement to obtain multi-scale fused features, which can then be used as defect recognition features.

[0173] in, Indicates multi-scale fusion features, This indicates a splicing operation. This indicates pixel rearrangement processing. express Convolution processing. By using the above method, information at different scales, such as micro-defects, conventional defects, dense structural regions, and global equipment structure, can be simultaneously fused, thereby enhancing the ability to identify defects in power grid equipment such as micro-cracks in insulators, corrosion of hardware, foreign objects in conductors, and localized thermal anomalies.

[0174] After obtaining the defect identification features, these features can be processed based on a preset defect detection model to determine the defect detection results for power grid equipment. The defect detection results can include at least one of the following: defect category, defect location, and defect confidence level. Specifically, based on the matching relationship between the defect identification features and preset defect categories, it can be determined whether defects such as insulator damage, hardware corrosion, foreign objects in conductors, and localized overheating exist, and the corresponding detection results can be output.

[0175] To enable those skilled in the art to understand the specific process of the power grid inspection method of the present invention, the following specific embodiments are provided: In one specific embodiment, the inspection target is the insulator strings and hardware connection areas in a power transmission line inspected by a drone. The drone is equipped with a visible light camera and an infrared thermal imaging device to collect visible light and infrared images of the power grid equipment at the same inspection location. The visible light images are used to reflect the appearance, edge contours, and surface texture of the insulators, hardware, conductors, and tower components; the infrared images are used to reflect the temperature distribution, localized heating areas, and abnormal thermal changes of the aforementioned power grid equipment.

[0176] The single-mode high-frequency feature vector and single-mode low-frequency feature vector corresponding to the visible light image, as well as the single-mode high-frequency feature vector and single-mode low-frequency feature vector corresponding to the infrared image, can be obtained respectively. Among them, the visible light high-frequency features are mainly used to characterize fine-grained appearance information such as insulator cracks, metal fitting corrosion edges, and conductor foreign object boundaries; the visible light low-frequency features are mainly used to characterize the main outline and spatial structure of the equipment; the infrared high-frequency features are mainly used to characterize local temperature abrupt changes and thermal anomaly boundaries; and the infrared low-frequency features are mainly used to characterize the overall temperature distribution and heat diffusion trend.

[0177] Visible light high-frequency features, visible light low-frequency features, infrared high-frequency features, and infrared low-frequency features are each used as single-mode frequency domain feature nodes to form a single-mode frequency domain feature node set. Based on the similarity and complementarity between visible light and infrared frequency domain features, a cross-modal shared feature node set is constructed. Further, the intra-modal association weights between high-frequency and low-frequency feature nodes within the same modality are determined based on information entropy, and intra-modal hyperedges are constructed based on these weights. Simultaneously, cross-modal association weights are determined based on the semantic relevance and spatial similarity between cross-modal shared feature nodes, and cross-modal hyperedges are constructed based on these weights. Thus, a hypergraph consisting of a node set, a hyperedge set, and hyperedge weights is obtained.

[0178] Subsequently, feature aggregation is performed based on the hypergraph to obtain hypergraph embedding features. In this process, visible light edge features, infrared thermal anomaly features, and equipment structure features related to suspected defect areas can be transferred and fused through hyperedges, enabling the hypergraph embedding features to simultaneously characterize both appearance and thermal anomaly information. Furthermore, modal redundancy stripping is performed on the hypergraph embedding features, preserving the equipment structure, defect edges, and anomaly region information that different modes commonly point to, while reducing modal-specific redundancy introduced by illumination variations, background textures, thermal noise, or imaging artifacts, thereby obtaining aligned features.

[0179] The effective region corresponding to the alignment feature is divided into multiple scan blocks, and each scan block is treated as a scan node. The scan priority of each scan node is determined based on its information weight, the weight of the hyperedges it participates in, and the average noise weight. Scan nodes in regions exhibiting both significant edge changes and temperature anomalies receive higher priority; those in background regions, low-information regions, or regions with strong noise receive lower priority. Furthermore, the path state is determined based on the current scan node, the set of used scan hyperedges, and the noise weight of the current scan node. A jump action space is then determined among adjacent scan nodes reachable from unused scan hyperedges, thereby generating a nonlinear scan sequence that expands from high-priority regions to low-priority regions.

[0180] Finally, following the nonlinear scanning sequence, feature scanning and extraction are performed on the node features of each scanning node. Specifically, local feature extraction is first performed on the node features of each scanning node to enhance fine-grained defect features such as insulator cracks, hardware corrosion, conductor foreign object edges, and local thermal anomaly boundaries; then, global correlation feature extraction is performed to model conductor continuity, tower structural relationships, and spatial correlations between multiple abnormal regions; subsequently, multi-scale fusion processing is performed to take into account micro-defects, conventional defects, and overall equipment structural information to obtain defect identification features. Based on the defect identification features, power grid equipment defect detection results can be output, which may include at least one of defect category, defect location, and defect confidence level. For example, when there is corrosion texture at the hardware connection in the visible light image and a local temperature rise in the corresponding area in the infrared image, the detection result of "hardware abnormality" or "connection part heating abnormality" can be output, and the corresponding defect area can be marked.

[0181] To illustrate the beneficial effects of the power grid inspection method provided by this invention, the following embodiments are given: The dataset used in this embodiment is InsPLAD-det: The InsPLAD-det dataset is specially designed for power grid inspection tasks. The data all come from real power grid scenarios and cover a variety of power grid facilities such as transmission lines and substation equipment at different voltage levels.

[0182] In this embodiment, the dataset is divided into two specialized subsets: the device target detection dataset (InsPLAD-Classification) and the device defect detection dataset (InsPLAD-Defect).

[0183] For InsPLAD-Classification: This subset focuses on locating and classifying key components in the power grid, with labeled targets including insulators, grading rings, tower bodies, and conductors. Its purpose is to train the model to accurately identify and locate various power grid devices in images, laying the foundation for subsequent analysis.

[0184] For InsPLAD-Defect: Building upon equipment localization, this subset further focuses on the health status diagnosis of equipment components, meticulously annotating subtle defect features such as insulator damage, spontaneous explosion, broken conductor strands, hardware corrosion, and loose bolts. This subset is directly used to train models to determine whether equipment has faults and their specific types.

[0185] In addition, the STN-PLAD dataset is provided, which focuses on the detection of key components and conditions in substation scenarios. It contains image data of various substation equipment (such as circuit breakers, disconnectors, and instrument transformers) and labels their normal operating status and various typical defects (such as component displacement, abnormal oil level, and abnormal meter readings).

[0186] A series of comparative experiments were designed below to compare and analyze the results with current mainstream object detection methods in four dimensions: detection accuracy, inference speed, model complexity, and generalization ability. The experimental environment included server-side (NVIDIA RTX 3090) and mobile devices (Huawei Mate 40 Pro, Android 12, Kirin 9000). The dataset used was a joint training and test set of InsPLAD-det and STN-PLAD, containing 20,052 training images and 3,786 validation images.

[0187] To further verify the effectiveness of the power grid inspection method of the present invention, this embodiment selects several mainstream target detection methods as comparative algorithms for experimental analysis. These comparative algorithms include DSSD, RefineDet, YOLOv3, YOLOv5, YOLOv8, YOLOv11, Gold-YOLO, BGF-YOLO, FFCA-YOLO, DINO, DNTR, RFLA, Mamba-YOLO, and FBRT-YOLO. These comparative algorithms cover classic target detectors, YOLO series detection methods, Transformer-based detection methods, remote sensing target detection methods, and improved methods for small target detection. They enable a comprehensive evaluation of the detection performance of the power grid inspection method of the present invention from the perspectives of different detection frameworks and different feature modeling methods. Specific comparison results are shown in Table 1.

[0188] Table 1 Comparison results of the embodiments of the present invention and the prior art

[0189] Wherein, mAP@0.5 represents the average precision calculated under the condition that the cross-union ratio threshold is 0.5, which is used to comprehensively evaluate the overall detection capability of the detection method for different types of defect targets; Precision represents the precision rate, which is used to characterize the proportion of real defects in the results detected as defect targets. The higher the precision rate, the fewer false detections; Recall represents the recall rate, which is used to characterize the proportion of real defect targets that are successfully detected. The higher the recall rate, the fewer missed detections.

[0190] As shown in Table 1, the present invention achieves 98.6%, 97.8%, and 96.1% accuracy in mAP@0.5, Precision, and Recall, respectively, all of which are superior to the listed comparison methods. This indicates that the present invention has high overall detection accuracy in power grid inspection, effectively reducing false detections and missed detections, and is particularly beneficial for improving the stability of defect identification in complex inspection scenarios.

[0191] To verify the contribution of different processing steps of this invention to power grid inspection performance, this embodiment designed an ablation experiment. Based on the benchmark detection method, "hypergraph fusion processing" and "nonlinear scanning and feature extraction processing" were introduced respectively, and the detection effect when both were introduced simultaneously was tested. Hypergraph fusion processing includes: constructing a two-layer hypergraph node set and hyperedge set based on multimodal frequency domain features, and performing feature aggregation and modal redundancy stripping based on the hypergraph to obtain aligned features; nonlinear scanning and feature extraction processing includes: determining the node priority of the scanning nodes based on the hypergraph and aligned features, generating a nonlinear scanning sequence, and performing feature scanning and extraction on the node features of the scanning nodes according to the nonlinear scanning sequence. YOLOv11 was selected as the benchmark detection method in the experiment, and the experimental settings and evaluation indicators were consistent with the previous experiments. "×" indicates that the corresponding processing was not introduced, and "√" indicates that the corresponding processing was introduced. Specific results are shown in Table 2.

[0192] Table 2 Ablation Experiment Results

[0193] As shown in Table 2, both introducing hypergraph fusion processing or nonlinear scanning and feature extraction processing alone can improve detection performance based on the benchmark method. When both are introduced simultaneously, mAP@0.5, Precision, and Recall reach 98.6%, 97.8%, and 96.1%, respectively, achieving the best detection effect. This indicates that the present invention improves the multimodal feature fusion effect through hypergraph fusion processing and enhances the feature representation of defect-related regions through nonlinear scanning and feature extraction processing. The combination of these two methods can further improve the accuracy of power grid inspection and reduce false detections and missed detections.

[0194] The power grid inspection device provided by the present invention is described below. The power grid inspection device described below and the power grid inspection method described above can be referred to in correspondence.

[0195] Figure 5 A structural block diagram of an example power grid inspection device of the present invention is shown.

[0196] like Figure 5As shown, the power grid inspection device 500 includes an acquisition module 510, a frequency domain decomposition module 520, a hypergraph construction module 530, a redundancy stripping module 540, a sequence determination module 550, and a scanning module 560.

[0197] The acquisition module 510 is used to acquire visible light and infrared images of power grid equipment; The frequency domain decomposition module 520 is used to extract features from visible light images and infrared images and perform frequency domain decomposition to obtain multimodal frequency domain features. The hypergraph construction module 530 is used to construct a two-layer hypergraph node set based on multimodal frequency domain features, and to construct a hyperedge set based on the two-layer hypergraph node set to obtain the hypergraph; The redundancy stripping module 540 is used to perform feature aggregation based on the hypergraph to obtain hypergraph embedding features, and to perform modal redundancy stripping on the hypergraph embedding features to obtain aligned features; The sequence determination module 550 is used to determine the node priority of each scan node based on the hypergraph and alignment features, and generate a nonlinear scan sequence based on the node priority; The scanning module 560 is used to perform feature scanning and extraction on aligned features based on a nonlinear scanning sequence to obtain the defect detection results of power grid equipment.

[0198] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6 As shown, the electronic device may include a processor 610, a communications interface 620, a memory 630, and a communication bus 640. The processor 610, communications interface 620, and memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions from the memory 630 to execute a power grid inspection method.

[0199] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0200] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the power grid inspection method provided by the above methods.

[0201] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0202] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0203] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A power grid inspection method, characterized in that, include: Acquire visible light and infrared images of power grid equipment; Feature extraction and frequency domain decomposition are performed on the visible light image and the infrared image respectively to obtain multimodal frequency domain features; A two-layer hypergraph node set is constructed based on the multimodal frequency domain features, and a hyperedge set is constructed based on the two-layer hypergraph node set to obtain the hypergraph; Based on the hypergraph, feature aggregation is performed to obtain hypergraph embedding features, and modal redundancy is stripped from the hypergraph embedding features to obtain alignment features; Based on the hypergraph and the alignment features, the node priority of each scan node is determined, and a nonlinear scan sequence is generated based on the node priority. Based on the nonlinear scanning sequence, the alignment features are scanned and extracted to obtain the power grid equipment defect detection results.

2. The power grid inspection method according to claim 1, characterized in that, The multimodal frequency domain features include the single-mode high-frequency feature vector and the single-mode low-frequency feature vector corresponding to the visible light image, and the single-mode high-frequency feature vector and the single-mode low-frequency feature vector corresponding to the infrared image; The step of extracting features from the visible light image and the infrared image respectively and performing frequency domain decomposition to obtain multimodal frequency domain features includes: Enhanced wavelet transforms are performed on the visible light image and the infrared image respectively to obtain their respective single-mode high-frequency feature vectors; Fast Fourier Transform is performed on the visible light image and the infrared image respectively to obtain their respective single-mode low-frequency feature vectors.

3. The power grid inspection method according to claim 2, characterized in that, The construction of a two-layer hypergraph node set based on the multimodal frequency domain features includes: Based on the single-mode high-frequency feature vector and single-mode low-frequency feature vector corresponding to the visible light image and the infrared image, a set of single-mode frequency domain feature nodes is constructed; Cosine similarity is calculated for each of the single-modal high-frequency feature vectors and single-modal low-frequency feature vectors to obtain inter-modal attention weights; Based on the intermodal attention weights, the channel dimensions of the single-modal frequency domain feature nodes corresponding to different modalities are concatenated to construct a cross-modal shared feature node set.

4. The power grid inspection method according to claim 3, characterized in that, The construction of the hyperedge set includes: The intramodal association weights between single-modal high-frequency feature nodes and single-modal low-frequency feature nodes within the same modality are determined based on the information entropy, and intramodal hyperedges are constructed based on the intramodal association weights. Based on the semantic relevance and spatial similarity among multiple cross-modal shared feature nodes, cross-modal association weights are determined, and cross-modal hyperedges are constructed based on the cross-modal association weights.

5. The power grid inspection method according to claim 1, characterized in that, The process of performing feature aggregation based on the two-layer hypergraph node set and the hyperedge set to obtain hypergraph embedding features, and then performing modal redundancy stripping on the hypergraph embedding features to obtain alignment features, includes: The hypergraph is subjected to feature aggregation using a hypergraph convolutional network to obtain hypergraph embedding features; Factor unwrapping is performed on the hypergraph embedding features to obtain cross-modal shared factors and modality-specific redundancy factors; The alignment feature is obtained by reconstructing and decoding based on the cross-modal shared factor and the modality-specific redundancy factor.

6. The power grid inspection method according to claim 1, characterized in that, The step of determining the node priority of each scan node based on the hyperedge set and the alignment feature, and generating a nonlinear scan sequence based on the node priority, includes: Determine the information weights corresponding to the alignment features, and crop the alignment features based on the information weights to obtain a preliminary feature map; The preliminary feature map is filtered to obtain an effective feature map; The effective feature map is divided into multiple scan blocks, each scan block is used as a scan node, and the node features of the scan node are determined. The scanning priority of each scanning node is determined based on the node information weight, hyperedge weight, and average noise weight of the scanning block corresponding to each scanning node. The current scan node, the set of used scan superedges, and the noise weight of the current scan node are used as the path state, and the jump action space is determined based on the adjacent scan nodes that the current scan node can reach through the unused scan superedges. Based on the path state, the jump action space, and the node priority of each scan node, the node features of each scan node are non-linearly sorted to generate the non-linear scan sequence.

7. The power grid inspection method according to claim 6, characterized in that, The step of performing feature scanning and extraction on the alignment features based on the nonlinear scanning sequence to obtain the power grid equipment defect detection results includes: According to the nonlinear scanning sequence, the node features of each scanning node are scanned and extracted to obtain defect identification features; Based on the defect identification features, the defect detection results of the power grid equipment are determined.

8. A power grid inspection device, characterized in that, include: The acquisition module is used to acquire visible light and infrared images of power grid equipment; The frequency domain decomposition module is used to extract features from the visible light image and the infrared image respectively and perform frequency domain decomposition to obtain multimodal frequency domain features; The hypergraph construction module is used to construct a two-layer hypergraph node set based on the multimodal frequency domain features, and to construct a hyperedge set based on the two-layer hypergraph node set to obtain the hypergraph; A redundancy stripping module is used to perform feature aggregation based on the hypergraph to obtain hypergraph embedding features, and to perform modal redundancy stripping on the hypergraph embedding features to obtain alignment features; The sequence determination module is used to determine the node priority of each scan node based on the hypergraph and the alignment features, and to generate a nonlinear scan sequence based on the node priority; The scanning module is used to perform feature scanning and extraction on the alignment features based on the nonlinear scanning sequence to obtain the power grid equipment defect detection results.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the power grid inspection method as described in any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the power grid inspection method as described in any one of claims 1 to 7.

Images