Power equipment defect identification and alarm method and system based on deep learning

Abstract

The application discloses a kind of power equipment defect identification and warning method and system based on deep learning.The method comprises: synchronously collecting and registering the visible light and infrared thermal imaging image on the surface of power equipment, constructs the instance segmentation network including light weight feature extraction network, multiscale feature fusion network and frequency domain mask prediction branch;Adopt the generative adversarial strategy to enhance the diversity of training sample;Based on graph neural network, analyze the association between defect and equipment topology, historical record, infer the cause-effect relationship of defect and risk level;Generate the interpretable warning information including heat map, natural language report and repair suggestion;Real-time detection and deep analysis are realized using end-edge-cloud collaborative architecture;Through closed-loop optimization mechanism, continuously improve system performance.The application realizes high-precision defect detection under multi-modal data fusion, has strong robustness and interpretability, significantly improves the intelligent level of power equipment operation and maintenance.

Description

Technical Field

[0001] This invention relates to the field of power equipment condition monitoring and intelligent operation and maintenance technology, and in particular to a method and system for power equipment defect identification and alarm based on deep learning. Background Technology

[0002] With the continuous expansion of power system scale and the ongoing improvement of its intelligence level, the safe and stable operation of power equipment has become a key link in ensuring the reliability of the power grid. Traditional power equipment defect detection mainly relies on manual inspection and basic image processing technology, which has prominent problems such as low efficiency, strong subjectivity, and high false negative rate. Especially under complex environmental conditions, such as changes in lighting, severe weather, or dirt on the equipment surface, the detection performance of traditional methods is significantly reduced.

[0003] In recent years, deep learning-based visual inspection technology has made some progress in the field of industrial defect detection. However, it still faces many challenges in specific application scenarios of power equipment: First, power equipment defects are diverse in type and shape, ranging from macroscopic structural deformation to microscopic surface cracks, and a single modal image is difficult to fully capture all defect features; second, there are modal differences between infrared thermal imaging and visible light images, and direct fusion will lead to feature mismatch problems; in addition, existing methods are not capable of fine segmentation of defect edges, making it difficult to meet the requirements of high-precision positioning; finally, there is a lack of in-depth analysis capabilities on defect causal relationships and risk propagation paths, resulting in a lack of foresight in operation and maintenance decisions.

[0004] Current research has attempted to improve detection performance using multimodal learning or attention mechanisms, but most methods have failed to effectively address core issues such as bimodal data alignment, multi-scale feature fusion, and edge detail optimization. Furthermore, existing systems typically lack end-to-end interpretable output and closed-loop optimization mechanisms, making them ill-suited to the dynamic detection needs of power field environments.

[0005] Therefore, there is an urgent need for an intelligent defect identification and alarm method that can integrate multimodal information, achieve fine segmentation, support correlation analysis, and have continuous evolution capabilities. Summary of the Invention

[0006] The present invention aims to overcome the shortcomings of the prior art and provide a method and system for identifying and alarming power equipment defects with high accuracy, high efficiency and strong robustness.

[0007] In a first aspect, embodiments of this application provide a method for identifying and alarming defects in power equipment based on deep learning, the method comprising:

[0008] S1. Simultaneously acquire visible light images and infrared thermal images of the surface of power equipment, perform timestamp alignment and spatial registration processing, and generate registered multimodal image pairs;

[0009] S2. Based on the registered multimodal image pairs, an instance segmentation network is constructed, wherein the instance segmentation network includes:

[0010] The feature extraction network employs a lightweight convolutional structure and embeds a spatial attention mechanism.

[0011] A multi-scale feature fusion network employs a pyramid structure with global context enhancement and bidirectional feature aggregation;

[0012] The mask prediction branch uses a frequency domain transformation method for boundary refinement.

[0013] S3. Based on the collected multimodal image data, a generative adversarial strategy is adopted to generate diverse training samples, thereby improving the robustness of the instance segmentation network in complex environments;

[0014] S4. Based on the defect detection results output by the instance segmentation network, a graph neural network is used to analyze the correlation between defects and equipment topology and historical records, and to infer the causal relationship and risk level of defects;

[0015] S5. Based on the defect correlation analysis results, generate interpretable alarm information including defect heatmaps, natural language reports, and maintenance suggestions;

[0016] S6. It adopts an edge-cloud collaborative architecture, deploying lightweight models at the edge for real-time detection, and performing in-depth analysis and model optimization in the cloud;

[0017] S7. Based on cloud-based analysis results and on-site maintenance verification data, establish a comparison mechanism between model predictions and actual results to continuously optimize the performance of the instance segmentation network and alarm system.

[0018] Secondly, embodiments of this application provide a deep learning-based power equipment defect identification and alarm system, applied to the deep learning-based power equipment defect identification and alarm method described in the first aspect, the system comprising:

[0019] The multimodal data acquisition and registration module is used to simultaneously acquire visible light images and infrared thermal imaging images of the surface of power equipment, perform timestamp alignment and spatial registration processing, and generate registered multimodal image pairs.

[0020] The instance segmentation network construction module is used to construct an instance segmentation network based on the registered multimodal image pairs, which includes a feature extraction network, a multi-scale feature fusion network, and a mask prediction branch. The feature extraction network adopts a lightweight convolutional structure and embeds a spatial attention mechanism. The multi-scale feature fusion network adopts a pyramid structure with global context enhancement and bidirectional feature aggregation. The mask prediction branch uses a frequency domain transformation method for boundary refinement.

[0021] The adversarial enhancement training module is used to generate diverse training samples based on the collected multimodal image data using a generative adversarial strategy, thereby improving the robustness of the instance segmentation network in complex environments.

[0022] The defect correlation analysis module is used to analyze the defect detection results output by the instance segmentation network, and uses graph neural networks to analyze the correlation between defects and equipment topology and historical records, and infer the causal relationship and risk level of defects.

[0023] The interpretable alarm generation module is used to generate interpretable alarm information, including defect heatmaps, natural language reports, and maintenance suggestions, based on the results of defect correlation analysis.

[0024] The collaborative reasoning deployment module is used to deploy lightweight models at the edge for real-time detection and perform in-depth analysis and model optimization in the cloud using an edge-cloud collaborative architecture.

[0025] The closed-loop optimization feedback module is used to establish a comparison mechanism between model predictions and actual results based on cloud analysis results and on-site maintenance verification data, and to continuously optimize the performance of the instance segmentation network and alarm system.

[0026] Thirdly, embodiments of this application provide an electronic device, including:

[0027] processor;

[0028] Memory used to store processor-executable instructions;

[0029] The processor is configured to implement the deep learning-based power equipment defect identification and alarm method as described in the first aspect when executing the instructions.

[0030] Fourthly, embodiments of this application provide a computer-readable storage medium storing a program that instructs a device to execute the deep learning-based power equipment defect identification and alarm method as described in the first aspect.

[0031] The beneficial effects of this invention are as follows:

[0032] 1. Improved detection accuracy and robustness: By fusing multimodal data from visible light and infrared thermal imaging, combined with a lightweight instance segmentation network and frequency domain mask prediction technology, high-precision identification of defects in power equipment is achieved, and strong robustness is maintained, especially in complex environments.

[0033] 2. Enhanced feature extraction capability: The dual-branch feature extraction network with embedded attention mechanism effectively captures complementary information from dual-modal data; the multi-scale feature pyramid structure and global context enhancement module strengthen the feature representation capability of defects at different scales.

[0034] 3. Optimize edge detail processing: Innovatively introduce a mask prediction method based on frequency domain transformation, which significantly improves the refinement of defect boundaries through multi-stage refinement processing, reducing missed detections and false detections.

[0035] 4. Achieve intelligent correlation analysis: Utilize graph neural networks to construct equipment knowledge graphs, deeply explore the correlation between defects and equipment topology and historical records, support defect causal reasoning and risk level assessment, and provide a scientific basis for operation and maintenance decisions.

[0036] 5. Improve system usability and efficiency: Adopt an edge-cloud collaborative architecture to achieve real-time lightweight detection at the edge and perform in-depth analysis and model optimization in the cloud, balancing detection efficiency and system resource constraints, and supporting large-scale deployment.

[0037] 6. Enhanced interpretability of results: Generates multimodal alarm information including heatmaps, natural language reports, and maintenance suggestions, improving the readability and operability of the results and helping maintenance personnel quickly locate and handle defects.

[0038] 7. Supports continuous self-optimization: Through a closed-loop feedback mechanism, the model performance is continuously optimized by combining on-site maintenance data, forming a data-driven self-evolution capability, and maintaining the system's high accuracy and adaptability in the long term. Attached Figure Description

[0039] Figure 1 This is a schematic diagram of a deep learning-based power equipment defect identification and alarm method provided in an embodiment of this application.

[0040] Figure 2 The system architecture diagram of the power equipment defect identification and alarm system based on deep learning provided for this application.

[0041] Figure 3 A schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0042] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them.

[0043] It should be noted that in the embodiments of this application, "at least one" refers to one or more, and "more than one" refers to two or more. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the specification of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application.

[0044] Based on the embodiments described in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0045] Example 1

[0046] Figure 1 This is a schematic flowchart illustrating a deep learning-based power equipment defect identification and alarm method according to an embodiment of this application. Figure 1 As shown, a method for defect identification and alarm of power equipment based on deep learning includes:

[0047] S1. Simultaneously acquire visible light images and infrared thermal images of the power equipment surface, perform timestamp alignment and spatial registration processing to generate registered multimodal image pairs. Multimodal image acquisition and registration simultaneously acquires visible light images (providing detailed information such as texture and shape) and infrared thermal images (providing information on temperature distribution anomalies) of the power equipment surface. Through rigorous timestamp alignment and spatial registration processing, accurately corresponding multimodal image pairs are generated, providing a high-quality, aligned input data foundation for subsequent fusion analysis.

[0048] Specifically, in this embodiment, S1 includes:

[0049] A synchronous triggering device is used to control the visible light camera and the infrared thermal imager to acquire images synchronously, ensuring that dual-modal image data of the power equipment surface is obtained at the same time. Synchronization is achieved from the source of data acquisition through hardware means. A dedicated synchronous triggering device (such as a hardware synchronization signal generator) is used to control the two cameras to expose at the same time, fundamentally reducing target displacement or deformation between images caused by asynchronous acquisition time. This is a prerequisite for achieving high-quality registration.

[0050] The acquired dual-modal images undergo timestamp matching, and time deviation is controlled within milliseconds through hardware clock synchronization and software time correction; precise time alignment is performed at the data level. Each acquired image is assigned a high-precision timestamp, and matching and fine-tuning are performed using algorithms. Controlling time deviation within milliseconds ensures that even continuously acquired image sequences can be accurately correlated, meeting the requirements of real-time or near-real-time processing systems.

[0051] A feature-point-based image registration algorithm is employed, which is the core step in the registration process. The algorithm involves: 1) Extracting SIFT feature points: SIFT (Scale Invariant Feature Transform) feature pairs are invariant to rotation, scaling, and brightness changes, making them ideal for extracting stable and reliable feature points from visible and infrared images. 2) Using the RANSAC algorithm to eliminate mismatches: Due to modal differences, a large number of erroneous feature matching pairs inevitably exist. The RANSAC (Random Sample Consensus) algorithm can robustly estimate the correct transformation model while ignoring the interference of these mismatched point pairs, improving registration accuracy. 3) Calculating the affine transformation matrix: Using the correct matching point pairs, an affine transformation model (including translation, rotation, scaling, and shearing) is calculated. This transformation is applied to map the infrared image to the coordinate system of the visible light image, ultimately achieving pixel-level precise spatial alignment.

[0052] The registered dual-modal images undergo resolution unification processing. Bicubic interpolation is used to upsample the infrared image to the same resolution as the visible light image, addressing the issue of inconsistent data scales. Infrared thermal imagers typically have lower resolution than visible light cameras. This step uses bicubic interpolation, a high-quality image scaling algorithm, to upsample the lower-resolution infrared image to the same resolution as the visible light image. This allows subsequent networks to process and fuse features from both images at the same spatial scale, simplifying network design.

[0053] A multimodal image database is established to store registered image pairs along with their corresponding acquisition time, device number, and environmental parameter metadata. This enables effective data organization and management. It not only stores processed image pairs but also their associated metadata (acquisition time, device number, environmental parameters, etc.). This provides structured data support for subsequent model training, data analysis, and traceability queries, and is a crucial part of building a complete system.

[0054] S2. Based on the registered multimodal image pairs, an instance segmentation network is constructed. This network includes: a feature extraction network employing a lightweight convolutional structure and embedding a spatial attention mechanism; a multi-scale feature fusion network using a pyramid structure with global context enhancement and bidirectional feature aggregation; and a mask prediction branch employing a frequency domain transformation method for boundary refinement. The instance segmentation network is constructed as a deep learning network model specifically designed for multimodal power equipment defect detection. This network integrates: a feature extraction network that efficiently extracts features from the registered bimodal images, employing a lightweight design (such as depthwise separable convolutions) to reduce computational burden and embedding an attention mechanism to focus on key regions; a multi-scale feature fusion network that fuses features from different network layers to simultaneously capture global semantic information and local detail information of defects, employing an enhanced pyramid structure to improve fusion performance; and a mask prediction branch that not only predicts the defect category and location (boundary box) but also generates accurate pixel-level segmentation masks, specifically employing a frequency domain transformation method to refine defect boundaries and improve segmentation accuracy.

[0055] Specifically, in this embodiment, S2 includes:

[0056] S2.1: Construct a bimodal feature extraction network, employing a parallel dual-branch structure to process visible light and infrared images separately. Each branch includes: a backbone network based on depthwise separable convolutions, using an inverse residual structure to reduce computational complexity; an embedded coordinate attention mechanism to capture spatial location features through coordinate information embedding blocks; and a cross-modal feature interaction module to achieve information exchange and fusion of bimodal features. The bimodal feature extraction network is designed to process and effectively fuse image data from two different modalities: visible light and infrared. The parallel dual-branch structure allows for independent feature extraction from the two modalities in the initial stage, avoiding direct interference from modal differences and preserving the unique features of each modality. The depthwise separable convolution-based backbone network significantly reduces the number of model parameters and computational cost while maintaining feature extraction capabilities, meeting the requirements of lightweight design and real-time performance. The embedded coordinate attention mechanism enables the network to focus not only on the specific features but also on their specific locations, enhancing spatial awareness and aiding in accurate defect localization. Cross-modal feature interaction module: Based on independent feature extraction, bimodal information is exchanged and fused in a controlled manner at a specific network layer, so that the texture details of visible light and the temperature anomaly information of infrared light complement and enhance each other, achieving a fusion effect of 1+1>2.

[0057] Specifically, in this embodiment, the specific implementation of constructing the dual-modal feature extraction network in S2.1 includes:

[0058] S2.1.1: Construct a parallel dual-branch input processing channel. The first branch receives the registered visible light image, and the second branch receives the registered infrared thermal image. Both branches share the same network structure but learn their parameters independently. Constructing this parallel dual-branch input processing channel establishes the network's basic input architecture. Two independent input branches are explicitly designated to process the registered visible light image and the infrared thermal image respectively. The strategy of sharing the same network structure but learning parameters independently ensures the consistency of the dual-modal processing flow while allowing the network to adaptively learn different feature representations based on modal characteristics, laying the foundation for subsequent feature fusion.

[0059] S2.1.2: Construct a feature extraction backbone network based on depthwise separable convolution in each branch, using an inverted residual structure. Specifically, the first layer uses 1×1 convolution to increase channel dimensionality and enhance feature representation; the second layer uses 3×3 depthwise separable convolution to extract spatial features, significantly reducing the number of parameters; the third layer uses 1×1 convolution to reduce channel dimensionality and restore the original number of channels; and a skip connection structure is used to add the input and output features to avoid gradient vanishing.

[0060] A feature extraction backbone network based on depthwise separable convolutions is constructed to achieve efficient and lightweight feature extraction. The network structure within each branch is specified as follows: inverted residual structure (1x1 dimensionality upscaling → 3x3 depthwise separable convolutions → 1x1 dimensionality reduction): This significantly reduces computational cost and parameter count while maintaining feature expressiveness, meeting the lightweight deployment requirements of edge devices. Skip connections: These alleviate the vanishing gradient problem during deep network training, ensuring that low-level detailed features are effectively passed to subsequent layers, helping to preserve fine-grained information about defects.

[0061] S2.1.3: An coordinate attention mechanism module is embedded after each inverted residual block. This module is implemented in the following way: global pooling is performed along the height and width directions respectively to obtain feature descriptors in two directions; the feature descriptors in the two directions are concatenated and transformed by 1×1 convolution; a spatial attention weight map is generated using the sigmoid activation function; the attention weights are multiplied with the original features to enhance the feature response of important spatial locations.

[0062] An embedded coordinate attention mechanism module enhances the network's ability to perceive and focus on spatial location information. Specifically, spatial attention generation is achieved through: global pooling along the height and width directions to capture spatially directional global information and obtain precise location information encoding; generating a spatial attention weight map to convert the location information encoding into attention weights, explicitly indicating which spatial locations in the image require focused attention; and enhancing feature responses by using weighted multiplication to strengthen features in key defect-related regions and suppress interference from irrelevant background regions.

[0063] S2.1.4: Construct a cross-modal feature interaction module to achieve information fusion of dual-modal features. Specifically, this includes: setting a feature exchange layer at a specific network depth to achieve feature interaction between modalities through a cross-attention mechanism; using a feature alignment module to ensure the consistency of different modal features in spatial location; and adopting an adaptive weight learning mechanism to dynamically adjust the fusion ratio of different modal features.

[0064] A cross-modal feature interaction module is constructed to achieve effective information exchange and adaptive fusion between bimodal features. This is the core of the fusion: A feature exchange layer is set at a specific network depth: interaction is performed at appropriate network layers (such as intermediate layers with sufficient semantic depth) to avoid problems caused by premature or late fusion. Interaction is achieved through a cross-attention mechanism: utilizing a query-key-value (QKV) mechanism, features from one modality (e.g., visible light as the query) search for and aggregate relevant information from features from another modality (e.g., infrared as the key and value), achieving guided and precise feature complementarity. An adaptive weight learning mechanism dynamically learns and adjusts the fusion ratio of features from different modalities, rather than simply adding or concatenating them, enabling the network to determine which modal information to rely on more based on the specific input content.

[0065] S2.1.5: By cascading multiple building blocks, bimodal features are gradually extracted and fused, ultimately outputting a fused feature map containing rich semantic information, providing input for the subsequent multi-scale feature fusion network. Cascading building blocks and outputting fused features constructs a complete feature extraction process and produces the final output. Cascading multiple building blocks: By stacking multiple specific basic units defined in steps S2.1.2, S2.1.3, and S2.1.4—specific feature extraction, attention enhancement, and cross-modal interaction—features are gradually and deeply extracted and fused, enabling the network to learn complex feature representations from low to high levels. Outputting a fused feature map: The final feature map simultaneously contains detailed texture information from the visible light modality and temperature anomaly information from the infrared modality. After spatial attention filtering and cross-modal complementary enhancement, a specific fused feature map containing rich semantic information is formed, providing high-quality input for the subsequent multi-scale feature fusion network (S2.2).

[0066] S2.2: Constructing an enhanced multi-scale feature pyramid network, specifically including: employing a bidirectional feature aggregation path (top-down and bottom-up); introducing a global context enhancement module to extract global context information through global average pooling and 1×1 convolution; and using a feature reweighting mechanism to dynamically adjust the contribution of features at different scales. This enhanced multi-scale feature pyramid network addresses the problem of varying defect scales in power equipment (e.g., the coexistence of large-area contamination and small-sized cracks), effectively fusing features from different levels. The bidirectional feature aggregation path includes: a top-down path that transmits strong semantic features from higher levels to lower levels, enriching the semantic information of lower-level features; and a bottom-up path that transmits fine-grained location and detail features from lower levels to higher levels, enhancing the localization capability of higher-level features. The global context enhancement module introduces global context information to help the network better understand the overall scene and avoid misjudging local noise as defects. The feature reweighting mechanism dynamically learns and assigns importance weights to feature maps at different scales, allowing the network to adaptively focus on the feature scale most effective for the current detection target.

[0067] The specific implementation of constructing the enhanced multi-scale feature pyramid network in step S2.2 includes:

[0068] S2.2.1: Construct a top-down feature propagation path, receive feature maps from the deepest layer of the backbone network, and pass high-level semantic information upward layer by layer through upsampling operations. Specifically, this includes: upsampling the high-level features by 2 times and using bilinear interpolation to maintain spatial continuity; adding the upsampled features element-wise with the features from the corresponding layers of the backbone network; and using 3×3 convolution to smooth the fused features and eliminate the aliasing effect caused by upsampling.

[0069] A top-down feature propagation path is constructed to achieve the transfer and fusion of high-level semantic information to lower levels. High-level features are upsampled by a factor of 2: deep, semantically rich but low-resolution feature maps are enlarged to match the spatial dimensions of shallow feature maps. The upsampled high-semantic features are then added element-wise to the corresponding layer features of the backbone network, fusing them with the detailed shallow features from the corresponding layer of the backbone network. This combines semantic information (what exactly it is) with locational detail information (where exactly it is). A 3×3 convolution is used for smoothing, eliminating aliasing effects (jawy edges) and inconsistencies after feature fusion that may result from the upsampling operation, making the fused feature map smoother and more consistent.

[0070] S2.2.2: Construct a bottom-up feature enhancement path, receive detailed features from the shallow layers of the backbone network, and pass spatial detail information down layer by layer through downsampling operations. Specifically, this includes: downsampling the bottom features by 2 times, using a 3×3 convolution with a stride of 2 to maintain feature integrity; concatenating the downsampled features with the features from the corresponding layers of the top-down path; and adjusting the number of channels using a 1×1 convolution to keep the number of feature channels consistent across layers.

[0071] A bottom-up feature enhancement path is constructed to strengthen the transmission and supplementation of low-level detailed information to high-level features. Low-level features are downsampled by a factor of 2: strided convolutions are used to reduce the dimensionality of shallow, high-resolution but low-semantic feature maps while preserving important feature information. Channel concatenation with top-down path features: The dimensionality-reduced detailed features are concatenated with the semantically integrated features from the top-down path to further enrich the channel dimensionality of the features. The number of channels is adjusted using 1×1 convolutions: the surge in the number of channels after concatenation is compressed, reducing computational complexity while maintaining consistency in the number of channels in the output feature maps of each level, facilitating subsequent processing.

[0072] S2.2.3: A global context enhancement module is introduced at each pyramid level, which is implemented in the following ways: global average pooling is performed on the input feature map to extract global context information; two 1×1 convolutions are used for feature transformation with a ReLU activation function added in between; channel attention weights are generated through the sigmoid activation function; and the attention weights are multiplied with the original features to enhance the discriminative feature channels.

[0073] A global context enhancement module is introduced to inject global context information into features at each pyramid level. Global average pooling compresses the input feature map spatially, resulting in a global feature description vector containing the global context information of the entire feature map. Feature transformation and activation transform and interact with the global features through fully connected layers (implemented with 1x1 convolutions) and a non-linear activation function (ReLU). Channel attention weights are generated by applying a sigmoid function to each channel of the transformed features, identifying which channels are more important for the current task. Discriminative features are enhanced by multiplying the channel weights by the original features, thereby enhancing the responses of important feature channels and suppressing the responses of unimportant channels, improving the discriminative power of the features.

[0074] S2.2.4: A feature reweighting mechanism is adopted to dynamically adjust the contribution of features at different scales. Specifically, this includes: performing global average pooling on the output features of each level to obtain the scale description vector; calculating the weight coefficients of each level through fully connected layers and the softmax function; and using the calculated weight coefficients to perform weighted fusion of features at each level.

[0075] A feature reweighting mechanism is employed to dynamically evaluate and fuse the contributions of feature maps at different scales. Global average pooling yields a scale description vector: global average pooling is performed on the feature maps output at each pyramid level to obtain a vector representing global information at that scale. Weight coefficients for each level are calculated: through a small fully connected network and the Softmax function, the importance weights of the feature maps at each scale are learned and calculated. Weighted fusion: the obtained weight coefficients are used to perform a weighted summation of the feature maps at each scale, achieving adaptive multi-scale feature fusion, allowing the network to focus more on the scale most effective for the current target.

[0076] S2.2.5: Outputs an enhanced multi-scale feature pyramid, containing four feature maps at different scales {P2, P3, P4, P5}, corresponding to downsampling rates of 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the input image, respectively, providing multi-scale feature support for subsequent mask prediction.

[0077] The output is an enhanced multi-scale feature pyramid, generating the final set of multi-scale feature maps for prediction. Four enhanced feature maps at different scales (P2, P3, P4, P5) are output, corresponding to different downsampling factors of the input image (1 / 4, 1 / 8, 1 / 16, 1 / 32). P2 / P3 (higher resolution): rich in detail information, beneficial for detecting small-scale defects and accurate boundary localization. P4 / P5 (lower resolution): rich in semantic information, beneficial for detecting large-scale defects and identifying defect categories. This provides comprehensive and enhanced multi-scale feature support for subsequent mask prediction branches (such as RPN, detection head, and segmentation head).

[0078] Specifically, the global context enhancement module in S2.2.3 also includes a spatial attention component, which generates a spatial attention map through convolution operations to further improve the feature response of important spatial regions. The feature reweighting mechanism in S2.2.4 adopts a lightweight gated recurrent unit structure to learn the long-term dependencies between features at different scales.

[0079] S2.3: Construct a mask prediction branch based on frequency domain transformation. The specific implementation is as follows: Divide the prediction mask into K×K image blocks; perform discrete cosine transform on each block to extract frequency domain features; use a lightweight regression network to predict key frequency domain coefficients; reconstruct a refined mask through inverse discrete cosine transform. Constructing a mask prediction branch based on frequency domain transformation achieves more accurate pixel-level defect segmentation, especially for refined processing of defect boundaries. Specifically, frequency domain transformation (DCT) converts image blocks from the spatial domain to the frequency domain. Low-frequency components represent the overall contour, and high-frequency components represent detail edges. This operation transforms the complex boundary optimization problem in space into an optimization problem of key coefficients in the frequency domain. Predicting key frequency domain coefficients involves the network directly learning and predicting how to adjust the frequency domain coefficients, thereby indirectly but more effectively controlling the overall shape and edge details of the generated mask. Inverse transform reconstructs the mask by inverse transforming the optimized frequency domain coefficients back to the spatial domain, generating a segmentation mask with clearer and more accurate boundaries.

[0080] The specific implementation of constructing the mask prediction branch based on frequency domain transformation in S2.3 includes:

[0081] S2.3.1: Mask Block Partitioning Process. The initial predicted mask is adaptively partitioned into blocks to prepare for frequency domain processing. Feature maps from a multi-scale feature fusion network are received, and the initial predicted mask is divided into K×K image blocks. This transforms the global mask prediction problem into a series of local block processing problems, reducing processing complexity. The block size is adaptively adjusted according to the defect scale. Specifically, a larger block size (8×8) is used for large-scale defects to ensure global semantic integrity. The block size is dynamically selected based on the defect scale (large blocks for large defects to preserve semantics, and small blocks for small defects to preserve details), reflecting the flexibility of the processing strategy. A smaller block size (4×4) is used for small-scale defects to retain detailed information. An overlapping partitioning strategy is used for edge regions to avoid boundary effects. An overlapping partitioning strategy is used for defect boundary regions to avoid obvious boundary gaps or artifacts during block stitching, ensuring the continuity of the final mask.

[0082] S2.3.2: Frequency Domain Feature Extraction. This involves converting spatial image data to the frequency domain and extracting key frequency components. A two-dimensional discrete cosine transform (DCT) is performed on each image block to convert spatial domain features to the frequency domain, resulting in a frequency domain coefficient matrix. DCT has energy concentration properties, focusing the main information on a small number of low-frequency coefficients. Specifically, this includes: applying DCT to each block to obtain the frequency domain coefficient matrix; extracting low-frequency components as the main semantic features, and retaining high-frequency components for edge refinement. Low-frequency components represent the overall outline and approximate shape of the image, serving as the main semantic features. High-frequency components represent the details, edges, and texture of the image, used for subsequent edge refinement. The frequency domain coefficients are scanned in a zigzag pattern to convert them into feature vectors; the DCT coefficient matrix is ​​then scanned in a zigzag order to convert it into a one-dimensional feature vector. This scanning order follows the distribution of DCT coefficient energy from low to high (i.e., from important to secondary).

[0083] S2.3.3: Frequency Domain Coefficient Regression Prediction. This method utilizes contextual information to predict the optimization amount of frequency domain coefficients that are crucial to mask quality. A lightweight regression network is used to predict key frequency domain coefficients. The lightweight regression network is constructed using a small, fully connected network to maintain low computational overhead. Specifically, a three-layer fully connected network is built as the regressor. The input is contextual features provided by a multi-scale feature fusion network. The regressor's input is semantically rich contextual features from the multi-scale feature fusion network (S2.2), which guide how to optimize the frequency domain coefficients. The output is the frequency domain coefficient offset to be optimized for each block. The network predicts the adjustment amount (offset) to the initial frequency domain coefficients, rather than directly predicting the coefficients themselves, which is a more learnable and stable strategy. The L2 loss function is used to supervise the coefficient prediction process; the regressor is optimized by minimizing the L2 distance between the predicted offset and the true offset.

[0084] S2.3.4: Mask Reconstruction and Fusion. The optimized frequency domain coefficients are converted back to the spatial domain, and a complete, refined mask is reconstructed. The optimized frequency domain coefficients are reconstructed into a spatial domain mask using inverse discrete cosine transform (Inco) . Specifically, this involves: performing an inverse zigzag scan on the predicted frequency domain coefficients to reconstruct the coefficient matrix; applying the one-dimensional coefficient offset output by the regressor to the initial coefficients; and reconstructing the two-dimensional frequency domain coefficient matrix using an inverse zigzag scan. An inverse DCT transform is applied to convert the frequency domain representation back to the spatial domain. The optimized frequency domain coefficients are then converted back to the spatial domain using an Inco) DCT transform to obtain optimized image blocks. A weighted fusion method is used to stitch the reconstruction results of each block into a complete mask. Bilinear interpolation is used to eliminate stitching gaps between blocks; bilinear interpolation is applied to the stitching points to smooth the transition and eliminate stitching gaps that may occur due to block processing, ensuring the overall smoothness of the mask.

[0085] S2.3.5: Multi-stage refinement processing, using a coarse-to-fine cascaded optimization strategy to gradually improve mask quality. Multiple frequency domain optimization modules are cascaded to achieve progressive mask refinement, optimizing different frequency components in stages. Specifically: Stage 1 (low frequency): Focuses on the overall shape and approximate range of the mask to ensure global semantic accuracy. Stage 2 (mid frequency): Optimizes the main outline and structure of the mask. Stage 3 (high frequency): Refines the edge details of defects, making boundaries clearer and sharper. Each stage shares a feature extraction network but uses independent regressors. All stages share the same backbone feature extraction network (S2.1, S2.2) to ensure feature consistency and save computational resources. Each stage uses an independent lightweight regressor specifically responsible for predicting the coefficient offset of the specific frequency range to be optimized in that stage.

[0086] The above steps define a fine-grained mask prediction method based on divide-and-conquer, transformation, optimization, and reconstruction: first, the mask is divided into blocks (S2.3.1), transformed to the frequency domain (S2.3.2), and the optimization quantity is predicted using semantic features (S2.3.3). Then, it is reconstructed back to the spatial domain (S2.3.4), and further refined through multiple stages (S2.3.5). This method transforms the boundary detail problem, which is difficult to optimize directly in the spatial domain, into a coefficient prediction problem that is easier to handle in the frequency domain, thereby achieving high-precision optimization of the defect segmentation mask boundary.

[0087] Specifically, the frequency domain feature extraction in S2.3.2 also includes a coefficient selection mechanism, which automatically selects important frequency domain components for processing through the learned attention weights. The multi-stage refinement in S2.3.5 adopts a residual learning strategy, in which each stage predicts the frequency domain coefficient residual relative to the previous stage, gradually approximating the frequency domain representation of the true mask.

[0088] S2.4: Design a multi-task joint optimization objective function, including: the classification loss adopts the focus loss function to solve the class imbalance problem; the localization loss adopts the GIoU loss to improve the bounding box regression accuracy; the masking loss adopts a weighted combination of Dice loss and binary cross-entropy; and the boundary refinement loss is specifically optimized to optimize the prediction accuracy of defect edge regions.

[0089] The design employs a multi-task joint optimization objective function, using a comprehensive loss function to simultaneously guide and optimize the three sub-tasks of instance segmentation (classification, localization, and segmentation), ensuring balanced network development and optimal overall performance. Specifically: Focus Loss (Classification): Addresses the severe imbalance between positive and negative samples (defect / normal) in power equipment defects by reducing the weight of numerous simple negative samples, making training more focused on difficult and defective samples. GIoU Loss (Localization): Compared to traditional IoU loss, it better optimizes the overlap between predicted and ground truth bounding boxes, providing effective gradients for optimization even when there is no overlap. Dice Loss + BCE Loss (Mask): Dice loss is particularly suitable for scenarios where foreground (defect) pixels are far fewer than background (normal) pixels, effectively improving segmentation performance; BCE (Binary Cross-Entropy) is a standard pixel-level classification loss. The combination of these two losses balances overall consistency and pixel accuracy. Boundary Refinement Loss: Specifically constrains the prediction error in defect edge regions, further improving the accuracy of segmentation boundaries.

[0090] The above steps systematically construct a high-performance instance segmentation network: S2.1 is responsible for extracting and fusing dual-modal features, S2.2 is responsible for aggregating multi-scale information, S2.3 is responsible for achieving refined segmentation output, and S2.4 ensures that all the aforementioned components can be trained and optimized collaboratively and effectively through a multi-task loss function.

[0091] Specifically, the implementation of the multi-task joint optimization objective function in S2.4 includes:

[0092] S2.4.1: The classification loss uses an adaptive focus loss function, the expression of which is:

[0093] ,

[0094] Where N is the number of samples and C is the total number of defect categories of power equipment (such as normal, cracks, corrosion, dirt, overheating, etc.). For the first Each sample is predicted to be of category [class]. The probability (from the output of the classification branch). For the corresponding real label (one-hot encoded), the value is either 0 or 1. category The positive sample weight coefficients are used to alleviate class imbalance. For negative sample weight coefficients; This is a dynamic focus parameter used to adjust the weights of easy and difficult samples. This is an adjustment term for the difference in probability distribution.

[0095] S2.4.2: The localization loss adopts the multi-scale GIoU loss function:

[0096] ,

[0097] in, The number of feature pyramid scales. This corresponds to the four scales of the feature pyramid, namely P2-P5. For the first Weights for each scale Generalized intersection-union ratio (CIU) is used to measure the predicted bounding box. With real frame The degree of overlap. , For prediction boxes With real frame The intersection-union ratio, i.e. , This is a smoothed L1 loss function used to regress the bounding box coordinate offset. These are the predicted bounding box parameters and the true bounding box parameters (such as center point coordinates, width and height) at the s-th scale, respectively.

[0098] S2.4.3: The mask loss adopts a multi-constraint composite loss function:

[0099] ,

[0100] in, For Dice's loss, For binary cross-entropy loss, For structural similarity loss, The loss is the total variational regularization loss.

[0101] Measure the probability of prediction With real labels The overlap is calculated as follows: The numerator is twice the intersection plus 1 (to avoid a denominator of 0), and the denominator is the sum of the squares of the predicted and actual graphs plus 1. It is more sensitive to small targets / sparse defects (such as small cracks in power equipment or broken insulators), avoiding missed detections due to class imbalance. H, W: The height and width of the mask.

[0102] The algorithm calculates the logarithmic loss between the predicted probability and the true label for each pixel, and averages the results to obtain the overall loss. This basic classification loss directly optimizes pixel-level prediction accuracy and is suitable for pixel-level annotation in defect detection (such as semantic segmentation). It is simple and efficient, but susceptible to class imbalance (e.g., when there are few defect pixels, negative samples dominate the loss).

[0103] Measure the prediction map ( ) and real picture ( Structural similarity. To calculate the mean of the predicted mask and the true mask, For variance, For covariance. Preserve spatial structural information of defects (such as crack direction in power equipment, distribution of contamination in insulators), avoiding models that only focus on pixel-level accuracy while ignoring structure. Robust to noise and brightness variations, suitable for defect detection in complex backgrounds.

[0104] It calculates the gradient changes of the predicted image in the horizontal and vertical directions, penalizes discontinuous pixel jumps and drastic changes in adjacent pixel values, and improves mask smoothness. It smooths the predicted image, reducing noise and artifacts (such as sensor noise in power equipment images and false defects caused by uneven lighting). It improves the visual consistency of the prediction results, making defect boundaries clearer.

[0105] S2.4.4: Boundary refinement loss uses an edge-aware loss function:

[0106] ,

[0107] in, This represents the number of pixels in the edge region (extracted via Canny edge detection). These are the gradients of the predicted mask and the real mask, respectively (used to capture edge details). For distance transformation weights; The curvature consistency loss penalizes the difference in curvature between the predicted mask and the true mask.

[0108] S2.4.5: The total loss function is:

[0109] ,

[0110] in, The Frobenius norm regularization term penalizes the model parameters. The magnitude of the loss function (L is the total number of network layers) is adjusted to prevent overfitting. Through the design of a multi-task loss function, the following are achieved: Classification task: Adaptive focus loss to address class imbalance; Localization task: Multi-scale GIoU loss to improve bounding box regression accuracy; Masking task: Dice+BCE+SSIM+TV composite loss to optimize segmentation quality; Edge refinement: Distance transform weighted loss + gradient + curvature loss to enhance boundary awareness; Total loss: Weighted summation + regularization to balance various tasks and control model complexity. All parameters are closely related to the characteristics of power equipment defects (such as multi-scale, edge sensitivity, and class imbalance), ensuring the model's effectiveness and robustness in real-world scenarios.

[0111] S3. Based on the collected multimodal image data, a generative adversarial strategy is adopted to generate diverse training samples, thereby improving the robustness of the instance segmentation network in complex environments. Through adversarial enhancement training, techniques such as generative adversarial networks (GANs) are used to expand the training dataset and generate diverse samples in complex environments (such as different lighting, weather, and occlusion conditions), thereby improving the generalization ability and robustness of the instance segmentation network constructed in step S2 in real complex scenes.

[0112] Specifically, in this embodiment, step S3 includes:

[0113] S3.1: Construct a training sample generation model based on Conditional Generative Adversarial Network (CGAN), whose objective function is expressed as:

[0114] ,

[0115] Where G is the generator, D is the discriminator, y is the conditional information (including defect category labels and environmental parameters), z is the latent space noise vector, and x is the real multimodal image pair. Generator G: The generator's task is to extract random noise from the latent space z and combine it with the conditional information y to generate realistic samples G(z|y). The generator's goal is to deceive the discriminator into believing that the generated samples are real. Discriminator D: The discriminator's task is to distinguish between real samples x and generated samples G(z|y). The discriminator's goal is to maximize the probability of correctly classifying real samples and generated samples.

[0116] Objective function V(D,G): The objective function consists of two parts: the first part This part represents the discriminator's expected probability of predicting the true sample x. The discriminator hopes this probability is close to 1, meaning the true sample is considered true. Part Two This part represents the discriminator's expected probability of predicting the generated sample G(z|y). The discriminator hopes this probability is close to 0, meaning the generated sample is considered fake.

[0117] S3.2: Design a multi-scale discriminator structure, including a global discriminator. and local discriminator Global discriminator The loss function is:

[0118] ,

[0119] The goal of this loss function is to maximize the discriminator's ability to distinguish between the real image x and the generated image G(z). Specifically, the discriminator aims to predict values ​​close to 1 for the real image and close to 0 for the generated image.

[0120] Local discriminator The loss function is:

[0121] ,

[0122] in, This represents a local patch randomly cropped from a real image. This represents a local patch corresponding to the generated image location. The loss function is similar to that of the global discriminator, but it operates on local regions of the image. Local discriminators are typically used to process local features of an image; for example, in image inpainting or image enhancement tasks, a local discriminator can focus on specific regions of the image to improve model performance. Multi-scale discriminator architectures typically include a global discriminator and local discriminators to process features or images at different scales, respectively. This structure is adopted in multiple domains, such as image inpainting, image recognition, and image enhancement tasks. By combining global and local discriminators, the model can better capture information at different scales, thereby improving model performance and robustness.

[0123] S3.3: Introduce Wasserstein distance and gradient penalty terms to optimize training stability. The objective function is:

[0124] ,

[0125] in, Indicates the generated sample The expected value of the discriminator output, i.e. the discriminator's evaluation of the generated sample. This represents the expected value of the discriminator's output for the real sample x, i.e., the discriminator's evaluation of the real sample. This is the gradient penalty term, used to ensure that the gradient of the discriminator satisfies the Lipschitz continuity condition at the linear interpolation points between the real and generated samples. The samples are obtained by linear interpolation between real samples and generated samples. λ It is the weight coefficient of the gradient penalty term, which is usually taken as a small constant (such as 10).

[0126] S3.4: Adopt a course learning strategy to control the generation difficulty and define a difficulty coefficient. How it changes with time step t:

[0127] ,

[0128] in, These represent the minimum and maximum noise injection levels, respectively. Total training steps Indicates at time step The difficulty level. Indicates the current time step. This represents the proportion of the current time step to the total number of training steps, with a value range of [0,1].

[0129] S3.5: Design a sample quality assessment mechanism to calculate the feature distance between generated samples and real samples:

[0130] ,

[0131] in, This represents a pre-trained feature extraction network used to extract feature representations of an image. For real samples, To generate samples, This is the Euclidean distance, used to calculate the difference between feature vectors. : Average feature distance, representing the feature difference between generated samples and real samples.

[0132] S3.6: Enhance the diversity of generated samples through adversarial training, focusing on generating defective samples under complex environments such as lighting changes, occlusion, and noise interference, thereby improving the robustness of the instance segmentation network in real-world scenarios.

[0133] S4. Based on the defect detection results output by the instance segmentation network, a graph neural network is used to analyze the correlation between defects and equipment topology and historical records, inferring the causal relationship and risk level of defects. Through defect correlation analysis, this method goes beyond single-point defect detection, utilizing a graph neural network (GNN) to analyze the correlation between currently detected defects and the topological connections of power equipment and historical maintenance records. The aim is to infer the causal relationships between defects (e.g., whether a certain defect is caused by another defect) and assess their comprehensive risk level, providing deeper insights for maintenance decisions.

[0134] Specifically, in this embodiment, the planning of a conflict-free cyclic inspection path in S4 that meets the range requirements includes:

[0135] The specific implementation of S4, which uses graph neural networks to analyze the correlation between defects and equipment topology and historical records to infer the causal relationship and risk level of defects, includes:

[0136] S4.1: Construct a knowledge graph for power equipment, specifically including: Node definition: power equipment, defect type, and historical maintenance records are defined as three types of nodes; Edge definition: equipment-defect edge (indicating that the equipment has a certain defect), equipment-maintenance edge (indicating the equipment's historical maintenance status), and defect-defect edge (indicating the causal relationship between defects); Node features: equipment node features include equipment type, commissioning time, and operating parameters; defect node features include defect type, severity, and location coordinates, derived from the output of the mask prediction branch in S2.3; maintenance node features include maintenance time, maintenance type, and maintenance effect.

[0137] S4.2: Knowledge Graph-Based Graph Neural Network Modeling: Graph Attention Network (GAT) is used for node representation learning, aggregating neighbor node information; the message passing function is: ,in, It is a node In the Layer feature representation, It is a node The set of neighboring nodes, It is the attention coefficient, representing the node For neighboring nodes Importance weight, It is the first The learnable weight matrix of the layer, It is an activation function (such as ReLU); node embedding representations are learned through a three-layer graph convolutional layer.

[0138] S4.3: Defect Causal Relationship Reasoning: Based on the learned node embeddings, calculate the correlation score between defective nodes: ,in, It is a defect node and The correlation score between them It is a defect node and The feature representation, where M is a learnable parameter matrix. It is an activation function that maps scores to the (0,1) interval; it establishes a defect propagation model to infer the causal relationship between fundamental defects and derived defects.

[0139] S4.4: Equipment Risk Level Assessment: This assessment considers the current severity of defects (from the classification loss output in S2.4), historical maintenance records, and causal relationships between defects; Risk scoring function: ,in, For equipment Risk score, The defect severity score (from the classification loss output of S2.4) represents the severity of the defect and is typically used to assess the direct impact of risk. To be with defects A set of defects with causal relationships. For risk propagation under causal relationships, it indicates the relationship with defects. The sum of the risk values ​​of other defects that are causally related reflects the propagation effect of risk. For equipment and The correlation score between them (Time factor) is a time decay function, which means that the maintenance effect decays over time, and represents the impact of the last maintenance time on risk, reflecting the dynamic change of the time factor on risk. This is the time since the last repair. To be with defects A set of defects with causal relationships. These are the weighting coefficients. ,in This is the attenuation coefficient.

[0140] S4.5: Output Defect Correlation Analysis Report: Generate a defect cause-and-effect diagram, marking the root cause and potential risks; output equipment risk level ranking to guide maintenance priority decisions; feed the analysis results back to the S5 interpretable alarm system to generate maintenance recommendations.

[0141] Specifically, the initial value of the defect-defect edge weight in S4.1 is calculated based on the spatial location relationship of the defects output in S2.3 and the classification confidence in S2.4. Specifically, the risk scoring function parameters in S4.4 are dynamically adjusted through reinforcement learning and continuously optimized in conjunction with the maintenance verification results from the feedback optimization mechanism in S7.

[0142] S5. Based on the defect correlation analysis results, generate interpretable alarm information including defect heatmaps, natural language reports, and maintenance suggestions. Through interpretable alarm generation, the analysis results of S4 are transformed into alarm information that is easy for maintenance personnel to understand and operate. It not only provides simple alarms but also generates reports containing defect heatmaps (visualization), natural language descriptions, and specific maintenance suggestions, enhancing the interpretability and action guidance of the results.

[0143] S6. Employing an edge-cloud collaborative architecture, lightweight models are deployed at the edge for real-time detection, while deep analysis and model optimization are performed in the cloud. A collaborative computing architecture is designed through edge-cloud collaborative deployment. Edge: Deploys lightweight models to handle initial real-time or near real-time detection, meeting low-latency requirements. Cloud: Responsible for receiving suspicious data from the edge or performing periodic deep analysis, model training, and optimization. This architecture balances computational efficiency, response speed, and system analysis capabilities.

[0144] Specifically, in this embodiment, the specific implementation of the edge-cloud collaborative architecture in S6 includes:

[0145] S6.1: Lightweight model deployment at the edge: Based on the lightweight depthwise separable convolutional structure in S2.1, a real-time detection model is deployed; the number of model parameters is compressed to 1 / 5 of the original model, and 8-bit integer quantization technology is adopted; it supports real-time processing of visible light and infrared dual-modal data streams, with inference latency of less than 50ms.

[0146] S6.2: Cloud-based Deep Analysis System: Receives suspicious samples and multimodal data uploaded from the edge; runs the complete S2 instance segmentation network for detailed analysis; combines the results of the graph neural network analysis in S4 to perform in-depth mining of defect correlations; S6.3: Collaborative Inference Mechanism: The edge performs preliminary detection, and results with a confidence level higher than 0.9 are directly output; suspicious samples with a confidence level between 0.7 and 0.9 are uploaded to the cloud for verification; samples with a confidence level lower than 0.7 are further trained and optimized by the edge model.

[0147] S6.4: Dynamic Model Update: The loss function in S2.4 is optimized in the cloud based on new samples. ,in, These are the updated model parameters. These are the new weights obtained after optimization using the new data in the cloud. These are the model parameters before the update. That is, the model weights in the current version on the cloud. is the learning rate. A hyperparameter that controls the step size for each parameter update. A value that is too large will cause oscillations, while a value that is too small will result in slow convergence. Gradient operator. Represents the gradient operator with respect to model parameters. Find the partial derivative. This is the total loss function. Specifically, the multi-task joint loss function defined in detail above is an overall measure of the difference between the model's predicted values ​​and the true values. This refers to new input data. It includes suspicious samples (such as image data) uploaded from the edge to the cloud. For new label data. (and) The corresponding real or pseudo-labels (such as defect category, bounding box, mask).

[0148] This formula describes the core process of stochastic gradient descent. The cloud receives new samples... Then, calculate the total loss of the current model on these samples. Then calculate the loss relative to each parameter of the model. gradient The gradient indicates the direction in which the loss function grows fastest. Therefore, we move along the opposite direction of the gradient, using... The parameters are updated according to the step size, thereby reducing the total loss and optimizing the model. The optimized model parameters are periodically distributed to the edge; incremental learning is supported to avoid catastrophic forgetting.

[0149] S6.5: Resource Adaptive Scheduling: Dynamically adjust upload strategy based on network bandwidth: upload raw multimodal data when bandwidth is sufficient; upload only feature vectors and key region images when bandwidth is limited; adjust edge computing frequency based on device power status.

[0150] S6.6: Security and Privacy Protection: Employs a federated learning framework, ensuring that raw data does not leave the edge; uses homomorphic encryption to protect feature data during transmission; and stores only anonymized analysis results in the cloud.

[0151] Specifically, the confidence threshold in S6.3 is dynamically adjusted based on the risk score in S4.4, with higher-risk defects using a lower threshold (0.6). The model update in S6.4 employs knowledge distillation technology, where the cloud-based teacher model guides the optimization of the edge-based student model, maintaining... minimize, This is the knowledge distillation loss. It measures the difference between the specific output probability distributions of the specific teacher model and the specific student model. This refers to the Kullback-Leibler divergence, a method for measuring the difference between two probability distributions. This refers to the output probability distribution of the teacher model. It indicates the class probability predicted by a large, high-precision model deployed in the cloud based on the input data (usually after adjusting for temperature parameters). (Scaled soft label) This represents the output probability distribution of the student model. It refers to the class probability predicted by the lightweight model deployed at the edge for the same input data.

[0152] This formula is the core of knowledge distillation. Its goal is not for the student model to directly learn real labels, but rather to make it mimic the teacher model's output in a more concrete and intelligent way. The teacher model, due to its powerful capabilities, produces soft labels containing rich information about inter-category relationships (e.g., "specific crack" and "specific damaged" might be probabilistically closer than "specific crack" and "specific normal"). By minimizing... The student model, while maintaining its lightweight nature, can learn the generalization ability of the teacher model, thus achieving better performance than direct training.

[0153] S7. Based on cloud-based analysis results and on-site maintenance verification data, establish a mechanism for comparing model predictions with actual results to continuously optimize the performance of the instance segmentation network and alarm system. Through closed-loop optimization feedback, establish a mechanism for continuous self-improvement. By comparing model prediction results with actual on-site maintenance verification data, continuously evaluate model performance, and use this feedback data to further optimize and adjust the instance segmentation model and alarm logic, enabling the system to continuously adapt to new situations and defects, and its performance to continuously evolve.

[0154] This method achieves comprehensive and intelligent management of power equipment defects from discovery, analysis, early warning to system evolution through a complete closed loop of multimodal perception (S1), intelligent core detection (S2, S3), deep correlation analysis (S4), humanized result presentation (S5), efficient system deployment (S6) and continuous self-optimization (S7).

[0155] Example 2

[0156] like Figure 2 As shown in the figure, this application provides an architecture diagram of a power equipment defect identification and alarm system based on deep learning, which is applied to the power equipment defect identification and alarm system based on deep learning as described in Embodiment 1. It includes a multimodal data acquisition and registration module 11, an instance segmentation network construction module 12, an adversarial enhancement training module 13, a defect correlation analysis module 14, an interpretable alarm generation module 15, a collaborative reasoning deployment module 16, and a closed-loop optimization feedback module 17.

[0157] The multimodal data acquisition and registration module 11 is used to simultaneously acquire visible light images and infrared thermal imaging images of the surface of power equipment, perform timestamp alignment and spatial registration processing, and generate registered multimodal image pairs.

[0158] The instance segmentation network construction module 12 is used to construct an instance segmentation network based on the registered multimodal image pairs, which includes a feature extraction network, a multi-scale feature fusion network, and a mask prediction branch. The feature extraction network adopts a lightweight convolutional structure and embeds a spatial attention mechanism. The multi-scale feature fusion network adopts a pyramid structure with global context enhancement and bidirectional feature aggregation. The mask prediction branch uses a frequency domain transformation method for boundary refinement.

[0159] The adversarial enhancement training module 13 is used to generate diverse training samples based on the collected multimodal image data using a generative adversarial strategy, thereby improving the robustness of the instance segmentation network in complex environments.

[0160] The defect association analysis module 14 is used to analyze the defect detection results output by the instance segmentation network, and uses graph neural network to analyze the association between defects and equipment topology and historical records, and infer the causal relationship and risk level of defects.

[0161] The interpretable alarm generation module 15 is used to generate interpretable alarm information, including defect heatmaps, natural language reports, and maintenance suggestions, based on the results of defect correlation analysis.

[0162] The collaborative reasoning deployment module 16 is used to deploy lightweight models at the edge for real-time detection and perform in-depth analysis and model optimization in the cloud using an edge-cloud collaborative architecture.

[0163] The closed-loop optimization feedback module 17 is used to establish a comparison mechanism between model predictions and actual results based on cloud analysis results and on-site maintenance verification data, and to continuously optimize the performance of the instance segmentation network and alarm system.

[0164] Figure 3 This is an electronic device provided in one embodiment of this application. For example... Figure 3 As shown, the electronic device includes at least the following components: processor 101 and memory 100, communication interface 103, and bus 102.

[0165] In this embodiment of the application, memory 100 is used to store executable instructions of processor 101, which, when configured to execute instructions, implements the method as described in the first aspect.

[0166] In embodiments of this application, a computer-readable storage medium includes instructions that instruct a device to perform the method as described in the first aspect. For example, the instructions instruct the device to perform... Figure 1 The method is shown in the process steps.

[0167] In one embodiment of this application, the program operating in the electronic device may be a program that controls a central processing unit (CPU) or similar device to achieve the functions of the above-described embodiments of the present invention (a program that enables the computer to function). Information processed by these systems is then temporarily stored in random access memory (RAM) during processing, and subsequently stored in various ROMs such as read-only memory (FlashROM) and hard disk drives (HDDs), and read, corrected, and written by the CPU as needed.

[0168] It should be noted that a portion of the electronic device described above can also be implemented using a computer. In this case, the program for implementing the control function can be recorded on a computer-readable recording medium, and the program recorded on the recording medium can be read into the computer and executed.

[0169] It should be noted that the computer mentioned here refers to a computer built into an electronic device, employing hardware including an operating system and peripheral devices. Furthermore, computer-readable recording media refers to removable media such as floppy disks, magneto-optical disks, ROMs, and CD-ROMs, as well as storage systems such as hard drives built into the computer.

[0170] Furthermore, computer-readable recording media can include: media that dynamically stores programs for short periods of time, such as communication lines used when transmitting programs via networks like the Internet or communication lines like telephone lines; and media that store programs for fixed periods of time, such as volatile memory inside a computer that serves as a server or client in this case. In addition, the aforementioned program can be a program used to implement the above-mentioned functions, or it can be a program that can implement the above-mentioned functions by combining them with programs already recorded in the computer.

[0171] Furthermore, the electronic device in the above embodiments can also be implemented as an assembly (system group) composed of multiple systems. Each system constituting the system group can possess some or all of the functions or functional blocks of the electronic device in the above embodiments. As a system group, it is sufficient to have all the functions or functional blocks of the electronic device.

[0172] Those skilled in the art should recognize that the above embodiments are only used to illustrate this application and are not intended to limit this application. Any appropriate changes and variations made to the above embodiments within the essential spirit and scope of this application fall within the scope of protection claimed in this application.

Claims

1. A method for defect identification and alarm of power equipment based on deep learning, characterized in that, Includes the following steps: S1. Simultaneously acquire visible light images and infrared thermal images of the surface of power equipment, perform timestamp alignment and spatial registration processing, and generate registered multimodal image pairs; S2. Based on the registered multimodal image pairs, an instance segmentation network is constructed, wherein the instance segmentation network includes: The feature extraction network employs a lightweight convolutional structure and embeds a spatial attention mechanism. A multi-scale feature fusion network employs a pyramid structure with global context enhancement and bidirectional feature aggregation; The mask prediction branch uses a frequency domain transformation method for boundary refinement. S2 includes: S2.1: Construct a dual-modal feature extraction network, using a parallel dual-branch structure to process visible light images and infrared images respectively. Each branch contains a backbone network based on depthwise separable convolutions and embeds a coordinate attention mechanism; a cross-modal feature interaction module is used to realize the information exchange and fusion of dual-modal features; S2.2: Construct an enhanced multi-scale feature pyramid network, adopt a bidirectional feature aggregation path from top to bottom and bottom to top, introduce a global context enhancement module, and use a feature reweighting mechanism to dynamically adjust the contribution of features at different scales; S2.3: Construct a mask prediction branch based on frequency domain transformation, divide the prediction mask into image blocks, extract frequency domain features by performing discrete cosine transform on each block, use a lightweight regression network to predict key frequency domain coefficients, and reconstruct a refined mask through inverse discrete cosine transform. S2.4: Design a multi-task joint optimization objective function, which includes a focus loss function for classification loss, a GIoU loss function for localization loss, a weighted combination of Dice loss and binary cross-entropy for masking loss, and a boundary refinement loss to optimize the prediction accuracy of defect edge regions. S3. Based on the collected multimodal image data, a generative adversarial strategy is adopted to generate diverse training samples, thereby improving the robustness of the instance segmentation network in complex environments; S4. Based on the defect detection results output by the instance segmentation network, a graph neural network is used to analyze the correlation between defects and equipment topology and historical records, and to infer the causal relationship and risk level of defects; S5. Based on the defect correlation analysis results, generate interpretable alarm information including defect heatmaps, natural language reports, and maintenance suggestions; S6. It adopts an edge-cloud collaborative architecture, deploying lightweight models at the edge for real-time detection, and performing in-depth analysis and model optimization in the cloud; S7. Based on cloud-based analysis results and on-site maintenance verification data, establish a comparison mechanism between model predictions and actual results to continuously optimize the performance of the instance segmentation network and alarm system.

2. The method for power equipment defect identification and alarm based on deep learning according to claim 1, characterized in that, S1 includes: A synchronous triggering device is used to control the visible light camera and the infrared thermal imager to acquire images synchronously. Timestamp matching is performed on the acquired dual-modal images to control the time deviation within milliseconds; A feature-point-based image registration algorithm is adopted to extract SIFT feature points from visible light and infrared images. The RANSAC algorithm is used to remove mismatched point pairs, and the affine transformation matrix is ​​calculated to achieve pixel-level spatial alignment. The registered dual-modal images are processed to unify the resolution, and the resolution of the infrared image is improved to be consistent with that of the visible light image by bicubic interpolation. Establish a multimodal image database to store registered image pairs and their corresponding acquisition time, device number, environmental parameter metadata.

3. The method for power equipment defect identification and alarm based on deep learning according to claim 1, characterized in that, The specific implementation of constructing the dual-modal feature extraction network in S2.1 includes: S2.1.1: Construct a parallel dual-branch input processing channel. The first branch receives the registered visible light image, and the second branch receives the registered infrared thermal imaging image. The two branches share the same network structure but the parameters are learned independently. S2.1.2: Construct a feature extraction backbone network based on depthwise separable convolution in each branch, using an inverted residual structure, which includes 1×1 convolutions for channel dimensionality upscaling, 3×3 depthwise separable convolutions for spatial feature extraction, 1×1 convolutions for channel dimensionality reduction, and a skip connection structure. S2.1.3: Embed a coordinate attention mechanism module after each inverted residual block. The feature descriptor is obtained by global pooling along the height and width directions, and a spatial attention weight map is generated to enhance the feature response of important spatial locations. S2.1.4: Construct a cross-modal feature interaction module, set a feature exchange layer at a specific network depth, realize intermodal feature interaction through a cross-attention mechanism, and dynamically adjust the fusion ratio of different modal features using an adaptive weight learning mechanism; S2.1.5: By cascading multiple feature extraction units consisting of depthwise separable convolutional blocks, coordinate attention mechanism modules, and cross-modal feature interaction modules, dual-modal features are extracted and fused step by step, and a fused feature map containing rich semantic information is output.

4. The method for power equipment defect identification and alarm based on deep learning according to claim 3, characterized in that, The specific implementation of constructing the enhanced multi-scale feature pyramid network in step S2.2 includes: S2.2.1: Construct a top-down feature propagation path, upsample the high-level features by 2 times, add the upsampled features to the features from the corresponding layers of the backbone network element by element, and use 3×3 convolution to smooth the fused features; S2.2.2: Construct a bottom-up feature enhancement path, downsample the bottom-level features by 2x, concatenate the downsampled features with the features from the corresponding level of the top-down path, and adjust the number of channels using a 1×1 convolution. S2.2.3: Introduce a global context enhancement module at each pyramid level. By performing global average pooling on the input feature map, global context information is extracted, and discriminative feature channels with enhanced channel attention weights are generated. S2.2.4: A feature reweighting mechanism is adopted to perform global average pooling on the output features of each level to obtain the scale description vector. The weight coefficients of each level are calculated through fully connected layers and softmax function, and the features at each level are weighted and fused. S2.2.5: Outputs an enhanced multi-scale feature pyramid, containing feature maps at four different scales {P2, P3, P4, P5}, providing multi-scale feature support for subsequent mask prediction.

5. The method for power equipment defect identification and alarm based on deep learning according to claim 4, characterized in that, The specific implementation of constructing the mask prediction branch based on frequency domain transformation in S2.3 includes: S2.3.1: Mask block segmentation processing: The predicted initial mask is divided into K×K image blocks. The block size is adaptively adjusted according to the defect scale. Larger block sizes are used for large-scale defects, and smaller block sizes are used for small-scale defects. An overlapping segmentation strategy is used for edge regions. S2.3.2: Frequency domain feature extraction. Perform a two-dimensional discrete cosine transform on each image block to obtain the frequency domain coefficient matrix. Extract low-frequency components as semantic features and retain high-frequency components for edge thinning. S2.3.3: Frequency domain coefficient regression prediction. A lightweight regression network is used to predict key frequency domain coefficients. A three-layer fully connected network is constructed as the regressor. The input is the context features provided by the multi-scale feature fusion network, and the output is the frequency domain coefficient offset that needs to be optimized for each block. S2.3.4: Mask reconstruction and fusion. The optimized frequency domain coefficients are reconstructed into a spatial domain mask through inverse discrete cosine transform. The reconstruction results of each block are spliced ​​into a complete mask using a weighted fusion method. Bilinear interpolation is used to eliminate the splicing gaps between blocks. S2.3.5: Multi-stage refinement processing. The mask is gradually refined by cascading multiple frequency domain optimization modules. The first stage processes low-frequency components, the second stage processes mid-frequency components, and the third stage processes high-frequency components. Each stage shares the feature extraction network but uses an independent regressor.

6. The method for power equipment defect identification and alarm based on deep learning according to claim 5, characterized in that, The specific implementation of the multi-task joint optimization objective function in S2.4 includes: S2.4.1: The classification loss uses an adaptive focus loss function. Its expression is: ， Where N is the number of samples, and C is the total number of power equipment defect categories. For the first Each sample is predicted to be of category [class]. The probability, For the corresponding real tags, category The positive sample weight coefficients, For negative sample weight coefficients, For dynamic focus parameters; S2.4.2: The localization loss adopts the multi-scale GIoU loss function. : ， in, The number of feature pyramid scales. For the first Weights for each scale For generalized intersection and comparison, To smooth the L1 loss function, These are the predicted bounding box parameters and the true bounding box parameters at the s-th scale, respectively. S2.4.3: The masking loss adopts a multi-constraint composite loss function. : ， in, For Dice's loss, For binary cross-entropy loss, For structural similarity loss, The total variational regularization loss; S2.4.4: Boundary refinement loss uses an edge-aware loss function: ， in, This represents the number of pixels in the edge region. These are the gradients of the predicted mask and the real mask, respectively. For distance transformation weights; The curvature consistency loss penalizes the difference in curvature between the predicted mask and the true mask, where H and W are the height and width of the mask. S2.4.5: The total loss function is : ， in, The Frobenius norm regularization term penalizes the model parameters. The range.

7. The method for power equipment defect identification and alarm based on deep learning according to claim 1, characterized in that, S4 includes: S4.1: Construct a knowledge graph for power equipment, taking power equipment, defect type, and historical maintenance record as three types of nodes, defining equipment-defect edges, equipment-maintenance edges, and defect-defect edges, and setting corresponding node features; S4.2: Knowledge graph-based graph neural network modeling, using graph attention network for node representation learning, aggregating neighbor node information, and learning node embedding representation through three layers of graph convolutional layers; S4.3: Defect causal relationship reasoning, calculate the correlation score between defect nodes based on the learned node embedding, establish a defect propagation model, and infer the causal relationship between the fundamental defect and the derived defect; S4.4: Equipment risk level assessment, which comprehensively considers the current severity of defects, historical maintenance, and causal relationships between defects, and calculates the equipment risk score through a risk scoring function, which takes into account the time decay function to represent the decay of maintenance effectiveness over time; S4.5: Output a defect correlation analysis report, generate a defect cause-and-effect diagram, mark the root causes and potential risks, output the risk level ranking of equipment, and feed the analysis results back to the interpretable alarm system to generate maintenance recommendations.

8. The method for power equipment defect identification and alarm based on deep learning according to claim 1, characterized in that, The specific implementation of the edge-cloud collaborative architecture in step S6 includes: S6.1: Lightweight model deployment at the edge, deploying a real-time detection model based on a lightweight depthwise separable convolutional structure, compressing the number of model parameters to one-fifth of the original model, using 8-bit integer quantization technology, and supporting real-time processing of visible light and infrared dual-modal data streams; S6.2: Cloud-based deep analysis system, which receives suspicious samples and multimodal data uploaded from the edge, runs a complete instance segmentation network for fine analysis, and combines graph neural network analysis results for in-depth mining of defect correlations; S6.3: Collaborative reasoning mechanism, the edge performs preliminary detection and processes samples according to confidence level: results with confidence level higher than the first threshold are directly output, suspicious samples with confidence level between the first and second thresholds are uploaded to the cloud for review, and samples with confidence level lower than the second threshold are trained and optimized by the edge model. S6.4: The model is dynamically updated. The cloud optimizes the total loss function based on new samples and regularly sends the optimized model parameters to the edge. This supports incremental learning and avoids catastrophic forgetting. S6.5: Resource adaptive scheduling, dynamically adjusts upload strategy based on network bandwidth, and adjusts edge computing frequency based on device power status; S6.6: Security and privacy protection. It adopts a federated learning framework to ensure that the original data does not leave the edge, uses homomorphic encryption technology to protect the feature data in transmission, and only stores the de-identified analysis results in the cloud.

9. A deep learning-based power equipment defect identification and alarm system, applied to the deep learning-based power equipment defect identification and alarm method as described in any one of claims 1 to 8, characterized in that, The system includes: The multimodal data acquisition and registration module is used to simultaneously acquire visible light images and infrared thermal imaging images of the surface of power equipment, perform timestamp alignment and spatial registration processing, and generate registered multimodal image pairs. The instance segmentation network construction module is used to construct an instance segmentation network based on the registered multimodal image pairs, which includes a feature extraction network, a multi-scale feature fusion network, and a mask prediction branch. The feature extraction network adopts a lightweight convolutional structure and embeds a spatial attention mechanism. The multi-scale feature fusion network adopts a pyramid structure with global context enhancement and bidirectional feature aggregation. The mask prediction branch uses a frequency domain transformation method for boundary refinement. The adversarial enhancement training module is used to generate diverse training samples based on the collected multimodal image data using a generative adversarial strategy, thereby improving the robustness of the instance segmentation network in complex environments. The defect correlation analysis module is used to analyze the defect detection results output by the instance segmentation network, and uses graph neural networks to analyze the correlation between defects and equipment topology and historical records, and infer the causal relationship and risk level of defects. The interpretable alarm generation module is used to generate interpretable alarm information, including defect heatmaps, natural language reports, and maintenance suggestions, based on the results of defect correlation analysis. The collaborative reasoning deployment module is used to deploy lightweight models at the edge for real-time detection and perform in-depth analysis and model optimization in the cloud using an edge-cloud collaborative architecture. The closed-loop optimization feedback module is used to establish a comparison mechanism between model predictions and actual results based on cloud analysis results and on-site maintenance verification data, and to continuously optimize the performance of the instance segmentation network and alarm system.

Images