Substation inspection image-oriented device identification and quality grade joint evaluation method

Abstract

The application discloses a substation inspection image-oriented equipment identification and quality grade joint evaluation method, which comprises inputting collected original substation images into an image identification and quality grade joint evaluation model, wherein the model comprises a detection perception image enhancement module, a target detection module, an ROI extraction module and an image quality evaluation module connected in sequence; the original substation images are subjected to enhancement processing by using the detection perception image enhancement module to obtain enhanced images; the enhanced images are subjected to target detection by using the target detection module to obtain power equipment category labels and ROI frame coordinate information; the corresponding equipment ROI images are obtained by cutting the original substation images according to the ROI frame coordinate information by using the ROI extraction module; and the equipment ROI images are subjected to no-reference image quality evaluation by using the image quality evaluation module to obtain quality grade labels; and the application realizes the improvement of equipment identification accuracy and the reliable output of image quality grades.

Description

Technical Field

[0001] This invention relates to the field of power equipment inspection and image processing technology, specifically to a method for equipment identification and quality level joint evaluation of substation inspection images. Background Technology

[0002] With the intelligent development of the power industry, the automation and intelligence levels of substation equipment are continuously improving. Their stable operation is crucial to the reliability of the power system, making equipment identification and condition monitoring key aspects of operation and maintenance. Deep learning has achieved remarkable results in the field of image recognition, but traditional models still have limitations when applied to substation equipment identification, and actual operation and maintenance also face multiple challenges.

[0003] As the core of the power grid, substations are prone to power outages due to thermal faults (such as overheating of joints and discharge of insulators), which account for more than half of all electrical equipment failures. This necessitates real-time, accurate equipment detection and condition assessment. Infrared thermal imaging has become the mainstream method for thermal fault detection due to its four advantages: no interference from electromagnetic noise and atmospheric attenuation. However, traditional inspection methods have significant problems: manual inspection is inefficient and has a high rate of missed detections due to the vast infrared image database; infrared images are susceptible to electromagnetic noise and atmospheric attenuation, leading to inaccurate temperature detection; existing technologies often separate image recognition and quality assessment, with image enhancement failing to address detection requirements, resulting in decreased accuracy; quality assessment relies on traditional indicators that are not adapted to the characteristics of equipment images, leading to a high rate of misjudgment; and multi-scale target detection in substations is difficult, with large-scale equipment (such as surge arresters) difficult to capture completely and small-scale equipment (such as insulators) difficult to identify effectively, while existing deep learning models lack robustness.

[0004] While current research on intelligent power inspection has made progress, shortcomings remain: detection models have poor adaptability, large models are difficult to deploy at edge devices, and lightweight models need to improve their accuracy in distinguishing small targets and similar equipment; image enhancement is disconnected from detection tasks, focusing only on visual effects, which can easily lead to blurred equipment edges and false detections; quality evaluation lacks scenario adaptability, traditional NR-IQA algorithms are not optimized for equipment thermal distribution and fault textures, and the classification of grades lacks differentiated standards, resulting in a high misjudgment rate.

[0005] In related technologies, patent application CN116228774A proposes a "quality assessment first, then identification" approach to address the problem of inaccurate defect identification caused by poor image quality during substation inspections. This approach uses a lightweight image quality assessment network to score each frame of the inspection image, selecting images with quality exceeding a threshold before inputting them into the defect identification model for detection. Essentially, this is a data filtering process, primarily focusing on quality screening before identification. The quality assessment and identification modules are independent at the feature level; the quality assessment is merely a pre-screening mechanism and does not provide any reverse optimization for the detection task, nor does it involve a collaborative optimization relationship between detection and quality assessment. Furthermore, this "quality assessment first, then identification" method directly discards low-quality images, failing to achieve real-time enhancement or adaptive optimization. Therefore, the system cannot intelligently compensate for "bad images" or reflect the quality level of the equipment image itself, resulting in unstable defect detection results. The thesis "Research on Defect Recognition Algorithm for Typical Equipment in Substations Based on Improved PP-YOLOE" by Sun Dedong, Master's Thesis, Liaoning University of Engineering and Technology, proposes an algorithm for defect recognition using Swin-Transformer semantic segmentation to assist in improving the PP-YOLOE model. The research focuses on how to improve defect recognition accuracy using deep learning models in complex substation scenarios. Therefore, this approach optimizes the accuracy and robustness of the PP-YOLOE model locally, such as introducing a Transformer module to enhance semantic perception and optimizing the feature fusion structure to improve small target detection. However, the overall approach remains within the scope of traditional detection, representing a single detection task enhancement within the PP-YOLOE framework. The goal is to improve the performance of the detection model itself, without addressing collaboration with other tasks (such as image quality assessment or image enhancement). Furthermore, while the improved detection model in this approach significantly improves defect recognition accuracy under visible light conditions, especially in multi-scale targets and small defects, its effectiveness is limited. However, this improvement still has three limitations: First, the computational load of the model increases significantly, which is not conducive to real-time deployment on edge devices such as inspection robots; second, the solution is only applicable to visible light images and is not robust enough to nighttime, strongly reflective, or occluded scenes; third, the output results are singular, limited to defect category and location, and cannot assess image quality or detection confidence reliability. Summary of the Invention

[0006] This invention aims to overcome the limitations of traditional substation inspection image enhancement, target detection, and quality evaluation being independent of each other, and to achieve improved equipment recognition accuracy and reliable output of image quality levels.

[0007] The present invention solves the above-mentioned technical problems through the following technical means:

[0008] A method for joint evaluation of equipment identification and quality level in substation inspection images is proposed, the method comprising:

[0009] The collected raw substation images are input into the image recognition and quality level joint evaluation model, which includes a detection and perception image enhancement module, a target detection module, a ROI extraction module and an image quality evaluation module connected in sequence.

[0010] The original substation image is enhanced using a detection and perception image enhancement module to obtain an enhanced image;

[0011] The target detection module is used to perform target detection on the enhanced image to obtain the category label of power equipment and the coordinate information of the ROI box.

[0012] The ROI extraction module is used to crop the corresponding equipment ROI image from the original substation image based on the ROI frame coordinate information;

[0013] The image quality assessment module is used to perform a no-reference image quality assessment on the ROI image of the device, and a quality level label is obtained.

[0014] Furthermore, the detection and perception image enhancement module includes a semantic segmentation network, a backbone network, and an enhancement network;

[0015] The enhancement process, which uses an enhancement module to enhance the original substation image to obtain an enhanced image, includes:

[0016] The original substation image is processed using a semantic segmentation network to generate a corresponding semantic mask to indicate whether a pixel belongs to the equipment area or the background area.

[0017] The original substation image is extracted using a backbone network to obtain a feature map of the original substation image.

[0018] Generate semantically sensitive features based on semantic masks and feature maps;

[0019] The semantically sensitive features are input into the U-Net-based enhancement network for end-to-end enhancement propagation to obtain the enhanced image.

[0020] Furthermore, the generation of semantically sensitive features based on semantic masks and feature maps includes:

[0021] After downsampling and global average pooling of the semantic mask, calculate the proportion vector of the device region in each channel of the feature map;

[0022] The proportion vector is mapped to channel attention and then multiplied channel by channel with the feature map to generate the semantically sensitive feature.

[0023] Furthermore, the original substation image includes visible light images and infrared images;

[0024] Accordingly, the enhanced image includes a visible light enhanced image and an infrared enhanced image.

[0025] Furthermore, the target detection module includes a backbone network, a multimodal feature dynamic fusion network, and a detection head;

[0026] The method of using a target detection module to perform target detection on the enhanced image to obtain power equipment category labels and ROI coordinate information includes:

[0027] Feature extraction was performed on the visible light enhanced image and the infrared enhanced image using a backbone network to obtain visible light features and infrared features;

[0028] A multimodal feature dynamic fusion network is used to fuse visible light features and infrared features to obtain a fused feature map;

[0029] The detection head processes the fused feature map to obtain the target center point position and width and height, and outputs the power equipment category label and ROI coordinate information.

[0030] Furthermore, the backbone network is a four-scale feature extraction network obtained by extending the CSPRepResNet structure in the PP-YOLOE network model;

[0031] The multimodal feature dynamic fusion network is a four-scale feature fusion network derived from the PAN network extension in the PP-YOLOE network model.

[0032] Furthermore, after obtaining the corresponding equipment ROI image from the original substation image using the ROI extraction module based on the ROI frame coordinate information, the method further includes:

[0033] Determine whether the confidence level of the device's ROI image is greater than or equal to the confidence threshold;

[0034] If so, the device ROI image is retained, and the semantic masking guidance mechanism is used to refine the device ROI image to obtain the refined ROI image;

[0035] If not, the saliency mask generation method based on intermediate feature maps is invoked to determine whether the device ROI image region contains a semantically salient region;

[0036] If so, the device ROI image is retained, and the semantic masking guidance mechanism is used to refine the device ROI image to obtain the refined ROI image;

[0037] If not, then remove the device ROI image.

[0038] Furthermore, the step of refining the device ROI image using a semantic masking guidance mechanism to obtain a refined ROI image includes:

[0039] The YOLO backbone feature network is used to extract the salient regions of power equipment;

[0040] A saliency prediction head is used to process salient regions to generate a saliency mask image;

[0041] Perform a logical AND operation between the mask image and the bounding box region of the device ROI image to obtain the refined ROI image.

[0042] Furthermore, the step of using the image quality assessment module to perform a no-reference image quality assessment on the device ROI image to obtain a quality level label includes:

[0043] The quality assessment module is used to perform a no-reference image quality assessment on the equipment ROI image to obtain a quality score;

[0044] Based on the quality thresholds for different types of power equipment, the quality score is determined as the corresponding quality grade label.

[0045] Furthermore, the quality evaluation module includes a semantic weighting layer, a feature extraction layer, a distortion branch, and a quality prediction branch; the step of using the quality evaluation module to perform no-reference image quality evaluation on the device ROI image to obtain a quality score includes:

[0046] Using a semantic weighting layer, pixels belonging to the device region in the device ROI image are assigned a weight of 1.5 times based on the saliency mask image, while pixels belonging to the background region are assigned a weight of 0.3 times.

[0047] The feature extraction layer is used to extract features from the device ROI image to obtain device features;

[0048] The distortion branch is used to process the device features to obtain the distortion feature vector;

[0049] The distorted feature vector is used as an auxiliary semantic for the quality prediction branch, and the device features are processed using the quality prediction branch to obtain the quality score.

[0050] Furthermore, the detection-sensory image enhancement module and the target detection module are jointly trained end-to-end, and the joint loss function used during the training of the detection-sensory image enhancement module is... for:

[0051]

[0052] In the formula, To enhance pixel-level reconstruction loss between the preceding and following images, Indicates the first Detection loss of similar equipment For instance-level feature consistency loss, , , For loss weighting coefficients, This represents the category difficulty weight.

[0053] Furthermore, the formula for calculating the category difficulty weight is as follows:

[0054]

[0055] In the formula, The momentum coefficient of the EMA. The category difficulty weights from the previous iteration. A lower value indicates that the category detection is more difficult. The larger, This represents an exponential function.

[0056] Furthermore, the formula for calculating the instance-level feature consistency loss is as follows:

[0057]

[0058] In the formula, Distance between similar instances Distance between heterogeneous instances This is the interval parameter in the instance-level feature consistency loss.

[0059] Furthermore, the detection-aware image enhancement module and the target detection module are jointly trained end-to-end, and the target detection module uses a loss function with fault perception weights during training. for:

[0060]

[0061] In the formula, For classifying losses, Predict the total loss for the bounding box. This is a set of abnormal regions marked in the infrared image. Weights for fault-related regions. For the first Fault-related losses in abnormal areas.

[0062] Furthermore, the formula for calculating the total loss of the bounding box prediction is as follows:

[0063]

[0064] In the formula, These are the weighting coefficients. The loss is used to measure the degree of overlap between the predicted bounding box and the ground truth bounding box. This represents the loss in accuracy of the coordinate regression used to constrain the bounding box.

[0065] Furthermore, this invention also proposes a joint evaluation system for equipment identification and quality level of substation inspection images, the system comprising:

[0066] The image input unit is used to input the acquired raw substation images into a pre-deployed image recognition and quality level joint evaluation model. The model includes a detection and perception image enhancement module, a target detection module, a ROI extraction module, and an image quality evaluation module connected in sequence.

[0067] The image recognition and quality level joint evaluation model is used to enhance the original substation image using a detection and perception image enhancement module to obtain an enhanced image; to perform target detection on the enhanced image using a target detection module to obtain power equipment category labels and ROI bounding box coordinate information; to use an ROI extraction module to crop the corresponding equipment ROI image from the original substation image based on the ROI bounding box coordinate information; and to use an image quality evaluation module to perform a no-reference image quality evaluation on the equipment ROI image to obtain a quality level label.

[0068] Furthermore, the present invention also proposes a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the method for joint evaluation of equipment identification and quality level based on substation inspection images as described above.

[0069] The advantages of this invention are:

[0070] (1) This invention constructs an end-to-end image recognition and quality level joint evaluation model using a detection and perception image enhancement module, a target detection module, a ROI extraction module, and an image quality evaluation module. The detection and perception image enhancement module enhances the original substation image, improving the accuracy and robustness of equipment detection. The target detection module performs target detection on the enhanced image, identifying various types of power equipment in the substation scene and outputting equipment category labels and ROI bounding box coordinate information. The ROI extraction module, based on the ROI bounding box coordinate information output by the target detection module, crops the equipment ROI image from the original substation image for subsequent quality evaluation, ensuring that the image quality evaluation is based on the actual acquired image and avoids... This invention avoids the problem of false high scores caused by directly evaluating enhanced images, ensuring the objectivity and accuracy of evaluation results. By combining ROI images extracted from original substation images with a no-reference quality evaluation module, objective quality scoring and grade output for equipment area images are achieved. Finally, the proposed method for joint evaluation of equipment detection and quality grades in substation inspection forms a collaborative closed-loop framework of detection, extraction, and evaluation. This method achieves high recognition accuracy under complex backgrounds and multi-scale target conditions, accurately focusing on equipment areas and quantifying quality grades, exhibiting strong scene adaptability and robustness. It can improve equipment recognition accuracy and provide reliable image quality grade output, offering strong support for intelligent and safe management of power operation and maintenance inspections.

[0071] (2) To address the quality issues such as blurriness, noise, and low contrast in substation inspection images, this invention designs a Detection-aware Enhancement Module (DAEM) through an end-to-end joint training method. By using a dynamic enhancement mechanism for detection task perception, it not only improves the visual quality of the image but also significantly enhances the downstream target detection model's ability to identify key areas of equipment (such as insulator skirts, switch contact points, and surge arrester connectors), thereby improving detection accuracy and robustness.

[0072] (3) The DAEM module combines semantic region guidance, difficulty weighting mechanism and instance-level feature consistency constraint to form a complete closed loop for detection perception image enhancement, which not only significantly improves the visual quality of the image, but also strengthens the robustness and accuracy of the downstream target detection model to the device region.

[0073] (4) Considering the interference of complex environment on detection effect, the Object DetectionModule (ODM) integrates semantic attention mechanism and online hard example mining strategy to jointly optimize detection robustness. First, it generates spatial attention map by using device semantic mask output by DAEM, and assigns higher feature weight to device region, thereby improving its response intensity in feature map. Second, it adopts OHEM (Online Hard Example Mining) strategy to dynamically select occluded samples with large loss in training for backpropagation, and strengthens the model's adaptability to adverse scenarios such as tree branch occlusion and background occlusion.

[0074] (5) The Raw Image ROI Extraction Module (RREM) designs a detection-guided ROI region extraction method, which directly extracts the device image region based on the original image and the target detection results. It achieves high-fidelity region extraction through semantic mask refinement and closed-loop fault tolerance mechanism, providing accurate, real and structurally complete input images for image quality evaluation. This solves the problem of image quality evaluation distortion after image enhancement and ensures the objectivity of the evaluation results based on real acquired images.

[0075] (6) In view of the problem that the sensitive features of different equipment are significantly different in equipment quality evaluation, the Image Quality Assessment Module (IQAM) of this invention designs a quality evaluation method driven by equipment-specific features. It adopts structure-texture dual-branch feature modeling and combines distortion type recognition and dynamic threshold division mechanism to achieve accurate quality scoring of key parts such as insulator skirt edges and transformer heat sinks. At the same time, it supports efficient operation on edge computing platforms, taking into account both real-time performance and accuracy.

[0076] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0077] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0078] Figure 1 This is a flowchart illustrating a method for joint evaluation of equipment identification and quality level in substation inspection images, as proposed in an embodiment of the present invention.

[0079] Figure 2 This is a schematic diagram of the framework of an image recognition and quality level joint evaluation model in one embodiment of the present invention;

[0080] Figure 3 This is a schematic diagram of the workflow of the detection and perception image enhancement module in one embodiment of the present invention;

[0081] Figure 4 This is a schematic diagram of the working process of the target detection module in one embodiment of the present invention;

[0082] Figure 5 This is a schematic diagram of the structure of Stage 1 in one embodiment of the present invention;

[0083] Figure 6 This is a schematic diagram of the structure of Stage 2 to Stage 4 in one embodiment of the present invention;

[0084] Figure 7 This is a schematic diagram of the workflow of the ROI extraction module in one embodiment of the present invention;

[0085] Figure 8 This is a schematic diagram of the workflow of the image quality evaluation module in one embodiment of the present invention;

[0086] Figure 9 This is a schematic diagram of the semantic weighting layer in one embodiment of the present invention;

[0087] Figure 10 This is a schematic diagram of the structure of a joint evaluation system for equipment identification and quality level based on substation inspection images, proposed in an embodiment of the present invention. Detailed Implementation

[0088] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0089] like Figures 1 to 2 As shown, the first embodiment of the present invention proposes a method for joint evaluation of equipment identification and quality level of substation inspection images, the method comprising the following steps:

[0090] S10. Input the collected raw substation images into the image recognition and quality level joint evaluation model. The model includes a detection and perception image enhancement module, a target detection module, a ROI extraction module and an image quality evaluation module connected in sequence.

[0091] It should be noted that the original substation images collected in this embodiment can be multimodal images of the substation, including visible light images and infrared images. By combining multimodal images, the detection failure problem of using a single mode in extreme scenarios (such as no light at night, high temperature and strong reflection) can be avoided.

[0092] S20. The original substation image is enhanced using the detection and perception image enhancement module to obtain an enhanced image;

[0093] S30. Use the target detection module to perform target detection on the enhanced image to obtain the power equipment category label and ROI coordinate information;

[0094] S40. Using the ROI extraction module, the corresponding equipment ROI image is obtained by cropping from the original substation image based on the ROI frame coordinate information;

[0095] S50. Use the image quality evaluation module to perform a no-reference image quality evaluation on the device ROI image and obtain a quality level label.

[0096] This embodiment enhances the original substation image using a detection and perception image enhancement module before inputting it into a target detection module to improve the accuracy and robustness of equipment detection. The target detection module then performs target detection on the enhanced image, identifying various power devices within the substation scene and outputting device category labels and ROI (Region of Interest) coordinate information. The ROI extraction module, based on the ROI coordinate information output by the target detection module, crops the equipment ROI images from the original substation image for subsequent quality evaluation. This ensures that the image quality evaluation is based on the actual acquired images, avoiding the false high scores that can result from directly evaluating the enhanced images, and guaranteeing the objectivity and accuracy of the evaluation results. The method avoids [the problem]; finally, by combining the ROI image extracted from the original substation image with the no-reference quality evaluation module, an objective quality score and grade output for the equipment area image is achieved. This embodiment integrates a multimodal adaptive target detection module, an original image ROI accurate extraction module, and an equipment-specific feature-driven image quality evaluation module on the basis of the detection and perception image enhancement module, forming a collaborative closed-loop framework of detection, extraction, and evaluation. This method has high recognition accuracy under complex backgrounds and multi-scale target conditions, and can significantly improve the accuracy and consistency of recognition and evaluation. It can accurately focus on the equipment area and quantify the quality grade, and has strong scene adaptability and robustness.

[0097] As a further preferred technical solution, this embodiment addresses the quality issues such as blurriness, noise, and low contrast in substation inspection images. It utilizes the Detection and Perception Image Enhancement Module (DAEM) to enhance the original substation images. Through a dynamic enhancement mechanism based on the detection task perception, it improves the visual quality of the images and enhances the downstream target detection model's ability to identify key areas of equipment (such as insulator skirts, switch contacts, and surge arrester connectors), thereby improving detection accuracy and robustness.

[0098] The detection-sensory image enhancement module DAEM includes a semantic segmentation network, a backbone network, and an enhancement network; specifically, as shown... Figure 3 As shown, step S20: using the enhancement module to enhance the original substation image to obtain an enhanced image, specifically includes the following steps:

[0099] S21. The original substation image is processed using a semantic segmentation network to generate a corresponding semantic mask to indicate whether a pixel belongs to the equipment area or the background area.

[0100] Specifically, this embodiment uses a lightweight semantic segmentation network (such as MobileSAM) to process the received raw substation image. The process is performed to generate the corresponding binary semantic mask. , The height of the image. The width of the image. It represents the real number space where the pixel values ​​of the image reside, and the intensity values ​​of all pixels are real numbers.

[0101] This semantic mask is used to guide the augmentation module (DAEM) to perform spatial differential augmentation: device regions are given higher augmentation weights, while background regions are weakened. It is a type of global semantic guidance signal, the purpose of which is to make the augmentation network focus on device regions and reduce background interference, thereby improving device saliency before feature extraction.

[0102] in Represents pixels This area belongs to the equipment area. Represents pixels This mask belongs to the background area and is used to guide the application of differentiated enhancement intensities to different areas during subsequent enhancement processes.

[0103] S22. Use a backbone network to extract features from the original substation image to obtain a feature map of the original substation image;

[0104] Specifically, in this embodiment, the backbone network (MobileNetV2) is used to process the input raw substation image. Feature extraction is performed to obtain the feature map of the original substation image. ,in, The height of the feature map, The width of the feature map. This represents the number of channels in the feature map.

[0105] S23. Generate semantically sensitive features based on semantic masks and feature maps;

[0106] It should be noted that this embodiment is based on semantic mask and feature map, and calculates semantically sensitive features through semantically sensitive channel attention.

[0107] S24. Input the semantically sensitive features into the U-Net-based enhancement network for end-to-end enhancement propagation to obtain the enhanced image.

[0108] Specifically, this embodiment will use semantically sensitive features. The input is an augmentation network based on the U-Net architecture, which performs end-to-end augmentation propagation. The U-Net consists of a symmetric encoder and decoder and includes skip connections to preserve spatial resolution information. The output is an augmented image. It offers higher contrast and clarity visually, while retaining key textures of the device.

[0109] As a further preferred technical solution, step S23: generating semantically sensitive features based on semantic masks and feature maps, specifically includes the following steps:

[0110] S231. After downsampling and global average pooling of the semantic mask, calculate the proportion vector of the device region in each channel of the feature map.

[0111] Specifically, in this embodiment, the semantic mask M is downsampled and subjected to global average pooling, and the computing device region is calculated in the feature map. Percentage vector for each channel As the channel attention vector, its calculation formula is:

[0112]

[0113] in, This is the mask after downsampling. Indicates the channel index. This represents the average weighted response value of the device region on the c-th feature channel.

[0114] S232. After mapping the proportion vector to channel attention, multiply it with the feature map channel by channel to generate the semantically sensitive feature.

[0115] Specifically, this embodiment uses a 1×1 convolution to convert the proportion vector Mapped to channel attention weights It is then multiplied channel-by-channel with the original feature map to generate semantically sensitive features. ,Right now:

[0116]

[0117] In the formula, This represents the semantic attention weight of the c-th channel.

[0118] It should be noted that this embodiment generates a semantic mask through semantic segmentation, assigns higher weights to pixels in the device region, and combines a channel attention mechanism to simultaneously enhance device features and suppress background features in both spatial and channel dimensions. Through this spatial and channel joint attention mechanism that strongly enhances the device region and weakly enhances the background region, semantic-level spatial guidance is introduced during the enhancement process, thereby achieving differentiated enhancement of detection perception. This makes device region features more prominent and reduces background interference, thereby improving detection accuracy and model robustness. It can enhance the discernibility of key device structures, reduce false detections and false negatives in the background, and improve the model's recognition stability and reliability in complex lighting and multi-target environments.

[0119] As a further preferred technical solution, to ensure the enhancement process is optimal for downstream detection tasks, the DAEM module employs end-to-end joint training with an object detection model (such as YOLOv5-Nano). The enhanced image... The input is fed into the object detection module to obtain the detection loss for each category. And the average precision (AP) of detection. Define the category difficulty weights. The key points are enhanced through dynamic adjustment, and the calculation formula is as follows:

[0120]

[0121] in, The momentum coefficient of the EMA. The category difficulty weights from the previous iteration. This represents an exponential function; the lower the AP value, the greater the difficulty of class detection. The larger the value, the more attention the network pays to that category.

[0122] Furthermore, to avoid introducing spurious features during the enhancement process, the DAEM module incorporates instance-level feature consistency constraints, ensuring that feature vectors from similar devices before and after enhancement are consistent. and Input the triplet loss function to calculate the distance between similar instances. Distance to heterogeneous instances Its loss is defined as:

[0123]

[0124] in, =0.3 is the margin parameter in the instance-level feature consistency loss, which is used to stipulate that the distance between heterogeneous instances must be at least a margin larger than the distance between homogeneous instances, thereby forming a clear inter-class separation in the feature space to ensure that the features of homogeneous devices are closer before and after enhancement, while maintaining distinction from heterogeneous devices.

[0125] It should be noted that feature vectors From feature map The global feature vector further extracted in the image is typically obtained through global average pooling or a fully connected layer. It is used to represent the semantic distribution features of the entire original image before enhancement. It is to enhance the image After being input into the aforementioned backbone network, an enhanced image is obtained. Feature map Then, pooling is used to obtain the enhanced feature vector. This is used to reflect the semantic features of the enhanced device region.

[0126] Distance between similar instances This represents the distance in feature space between the same device category (i.e., the same target before and after enhancement), for example, the difference between the feature vectors of the original and enhanced images of the same transformer, typically expressed using Euclidean distance. Norm calculation:

[0127]

[0128] This indicates the degree of feature difference before and after enhancement. The smaller the value, the better the consistency, indicating that the enhancement process did not change the device semantics.

[0129] Distance between different instances This represents the distance between different equipment categories (such as transformers and insulators) in the feature space. This distance is used to ensure that the enhanced features can still distinguish between different categories. Euclidean or cosine distances are also used, for example:

[0130]

[0131] in, It is a randomly selected feature vector (negative sample) of a different type of device.

[0132] Finally, the DAEM module is trained using a joint loss function, which is defined as follows:

[0133]

[0134] in, To enhance the pixel-level reconstruction loss of the before and after images (using the classic mean squared error loss MSE). Indicates the first Detection loss of similar equipment =1, =0.5, =0.1 represents the loss weight coefficient. Through end-to-end backpropagation optimization, the parameters of the enhancement network and the detection model are updated, outputting an enhanced image optimized for detection and perception. This serves as the input for the subsequent target detection module.

[0135] Specifically, pixel-level reconstruction loss To ensure the enhanced image With the original image To maintain consistency at the pixel level and avoid over-enhancement or the introduction of spurious features, the mean squared error loss (MSE) is defined as follows:

[0136]

[0137] in, To enhance the image at the pixel level The value; H represents the pixel value corresponding to the original substation image; H and W represent the height and width of the original substation image.

[0138] It should be noted that the pixel-level reconstruction loss is used to constrain the enhanced image to maintain consistency with the original substation image in terms of overall distribution of brightness, color, etc., and only enhances the texture details of the equipment area without changing the global semantics.

[0139] Specifically, detecting loss To maintain the recognizability of the enhanced image in the detection task, for each category The detection error is typically defined as the sum of the bounding box loss and the classification loss of the detection model, i.e.:

[0140]

[0141] Among them, bounding box loss The aforementioned composite method is used to constrain the localization error between the predicted bounding box and the ground truth bounding box:

[0142]

[0143] in, These are weighting coefficients used to control the proportion of the two types of loss (IoU loss and L1 loss) in the total loss; For use in measuring the first The loss factor is the degree of overlap between the predicted bounding box and the ground truth bounding box in similar devices. For use in constraining the first The loss of coordinate regression accuracy of the bounding box of the device class.

[0144] No. Classification loss of equipment Cross-entropy loss is typically used.

[0145]

[0146] in, These are real category labels. It is the probability predicted by the object detection model.

[0147] Combined with category difficulty weights The overall detection loss is written as:

[0148] .

[0149] In this embodiment, the DAEM module combines semantic region guidance, difficulty weighting mechanism and instance-level feature consistency constraint to form a complete closed loop for detection and perception image enhancement. This not only significantly improves the visual quality of the image, but also strengthens the robustness and accuracy of the downstream target detection model to the device region.

[0150] As a further preferred technical solution, this embodiment uses a DAEM module to enhance the visible light image and infrared image in the multimodal image of the substation, respectively, to obtain the visible light enhanced image and the infrared enhanced image.

[0151] As a further preferred technical solution, the target detection module (ODM) serves as the core functional module of the substation inspection image analysis system. It enables real-time and accurate identification and positioning of eight core equipment categories—transformers, surge arresters, insulators, switchgear, cable heads, instrument transformers, disconnect switches, and fuses—based on the substation multimodal images (covering visible and infrared images) processed by the detection and perception image enhancement module (DAEM). The module outputs detection results including equipment category, bounding box coordinates, and confidence level, providing fundamental data support for subsequent equipment quality level evaluation and fault early warning.

[0152] Specifically, the target detection module includes a backbone network, a multimodal feature dynamic fusion network, and a detection head; such as Figure 4 As shown, step S30: using the target detection module to perform target detection on the enhanced image to obtain the power equipment category label and ROI coordinate information, specifically includes the following steps:

[0153] S31. Use the backbone network to extract features from the visible light enhanced image and the infrared enhanced image respectively to obtain visible light features and infrared features;

[0154] Specifically, the object detection module ODM is based on the efficient single-stage detection model PP-YOLOE. Through multi-dimensional scene adaptation, it is made to deeply meet the needs of substation inspection: the backbone network adopts the PP-YOLOE CSPResNet structure, which reduces computational redundancy by using cross-stage partial connections (CSP) and balances computational efficiency and feature extraction accuracy by relying on the RepResBlock mechanism of "multi-path fusion in the training stage + reparameterization in the inference stage". For the characteristics of metal texture (such as transformer shell) and insulating material (such as insulator) of substation equipment, the backbone network is set to display four stages: Stem + Stage1–Stage4. In the four-stage feature extraction architecture, Stage1 is responsible for low-level edge and texture feature extraction, maintaining the original structure without adding branches; Stage2, Stage3, and Stage4 are all mid-to-high-level feature stages, and a parallel grouped convolutional path is added to the RepResBlock module within each stage. Compared to the CSPRepResNet structure of PP-YOLOE, this embodiment adds grouped convolutional branches in Stages 2-4 of the backbone network to enhance the extraction capability of high-frequency edge features of the device (such as heat sink gaps and insulator skirt edges), thereby improving the recognition of key details. Therefore, four feature maps are extracted from the backbone network (CSPRepResStage1–Stage4), corresponding to features from shallow to deep layers respectively:

[0155]

[0156] Compared with the traditional PP-YOLOE three-scale structure (128×128, 256×256, 512×512), this embodiment adds a 64×64 feature layer to better preserve high-frequency information such as equipment edges and fine textures, and realizes simultaneous modeling of large-size equipment and small-scale components, thereby adapting to different-sized targets such as transformers, insulators, and switchgear in substations.

[0157] Specifically, the Stage1 structure diagram is as follows: Figure 5As shown, Stage 1 receives the output feature map from the Stem layer and uses a Conv3×3 convolutional layer + BN normalization layer + SiLU for downsampling and preliminary feature extraction. Channel splitting is used to form shortcut branches and residual branches. The residual branch consists of RepResBlock×n1, and the shortcut branch consists of Conv1×1 convolutional layers. A multi-branch parallel structure is used during training, and the inference stage is reparameterized to a single 3×3 convolution. The outputs of the two branches are then concatenated and fused with a 1×1 convolution to obtain a shallow feature map, providing rich edge and texture information for subsequent stages. SiLU is a non-linear activation function layer, short for Sigmoid Linear Unit, also commonly known as the Swish activation function.

[0158] Stages 2 through 4 have the same structure. Taking Stage 2 as an example, ... Figure 6 As shown, Stage 2 takes the output of Stage 1 as input and continues the CSP framework, but adds a high-frequency enhancement branch to the main path. This high-frequency enhancement branch consists of sequentially connected GroupConv3×3 convolutions, a normalization layer (BN), and a SiLU layer. The main path contains RepResBlock×n2 stacks to extract mid-level semantic features, and achieves feature fusion with the high-frequency enhancement branch through multi-branch concatenation. Finally, the fused result is output via a 1×1 convolution. This structure enhances the model's ability to perceive high-frequency details such as metal edges and insulating umbrella skirts, providing a more stable feature input for mid-to-high-level semantic modeling.

[0159] It should be noted that, compared with the scheme proposed in the literature "Research on Defect Identification Algorithm for Typical Equipment in Substations Based on Improved PP-YOLOE", Sun Dedong, Master's Thesis of Liaoning University of Engineering and Technology, the literature achieves global semantic enhancement through the Transformer module, which improves semantic understanding but has high complexity; while this embodiment achieves high-frequency enhancement at the structural level by introducing GroupConv parallel branches in the mid-to-high-level stages, which strengthens local details and maintains lightweight efficiency, making it more suitable for edge deployment, thus achieving a better balance between detection accuracy and real-time performance in complex substation scenarios.

[0160] S32. Use a multimodal feature dynamic fusion network to fuse visible light features and infrared features to obtain a fused feature map;

[0161] Specifically, to improve the robustness of feature fusion and detection in multi-source inspection scenarios, this embodiment extends the original PP-YOLOE path aggregation network (PAN) both structurally and modally, constructing a multi-modal dynamic PAN (MD-PAN). The MD-PAN is a four-scale feature fusion network (64×64, 128×128, 256×256, 512×512) based on the PAN structure extension of the PP-YOLOE network model, adapting to the size differences of substation equipment: the 64×64 branch fuses shallow detail features through 4x upsampling, specifically handling small targets such as insulators and cable heads; the 512×512 branch retains deep semantic features for detecting large targets such as transformers and switchgear. The newly added modal adaptive fusion module can dynamically allocate feature weights between visible light and infrared images according to real-time lighting conditions, solving the detection failure problem of single modality in extreme scenarios (such as nighttime darkness, high temperature and strong reflection). The specific implementation process is as follows:

[0162] (1) Dynamic Fusion Unit (DFU)

[0163] For input multi-source images (visible light and infrared), this embodiment designs a multimodal feature dynamic fusion network for feature-level adaptive weighting. For each scale... Visible light characteristics infrared features First, modality fusion weights are generated using global average pooling and a two-layer perceptron:

[0164]

[0165] Then, linear fusion is performed based on modal importance to obtain the scale. Fusion feature map:

[0166]

[0167]

[0168] Among them, weight , It can dynamically adjust according to changes in ambient brightness, infrared contrast, etc., to achieve modal adaptation in day / night scenes.

[0169] (2) Dynamic Path Aggregation Network

[0170] Based on the fused feature maps, this embodiment dynamically expands the PAN structure by introducing an Adaptive Path Attention (APA) mechanism. This dynamically adjusts the information flow weights during feature transfer between upper and lower layers, preventing high-level semantic features from overshadowing lower-level details. The fusion process can be represented as follows:

[0171]

[0172] in, These are path adaptive weights learned by the convolutional gating layer (Sigmoid), used to adjust the ratio of top-down to bottom-up information flow.

[0173] This module allows the network to automatically adjust the fusion strength based on the importance of features at different levels, thereby preserving texture details in shallow layers and enhancing semantic consistency in deep layers, achieving dynamic feature aggregation at spatial scales.

[0174] (3) Multimodal fusion output and feature recalibration

[0175] The fused features are then weighted again in the output stage to generate the final multimodal aggregated features:

[0176]

[0177] in, These are adaptive fusion coefficients, jointly optimized during the training phase. This output feature incorporates multi-scale spatial hierarchical information and modal complementarity features, providing high-resolution and robust input for the subsequent ET-Head detection head.

[0178] It should be noted that the paper "Research on Defect Recognition Algorithm for Typical Equipment in Substations Based on Improved PP-YOLOE", Sun Dedong, Master's Thesis, Liaoning University of Engineering and Technology, only performs fixed feature splicing and static weighting on the original three-scale PAN structure, without considering the differences between different modal features. The static fusion method is prone to modal imbalance and feature drift under complex lighting or environmental changes, resulting in insufficient recognition ability for small targets and low-contrast areas. In contrast, this embodiment expands the PAN into a four-scale feature fusion structure at the structural level to adapt to the size differences of substation equipment, and introduces a multimodal dynamic fusion mechanism (DFU + APA) at the modal level. Through dynamic weight prediction, it achieves adaptive fusion of visible light and infrared features and path attention control, thereby realizing adaptive balance between modalities and dynamic aggregation of multi-scale features. This enables the model to maintain stable detection performance in various scenarios such as day and night, shadows, and reflections, improving the recognition accuracy of equipment of different sizes and detailed defects, improving the average detection accuracy, while maintaining lightweight and high real-time performance.

[0179] S33. Use the detection head to process the fused feature map to obtain the target center point position and width and height, and output the power equipment category label and ROI box coordinate information.

[0180] Specifically, the detection head continues the Efficient Task-aligned Head (ET-head) no-anchor-frame mechanism in PP-YOLOE, directly outputting bounding box information by predicting the target center point position and regressing the width and height, thus avoiding the matching error problem of the traditional anchor-frame mechanism in the multi-size environment of substation equipment.

[0181] As a further preferred technical solution, this embodiment combines the fault association loss function with bounding box optimization to train the target detection module. A weighted loss function of IoU and L1 is introduced into the bounding box prediction, i.e., the total loss for bounding box prediction:

[0182]

[0183] Where β is a weighting coefficient used to control the proportion of the two types of loss (IoU loss and L1 loss) in the total loss; Used to measure the degree of overlap between the predicted bounding box and the ground truth bounding box, reflecting the localization accuracy; The coordinate regression accuracy of the bounding box is used to constrain the bounding box, directly penalizing the difference between the predicted box and the true box coordinates.

[0184] This design improves the boundary fitting accuracy of regular rectangular devices.

[0185] Furthermore, the target detection module incorporates a differentiated training strategy targeting fault feature regions in infrared images, by setting weights for fault-related regions. Construct a loss function with fault-aware weights:

[0186]

[0187] in, For classifying losses, This is a set of abnormal regions marked in the infrared image. For the first Fault-related losses in each abnormal region By combining the semantic mask output by the DAEM module with the infrared temperature rise gradient, the model prioritizes optimizing equipment areas with high failure risk during training, thereby improving the system's response capability in anomaly warning tasks.

[0188] It should be noted that, considering the interference of complex environments on detection performance, the object detection module and the enhancement module are trained jointly. This integrates a semantic attention mechanism and an online hard example mining (OHEM) strategy to collaboratively optimize detection robustness. Specifically, the semantic attention mechanism provides spatial weights to guide the detection model to focus on key device regions; OHEM selects hard examples for reinforcement training at the sample level. Both work together to improve detection robustness and small object recognition capabilities. Specifically, a spatial attention map is first generated using the device semantic mask output by the DAEM module (i.e., the semantic mask generated by the semantic segmentation network is downsampled and converted into an attention vector, which is then mapped to generate a spatial attention map A(x,y), used to enhance device regions and suppress background regions at the feature layer to guide the object detection model to focus on device regions). Higher feature weights are assigned to device regions, thereby increasing their response intensity in the feature map. Secondly, the OHEM strategy is adopted (i.e., in each training iteration, the detection loss of all samples is calculated, and the "most difficult samples" (the samples with the highest loss values) are dynamically selected based on the magnitude of the loss, and only the gradients of these samples are used for backpropagation). This dynamically selects occluded samples with large losses during training for backpropagation, enhancing the model's adaptability to adverse scenarios such as tree branch occlusion and background occlusion. The training data covers common substation environments (sunny, rainy, and nighttime) and full-modal inputs. Equipment labels include three status labels: normal, overheated, and damaged, providing a data foundation for the promotion and application of this module in practical scenarios.

[0189] It should be noted that the specific implementation process of the OHEM strategy includes:

[0190] (1) Calculate sample loss: Calculate the total detection loss (including classification loss + bounding box loss) for each sample in the training batch.

[0191] (2) Sample sorting: sort by loss value from high to low to reflect the difficulty of each sample.

[0192] (3) Selecting difficult sample: Dynamically select the top k% of the samples with the largest loss (e.g., the top 30%) as difficult sample, and other samples do not participate in this round of backpropagation.

[0193] (4) Backpropagation optimization: only calculate the gradient and update the parameters for the selected difficult examples, so that the model focuses on learning difficult scenarios such as occlusion, noise, and high complexity.

[0194] Furthermore, to adapt to edge deployment scenarios, the ODM module also features a dynamic inference mechanism. The process involves: when the model is deployed on an edge device, the computing load of the current device is detected in real time; and adjustments are made dynamically based on computing power metrics: input resolution (e.g., reducing from 640×640 to 480×480), and forward path depth (e.g., skipping some PAN / FPN layers or reducing the number of feature fusion steps). When computing power is sufficient, full-path inference is used to achieve the highest accuracy; when computing power is limited, lightweight paths are automatically enabled to ensure real-time performance.

[0195] Then, while maintaining the integrity of the model backbone, the model parameters are compressed by combining quantization strategies. In the model export stage, a weight quantization strategy is adopted to convert floating-point weights into INT8 representation to reduce storage and computational overhead. At the same time, channel pruning is used to remove redundant convolution kernels. Based on the real-time computing power indicators provided by the actual terminal, the image input resolution and forward path depth are dynamically adjusted to achieve a dynamic balance between power consumption constraints and detection accuracy.

[0196] As a further preferred technical solution, the ROI extraction module in this embodiment aims to solve the problem of image quality evaluation distortion after image enhancement, ensuring the objectivity of the evaluation results based on real acquired images. To this end, the RREM module designs a detection-guided ROI region extraction method, which directly extracts the device image region based on the original image and target detection results, and achieves high-fidelity region extraction through semantic mask refinement and closed-loop fault tolerance mechanism, providing accurate, realistic, and structurally complete input images for image quality evaluation.

[0197] Specifically, such as Figure 7 As shown, after step S40: using the ROI extraction module to crop the corresponding equipment ROI image from the original substation image based on the ROI box coordinate information, the method further includes the following steps:

[0198] S41. Determine whether the confidence level of the device ROI image is greater than or equal to the confidence threshold. If yes, proceed to step S44; otherwise, proceed to step S42.

[0199] S42. Call the saliency mask generation method based on intermediate feature map to determine whether the device ROI image region contains a semantically salient region. If not, proceed to step S43; if so, proceed to step S44.

[0200] When handling low-confidence targets, the RREM module and ODM form a closed-loop fault-tolerance mechanism. If the confidence of the detection box in the ROI image is lower than a preset confidence threshold, a mask-assisted verification mechanism is triggered: the RREM module calls the Grad-CAM method based on intermediate feature maps to determine whether the detection box contains a semantically salient region. If not, the ROI is removed; if the salient score is higher than the set salient score threshold, the region is retained for subsequent quality evaluation. This strategy effectively avoids the problem of false positives or false negatives leading to "empty ROIs" or "non-device regions," enhancing the system's robustness.

[0201] This embodiment uses the Grad-CAM saliency mask generation method as a "fault-tolerant verification" mechanism triggered when the confidence level of the detection result is low. It generates a class activation saliency map by calculating the gradient response of the intermediate feature map of the detection network, which is used to determine whether the detection box region truly contains device semantics. If the saliency response is low, it indicates that the detection box may be a false detection or an empty region, and the ROI is removed; if the saliency score is high, it is considered that there is indeed device information in the region and it is retained. The goal of this process is to prevent non-device regions from being mistakenly sent to subsequent modules, ensuring the validity of the ROI.

[0202] S43. Remove equipment ROI images;

[0203] S44. Retain the device ROI image and use the semantic masking guidance mechanism to refine the device ROI image to obtain the refined ROI image.

[0204] It should be noted that the saliency map-guided segmentation method in step S44 refines the confirmed and retained device ROIs. It utilizes intermediate features extracted from the detection network backbone (such as the FPN layer) to generate pixel-level saliency maps using a lightweight saliency prediction head (3×3 convolution + Sigmoid). These maps are then logically ANDed with the ROI's bounding box, retaining only highly saliency device pixel regions while removing background and noise. The purpose of this step is to increase the proportion of device pixels within the ROI and reduce background interference, thereby providing clean device image input for subsequent quality assessment.

[0205] It should be noted that in this embodiment, the ROI extraction module receives the equipment bounding box information (including equipment category, coordinates, and confidence level) output by the target detection module ODM, and crops the corresponding area in the original substation image as the ROI image. By dynamically adapting to equipment characteristics, it ensures that the equipment area and its key context (such as insulation gaps and support structures) are completely preserved, taking into account both structural coverage integrity and semantic continuity.

[0206] As a further preferred technical solution, in step S44, the device ROI image is refined using a semantic masking guidance mechanism to obtain a refined ROI image, specifically including the following steps:

[0207] The YOLO backbone feature network is used to extract the salient regions of power equipment;

[0208] A saliency prediction head is used to process salient regions to generate a saliency mask image;

[0209] Perform a logical AND operation between the mask image and the bounding box region of the device ROI image to obtain the refined ROI image.

[0210] It should be noted that, to improve the purity of the device region in the ROI, the RREM module in this embodiment incorporates a semantic mask guidance mechanism. Referring to saliency map-guided segmentation methods in unsupervised anomaly detection, the RREM module extracts salient regions of the device based on YOLO backbone feature maps (such as FPN), and generates a saliency mask map M'(x,y) through a lightweight saliency prediction head (3×3 convolution + Sigmoid). This mask map is a grayscale image obtained through feature saliency learning, reflecting the model's attention intensity to different pixels within the ROI. A logical AND operation is performed between this mask and the bounding box region to refine and matte the original ROI image. This mechanism can increase the proportion of device pixels in the total ROI pixels, reduce background interference, and improve the accuracy of quality assessment.

[0211] As a further optimized technical solution, the Image Quality Assessment (IQAM) module is designed around the real-world application scenarios of substation inspection images. It fully focuses on the core concept that the quality of equipment areas determines the effectiveness of the assessment, aiming to solve the problems of traditional IQA methods being sensitive to background areas and unable to distinguish differences in equipment structural quality. The IQAM module constructs a dedicated solution for industrial scenarios through structural reconstruction, target migration, and deployment adaptation.

[0212] Specifically, step S50: using the image quality assessment module to perform a no-reference image quality assessment on the device ROI image to obtain a quality level label, specifically includes the following steps:

[0213] S51. Use the quality evaluation module to perform a no-reference image quality evaluation on the equipment ROI image and obtain a quality score;

[0214] S52. Based on the quality threshold of different types of power equipment, the quality score is determined as the corresponding quality grade label.

[0215] Specifically, regarding the quality grading mechanism, to achieve standardized scoring, this embodiment normalizes the quality scores, standardizing the final score to a range of 0 to 1, ensuring consistency across different image and device types. The quality scores are categorized into four levels: Excellent, Good, Average, and Poor, based on preset or dynamically adjusted thresholds. Specifically: Excellent score > 0.90; Good score > 0.75; Average score > 0.50; Poor score ≤ 0.50.

[0216] Furthermore, such as Figure 8 As shown, the quality evaluation module includes a semantic weighting layer, a feature extraction layer, a distortion branch, and a quality prediction branch; step S51: using the quality evaluation module to perform a no-reference image quality evaluation on the device ROI image to obtain a quality score, specifically includes the following steps:

[0217] S511. Using a semantic weighting layer, pixels belonging to the device region in the device ROI image are assigned a weight of 1.5 times based on the saliency mask image, and pixels belonging to the background region are assigned a weight of 0.3 times.

[0218] Specifically, the image quality assessment module receives the device region image and its semantic mask output by the ROI extraction module. A weighted local quality assessment mechanism is introduced, and the IQAM module further upgrades this idea to semantically guided device region enhancement: by assigning a 1.5× weight to the device region pixels in the semantic mask and only a 0.3× weight to the background region, the influence of interfering regions such as weeds and sky on the scoring process is effectively suppressed. Compared to the mean-blurred weighting processing strategy in general images, this method achieves a device-level spatial perception control mechanism for the first time, ensuring that the evaluation results truly reflect the device quality ontology.

[0219] Furthermore, such as Figure 9 As shown, the semantic weighting layer consists of four parts: mask alignment, weight mapping, spatial weighting, and normalization.

[0220] First, the input semantic mask ( Downsampling to the feature using bilinear interpolation Figure 1 The spatial dimensions are obtained ( Subsequently, a lightweight smoothing layer (3×3 convolution + Sigmoid) can be used to continuum the mask, resulting in ( If smoothness is not required, then ( ).

[0221] Next, a weighted graph (W(x,y)) is generated based on the mask values, and its calculation formula is as follows:

[0222]

[0223] That is, the weight of the device area is 1.5, and the weight of the background area is 0.3.

[0224] The weight map is then broadcast along the channel dimension to the feature map ( Perform pixel-by-pixel weighted operation:

[0225]

[0226] This achieves the enhancement of the device area features and the suppression of background features.

[0227] Finally, LayerNorm can be optionally used to apply weighted features ( Normalization is performed to stabilize the amplitude distribution, and finally the semantically weighted feature map is output, providing more focused and purer device feature input for subsequent structural and texture branches.

[0228] S512. Use the feature extraction layer to extract features from the device ROI image to obtain device features;

[0229] It should be noted that the weighted result is fed into a subsequent feature extraction network for structural and texture feature modeling, thereby improving the accuracy of quality assessment and the focus on device regions.

[0230] Specifically, the image quality assessment module is based on a lightweight, improved VGG16 architecture, introducing the GDN activation function and a structure-texture separation strategy. At the network structure level, GDN is introduced to replace ReLU, achieving dynamic normalization compression and reducing the number of parameters without sacrificing feature expressiveness, thus meeting the requirements of edge computing deployment. The feature channels are divided into two sub-branches: the structure branch focuses on edge and shape continuity (e.g., the alignment of transformer heat sinks), while the texture branch focuses on surface detail integrity (e.g., dirt accumulation and damage on insulator skirts). The structure and texture branches are two sub-branches after the main feature extraction; the former models the edge and shape continuity of the equipment, while the latter models surface details and texture integrity. The fusion of these two branches yields the equipment features, which are then processed by the distortion branch and the quality prediction branch, respectively. This feature modeling approach, which targets the material properties of the equipment (e.g., metal casing and insulation material), is superior to shallow texture modeling for general images.

[0231] S513. Process the device features using the distortion branch to obtain the distortion feature vector;

[0232] It should be noted that this embodiment identifies the type or degree of distortion in the image through distortion branches and outputs a distortion feature vector to provide auxiliary semantics for quality prediction.

[0233] S514. Use the distorted feature vector as an auxiliary semantic for the quality prediction branch, and use the quality prediction branch to process the device features to obtain the quality score.

[0234] The distortion branch is used to perform convolution and fully connected mapping on the fused device features to extract distortion-aware features; the quality prediction branch performs nonlinear regression prediction based on the distortion features and achieves adaptive quality score output through Sigmoid activation and dynamic thresholding layer. Figure 8 The dynamic threshold layer in the model is a component of the quality prediction branch.

[0235] Specifically, the distortion branch includes a 3×3 convolutional layer and one or two fully connected layers. The convolutional layer is followed by batch normalization (BN) and SiLU activation function to capture local distortion patterns. The fully connected layer further compresses the feature dimension through nonlinear mapping and outputs a distortion feature vector.

[0236] The quality prediction branch receives output features from the distortion branch and combines them with spatial features enhanced by the semantic weighting layer to construct a globally aware quality prediction path. This branch consists of two fully connected layers: the first layer uses ReLU activation for non-linear feature reconstruction; the second layer uses Sigmoid activation to map the output to the [0,1] interval, forming a normalized quality score.

[0237] To enhance the adaptability of the prediction, this embodiment sets a dynamic threshold layer at the end of the quality prediction branch. The judgment threshold is dynamically adjusted according to the device category or detection confidence level, thereby taking into account the differences in quality sensitivity among different device types. The final output quality score serves as the basis for determining the device image quality level.

[0238] Specifically, in terms of multi-task modeling, the IQAM module constructs a dual-task structure for distortion type identification and quality level prediction. The distortion branch not only covers basic distortion types such as blur, noise, and overexposure, but also extends to include substation-specific fault features such as hot spot diffusion and dirt accumulation occlusion, effectively enhancing the system's ability to identify scene-specific noise. The quality prediction branch adopts the DCT-based GGD texture perturbation modeling method and introduces local structural gradient features to form a joint modeling mechanism of texture perturbation and edge collapse, which more closely reflects the actual degradation performance of substation equipment.

[0239] It should be noted that this embodiment proposes a collaborative closed-loop network architecture for detection, extraction, and evaluation. It deeply integrates the modules for image enhancement, target detection, precise ROI extraction, and image quality evaluation, retaining the multi-scale feature extraction capabilities of the detection network while introducing a region-focusing mechanism using device semantic masks. This ensures the model focuses on the core equipment areas of the substation, significantly improving the accuracy and consistency of identification and evaluation. Addressing the issues of missed detection of small targets and strong background interference in multimodal inspection images of substations, a detection strategy of dynamic multimodal feature fusion and device type adaptation is proposed. Four-scale feature fusion is extended into the PAN structure, and a modal weight adjustment mechanism is introduced to effectively enhance the complementary capabilities of visible light and infrared light in different scenarios, achieving high-precision detection and ROI localization in complex environments. To address the significant differences in sensitive features among different devices in equipment quality evaluation, a device-specific feature-driven quality evaluation method is designed. This method employs structure-texture dual-branch feature modeling and combines distortion type recognition and dynamic thresholding mechanisms to achieve accurate quality scoring for key components such as insulator skirt edges and transformer heat sinks. It also supports efficient operation on edge computing platforms, balancing real-time performance and accuracy.

[0240] In addition, such as Figure 10 As shown, the second embodiment of the present invention also proposes a joint evaluation system for equipment identification and quality level of substation inspection images, the system comprising:

[0241] The image input unit 10 is used to input the acquired raw substation images into a pre-deployed image recognition and quality level joint evaluation model. The model includes a detection and perception image enhancement module, a target detection module, a ROI extraction module, and an image quality evaluation module connected in sequence.

[0242] The image recognition and quality level joint evaluation model 20 is used to enhance the original substation image using the detection and perception image enhancement module to obtain the enhanced image; to perform target detection on the enhanced image using the target detection module to obtain the power equipment category label and ROI bounding box coordinate information; to use the ROI extraction module to crop the corresponding equipment ROI image from the original substation image based on the ROI bounding box coordinate information; and to use the image quality evaluation module to perform a no-reference image quality evaluation on the equipment ROI image to obtain the quality level label.

[0243] It should be noted that other embodiments or specific implementation methods of the image recognition and quality level joint evaluation model in the equipment recognition and quality level joint evaluation system for substation inspection images described in this invention can refer to the above-mentioned method embodiments, and will not be repeated here.

[0244] Furthermore, the third embodiment of the present invention also proposes a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, is used to implement the equipment identification and quality level joint evaluation method for substation inspection images as described in the first embodiment above.

[0245] It should be noted that the computer-readable medium disclosed in this embodiment may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, and portable compact disk read-only memory (CD-ROM). ROM, optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0246] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform a zero-sample image anomaly detection method according to the above embodiments.

[0247] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server.

[0248] In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0249] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0250] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0251] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" or "several" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0252] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A method for joint evaluation of equipment identification and quality level in substation inspection images, characterized in that, include: The acquired raw substation images are input into the image recognition and quality level joint evaluation model. The model includes a detection and perception image enhancement module, a target detection module, a ROI extraction module and an image quality evaluation module connected in sequence. The detection and perception image enhancement module includes a semantic segmentation network, a backbone network and an enhancement network. An enhanced image is obtained by using a detection and perception image enhancement module to enhance the original substation image. This includes processing the original substation image using a semantic segmentation network to generate a corresponding semantic mask to indicate whether a pixel belongs to the equipment area or the background area; extracting features from the original substation image using a backbone network to obtain a feature map of the original substation image; and generating semantically sensitive features based on the semantic mask and the feature map. The semantically sensitive features are input into an enhancement network based on the U-Net architecture for end-to-end enhancement propagation to obtain an enhanced image. The step of generating semantically sensitive features based on a semantic mask and a feature map includes: performing global average pooling based on the feature map and a downsampled semantic mask to calculate the proportion vector of the device region in each channel of the feature map; mapping the proportion vector to channel attention weights and multiplying it channel-by-channel with the feature map to generate the semantically sensitive features. The target detection module is used to perform target detection on the enhanced image to obtain the category label of power equipment and the coordinate information of the ROI box. The ROI extraction module is used to crop the corresponding equipment ROI image from the original substation image based on the ROI frame coordinate information; The image quality assessment module is used to perform a no-reference image quality assessment on the ROI image of the device, and a quality level label is obtained.

2. The method for joint evaluation of equipment identification and quality level based on substation inspection images as described in claim 1, characterized in that, The original substation images include visible light images and infrared images; Accordingly, the enhanced image includes a visible light enhanced image and an infrared enhanced image.

3. The method for joint evaluation of equipment identification and quality level based on substation inspection images as described in claim 1, characterized in that, The target detection module includes a backbone network, a multimodal feature dynamic fusion network, and a detection head; The method of using a target detection module to perform target detection on the enhanced image to obtain power equipment category labels and ROI coordinate information includes: Feature extraction was performed on the visible light enhanced image and the infrared enhanced image using a backbone network to obtain visible light features and infrared features; A multimodal feature dynamic fusion network is used to fuse visible light features and infrared features to obtain a fused feature map; The detection head processes the fused feature map to obtain the target center point position and width and height, and outputs the power equipment category label and ROI coordinate information.

4. The method for joint evaluation of equipment identification and quality level based on substation inspection images as described in claim 3, characterized in that, The backbone network is a four-scale feature extraction network obtained by extending the CSPRepResNet structure in the PP-YOLOE network model. The multimodal feature dynamic fusion network is a four-scale feature fusion network derived from the PAN network extension in the PP-YOLOE network model.

5. The method for joint evaluation of equipment identification and quality level based on substation inspection images as described in claim 1, characterized in that, After the ROI extraction module extracts the corresponding equipment ROI image from the original substation image based on the ROI frame coordinate information, the method further includes: Determine whether the confidence level of the device's ROI image is greater than or equal to the confidence threshold; If so, the device ROI image is retained, and the semantic masking guidance mechanism is used to refine the device ROI image to obtain the refined ROI image; If not, the saliency mask generation method based on intermediate feature maps is invoked to determine whether the device ROI image region contains a semantically salient region; If so, the device ROI image is retained, and the semantic masking guidance mechanism is used to refine the device ROI image to obtain the refined ROI image; If not, then remove the device ROI image.

6. The method for joint evaluation of equipment identification and quality level based on substation inspection images as described in claim 5, characterized in that, The step of refining the device ROI image using a semantic masking guidance mechanism to obtain a refined ROI image includes: The YOLO backbone feature network is used to extract the salient regions of power equipment; A saliency prediction head is used to process salient regions to generate a saliency mask image; Perform a logical AND operation between the mask image and the bounding box region of the device ROI image to obtain the refined ROI image.

7. The method for joint evaluation of equipment identification and quality level based on substation inspection images as described in claim 1, characterized in that, The image quality assessment module performs a no-reference image quality assessment on the device ROI image to obtain a quality level label, including: The quality assessment module is used to perform a no-reference image quality assessment on the equipment ROI image to obtain a quality score; Based on the quality thresholds for different types of power equipment, the quality score is determined as the corresponding quality grade label.

8. The method for joint evaluation of equipment identification and quality level based on substation inspection images as described in claim 7, characterized in that, The quality evaluation module includes a semantic weighting layer, a feature extraction layer, a distortion branch, and a quality prediction branch; the process of using the quality evaluation module to perform no-reference image quality evaluation on the device ROI image to obtain a quality score includes: Using a semantic weighting layer, pixels belonging to the device region in the device ROI image are assigned a weight of 1.5 times based on the saliency mask image, while pixels belonging to the background region are assigned a weight of 0.3 times. The feature extraction layer is used to extract features from the device ROI image to obtain device features; The distortion branch is used to process the device features to obtain the distortion feature vector; The distorted feature vector is used as an auxiliary semantic for the quality prediction branch, and the device features are processed using the quality prediction branch to obtain the quality score.

9. The method for joint evaluation of equipment identification and quality level based on substation inspection images as described in claim 1, characterized in that, The detection and perception image enhancement module and the target detection module are jointly trained end-to-end. The joint loss function used during the training of the detection and perception image enhancement module is... for: In the formula, To enhance pixel-level reconstruction loss between the preceding and following images, Indicates the first Detection loss of similar equipment For instance-level feature consistency loss, , , For loss weighting coefficients, This represents the category difficulty weight.

10. The method for joint evaluation of equipment identification and quality level based on substation inspection images as described in claim 9, characterized in that, The formula for calculating the category difficulty weight is as follows: In the formula, The momentum coefficient of the EMA. The category difficulty weights from the previous iteration. A lower value indicates that the category detection is more difficult. The larger, This represents an exponential function.

11. The method for joint evaluation of equipment identification and quality level based on substation inspection images as described in claim 9, characterized in that, The formula for calculating the instance-level feature consistency loss is as follows: In the formula, Distance between similar instances Distance between heterogeneous instances This is the interval parameter in the instance-level feature consistency loss.

12. The method for joint evaluation of equipment identification and quality level based on substation inspection images as described in claim 1, characterized in that, The detection and perception image enhancement module and the target detection module are jointly trained end-to-end. The target detection module uses a loss function with fault perception weights during training. for: In the formula, For classifying losses, Predict the total loss for the bounding box. This is a set of abnormal regions marked in the infrared image. Weights for fault-related regions. For the first Fault-related losses in abnormal areas.

13. The method for joint evaluation of equipment identification and quality level based on substation inspection images as described in claim 12, characterized in that, The formula for calculating the total loss of bounding box prediction is as follows: In the formula, These are the weighting coefficients. The loss is used to measure the degree of overlap between the predicted bounding box and the ground truth bounding box. This represents the loss in accuracy of the coordinate regression used to constrain the bounding box.

14. A joint evaluation system for equipment identification and quality level of substation inspection images, characterized in that, include: The image input unit is used to input the acquired raw substation images into a pre-deployed image recognition and quality level joint evaluation model. The model includes a detection and perception image enhancement module, a target detection module, a ROI extraction module, and an image quality evaluation module connected in sequence. The detection and perception image enhancement module includes a semantic segmentation network, a backbone network, and an enhancement network. The image recognition and quality level joint evaluation model is used to enhance the original substation image using the detection and perception image enhancement module to obtain the enhanced image; The target detection module is used to perform target detection on the enhanced image to obtain the category label of power equipment and the coordinate information of the ROI box. The ROI extraction module is used to crop the corresponding equipment ROI image from the original substation image based on the ROI frame coordinate information; The image quality assessment module is used to perform a no-reference image quality assessment on the device ROI image to obtain a quality level label. An enhanced image is obtained by using a detection and perception image enhancement module to enhance the original substation image. This includes processing the original substation image using a semantic segmentation network to generate a corresponding semantic mask to indicate whether a pixel belongs to the equipment area or the background area; extracting features from the original substation image using a backbone network to obtain a feature map of the original substation image; and generating semantically sensitive features based on the semantic mask and the feature map. The semantically sensitive features are input into an enhancement network based on the U-Net architecture for end-to-end enhancement propagation to obtain an enhanced image. The generation of semantically sensitive features based on the semantic mask and feature map includes: performing global average pooling based on the feature map and the downsampled semantic mask to calculate the proportion vector of the device region in each channel of the feature map; mapping the proportion vector to channel attention weights and multiplying it channel-by-channel with the feature map to generate the semantically sensitive features.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-13.

Images