Defect detection method for high-voltage equipment based on deep learning and multispectral image fusion

Abstract

The invention relates to a high-voltage equipment defect detection method based on deep learning and multispectral image fusion, and relates to the technical field of electric power high-voltage equipment state detection. The method comprises the following steps: acquiring an ultraviolet image, an infrared image and a visible light image of the surface of the high-voltage equipment; carrying out image pixel feature-based fusion processing on the ultraviolet image, the infrared image and the visible light image through an image fusion method; establishing a high-voltage equipment defect detection model, and training the high-voltage equipment defect detection model by using the fused image data to obtain a high-voltage equipment defect identification model based on the YOLO-STrans multispectral fusion network; and inputting the ultraviolet image, the infrared image and the visible light image of the outer surface of the power high-voltage equipment into a high-voltage equipment defect identification model to obtain a fault identification result of the to-be-detected power high-voltage equipment. The method can improve the recognition precision of the extremely early insulation degradation and temperature anomaly defects of the surface of the high-voltage power equipment.

Description

Technical Field

[0001] The present invention relates to the technical field of power high-voltage equipment condition detection, and specifically to a high-voltage equipment defect detection method based on deep learning and multi-spectral image fusion. Background Art

[0002] As an important part of the power system, high-voltage equipment will generate ionization discharge during operation due to reasons such as insulation deterioration and moisture ingress, which will in turn cause equipment and system failures. To ensure the stable operation of the power system, it is necessary to monitor the defects of high-voltage equipment in real time.

[0003] Currently, conventional high-voltage equipment defect detection uses two methods: visible light detection and infrared detection, as well as a regular maintenance system. When defects are detected by visible light and infrared, the equipment defects often have deteriorated severely, and it is impossible to monitor and give early warnings in a timely manner when the deterioration trend occurs.

[0004] Therefore, there is an urgent need for a high-voltage equipment defect detection technology that can solve the problem that the existing detection methods cannot monitor and give early warnings in the early stage of equipment defect deterioration, so as to more accurately identify defects such as very early insulation deterioration and temperature anomalies on the surface of high-voltage equipment. Summary of the Invention

[0005] The high-voltage equipment defect detection method of the present invention based on deep learning and multi-spectral image fusion can more accurately identify defects such as very early insulation deterioration and temperature anomalies on the surface of high-voltage equipment by fusing ultraviolet, infrared, and visible light information, and provides a high-voltage equipment defect detection method based on deep learning and multi-spectral image fusion to solve the above problems.

[0006] To achieve the above object, the technical solution of the present invention is as follows:

[0007] In the first aspect of the present invention, a high-voltage equipment defect detection method based on deep learning and multi-spectral image fusion is provided, and the method includes the following steps:

[0008] S1. Obtain the multi-spectral image of the high-voltage equipment and perform preprocessing;

[0009] S2. Perform cross-modal fusion processing on the preprocessed image data;

[0010] S3. Construct a high-voltage equipment defect detection model based on the YOLO-STrans network;

[0011] S4. Use the fused image data to train the high-voltage equipment defect detection model to obtain a high-voltage equipment defect recognition model;

[0012] S5. Obtain the multi - spectral image of the high - voltage equipment to be detected, and use the high - voltage equipment defect recognition model to identify defects in the multi - spectral image.

[0013] Further, in step S1, the obtaining of the multi - spectral image of the high - voltage equipment and the pre - processing specifically include the following steps:

[0014] S11. The multi - spectral image includes an ultraviolet image, an infrared image, and a visible - light image. Pre - process the ultraviolet image, the infrared image, and the visible - light image respectively: Synchronously collect the ultraviolet image of the high - voltage equipment in the target area , infrared image and visible - light image , and establish a three - channel feature space including the ultraviolet image, the infrared image, and the visible - light image with the height and width of the image as dimensions; The three - channel feature space is shown in formula (1):

[0015] (1);

[0016] Among them, represents the three - channel feature quantity, represents the tensor real - number field, represents the height of the image, represents the width of the image, is the pixel value of the ultraviolet channel, is the pixel value of the infrared channel, is the pixel value of the visible - light channel;

[0017] S12. Based on the three - channel feature space, pre - process each modal image in the three - channel feature space.

[0018] Further, in step S2, the cross - modal fusion processing of the pre - processed image data specifically includes the following steps:

[0019] S21. Fuse and superimpose the pre - processed ultraviolet image and visible - light image to obtain the first fusion - superimposed image;

[0020] S22. Fuse and superimpose the pre - processed ultraviolet image and infrared image to obtain the second fusion - superimposed image.

[0021] Further, step S21 specifically includes the following steps:

[0022] S211. Perform edge detection on the pre - processed ultraviolet image and visible - light image respectively, and extract the ultraviolet discharge edge image and the high - voltage equipment structure edge image;

[0023] S212. Extract features from the preprocessed ultraviolet image and visible light image through a dual-channel attention mechanism to generate a first edge weight matrix ;

[0024] The first edge weight matrix is generated by an improved YOLOv8_seg network. The edge weight matrix is as shown in Equation (2):

[0025] (2);

[0026] where represents the first edge weight matrix, represents the Sigmoid activation function, represents the convolution operation, represents the channel concatenation operation, is the pixel value of the ultraviolet channel, is the pixel value of the visible light channel;

[0027] S213. Use the first edge weight matrix to perform weighted fusion on the ultraviolet discharge edge image and the high-voltage equipment structure edge image to generate a first region to be recognized; the first region to be recognized is the overlapping region of the ultraviolet discharge edge image and the high-voltage equipment structure edge image;

[0028] S214. Perform multi-scale feature extraction on the first region to be recognized to obtain an ultraviolet light feature image and a visible light feature image, and perform magnification processing on the ultraviolet light feature image and the visible light feature image;

[0029] The multi-scale feature extraction of the first region to be recognized includes:

[0030] Construct a dual-branch feature pyramid through a residual network, and use the dual-branch feature pyramid to perform multi-scale feature extraction on the first region to be recognized;

[0031] The dual-branch feature pyramid is as shown in Equation (3):

[0032] (3);

[0033] where represents the ultraviolet image pixel feature, represents the output of the third layer of features extracted by the residual network, represents the visible light image pixel feature, is the pixel value of the ultraviolet channel, is the pixel value of the visible light channel;

[0034] S215. The pyramid fusion algorithm with dynamic weight allocation is adopted to perform cross-modal fusion on the magnified ultraviolet feature image and visible light feature image by using Equation (4) to obtain the first fusion superposition image:

[0035] (4);

[0036] Among them, is the pixel value of the first fusion image, represents the dynamic weight, and respectively represent the layer Gaussian pyramid and Laplacian pyramid operations; represents the image modality identifier;

[0037] The dynamic weight is calculated through the attention mechanism shown in Equation (5):

[0038] (5);

[0039] Among them, represents the dynamic weight, represents the exponential operation, represents the ultraviolet image pixel feature, and respectively represent the layer Gaussian pyramid and Laplacian pyramid operations, represents the image modality identifier, represents the visible light image pixel feature.

[0040] Furthermore, step S22 specifically includes the following steps:

[0041] S221. Edge detection is respectively performed on the preprocessed ultraviolet image and infrared image to extract the ultraviolet discharge edge image and infrared edge image;

[0042] S222. Feature extraction is performed on the preprocessed ultraviolet image and infrared image through the dual-channel attention mechanism to generate the second edge weight matrix ;

[0043] The second edge weight matrix is generated by using the improved YOLOv8_seg network, and the second edge weight matrix is as shown in Equation (6):

[0044] (6);

[0045] Among them, is the dynamically generated edge weight matrix, Denote the real number field of tensors, denote the height of the image, denote the width of the image; is an activation function, used to convert the original scores of the model into a probability distribution; denote a convolution operation; denote tensor concatenation; be the pixel value of the ultraviolet channel; be the pixel value of the infrared channel;

[0046] S223. Use the second edge weight matrix to perform weighted fusion on the ultraviolet discharge edge image and the infrared edge image to generate a second region to be recognized; the second region to be recognized is the overlapping region of the ultraviolet discharge edge image and the infrared edge image;

[0047] S224. Perform texture analysis and magnification processing on the second region to be recognized to obtain an ultraviolet light detail image and an infrared light detail image, and respectively extract features from the ultraviolet light detail image and the infrared light detail image to generate a corona intensity feature image and a heat temperature feature image;

[0048] S225. Use the thermodynamic diffusion equation shown in Equation (9) to perform cross-physical domain feature coupling on the corona intensity feature image and the heat temperature feature image to generate a corona-temperature coupling feature:

[0049] (9);

[0050] where, be the heat temperature feature, denote time, be the heat conduction coefficient, be the heat temperature feature of the spatial gradient, be the corona-temperature coupling coefficient, be the corona intensity feature;

[0051] S226. Use Equation (10) to construct a feature importance evaluation function, and based on the feature importance evaluation function, use Equation (11) to perform adaptive magnification processing on the corona-temperature coupling feature:

[0052] (10);

[0053] (11);

[0054] where, denote the enhancement coefficient, be the image pixel coordinates, be the image pixel coordinates The corona intensity feature at is the image pixel coordinate The heat temperature feature at is the corona intensity feature before amplification, is the heat temperature feature before amplification, represents the global maximum corona intensity feature value, represents the global maximum heat temperature feature value, is the corona intensity feature after amplification, is the heat temperature feature after amplification;

[0055] S227. Use Equation (12) to perform multi-physical field modeling on the second area to be recognized :

[0056] (12);

[0057] Wherein, is the corona intensity feature before amplification, is the corona intensity diffusion function; represents the Laplace operator; represents the corona intensity feature image; represents the exponential operation; represents a hyperparameter, which is set manually or determined through training, and is used to control the attenuation degree of the exponential term and adjust the influence weight of this term in the formula; is the ultraviolet intensity threshold, is the infrared intensity threshold, is the spatial gradient of the heat temperature feature of the infrared channel ;

[0058] S228. Use the dynamic Alpha coefficient generation algorithm shown in Equation (13) to determine the adaptive fusion weight of each physical field feature, and use Equation (14) to generate the second fusion superposition image:

[0059] (13);

[0060] (14);

[0061] Wherein, is the dynamic coefficient, is the activation function, is the corona intensity feature after amplification, represents generating the second fusion superposition image, is the heat temperature feature after amplification, represents the function used to take the global maximum value from the entire image;

[0062] S229. Establish a temperature-corona intensity correlation model using Equation (15), and perform hot spot detection on the second coupled superimposed image based on the temperature-corona intensity correlation model to determine the fault type in the second coupled superimposed image;

[0063] (15);

[0064] wherein, represents the pixel value of the output corona intensity feature image or ultraviolet feature image at the image pixel coordinate ; represents the metric value of the ultraviolet image at the image pixel coordinate ; , is the anomaly determination coefficient, is the ultraviolet intensity threshold;

[0065] The anomaly determination coefficient is calculated using the adaptive algorithm shown in Equation (16):

[0066] (16);

[0067] wherein, is the mean value of the ultraviolet image, is the standard deviation of the ultraviolet image.

[0068] Further, in the step S224, the feature extraction of the ultraviolet light detail image and the infrared light detail image respectively to generate a corona intensity feature image and a heat temperature feature image specifically includes the following steps:

[0069] Construct a multi-branch residual network using Equation (7), determine a three-path feature extraction structure including a shallow detail branch, a deep semantic branch, and a cross-modal interaction branch, and perform multi-scale feature extraction on the ultraviolet light detail image and the infrared light detail image through a channel attention module and a spatial-channel joint attention mechanism, and output a corona intensity feature image and a heat temperature feature image:

[0070] (7);

[0071] wherein, represents the corona intensity feature image, represents the heat temperature feature image, represents the output of the fourth layer feature extracted by the residual network, is the channel attention module, is the spatial-channel joint attention mechanism, is the pixel value of the ultraviolet channel, is the pixel value of the infrared channel;

[0072] In the step S224, the corona intensity feature image is the area where the ultraviolet light intensity is greater than the ultraviolet light intensity threshold extracted from the ultraviolet light detail image, including:

[0073] Construct a dynamic threshold calculation model using Equation (8). Based on the dynamic threshold calculation model, segment the area where the ultraviolet light intensity is greater than the ultraviolet light intensity threshold from the ultraviolet detail image to generate a corona intensity feature image:

[0074] (8);

[0075] where represents the ultraviolet intensity threshold, is the mean of the ultraviolet image, is the standard deviation of the ultraviolet image, is the curvature adjustment coefficient; represents with respect to direction and the sum of the second-order partial derivatives in the

[0076] Furthermore, the high-voltage equipment defect detection model is implemented based on the YOLO-STrans network, and the YOLO-STrans network is constructed based on the improved YOLO and Transformer networks;

[0077] The high-voltage equipment defect detection model includes a multi-spectral input end, a CSP-YOLO detection module, a Transformer target extractor, a dynamic fusion module, a multi-task classifier, and a multi-scale output end;

[0078] The multi-spectral input end is used to input multi-spectral images, that is, the collected ultraviolet image, infrared image, and visible light image, and preprocess the input images using the spectral normalization module shown in Equation (17):

[0079] (17);

[0080] where represents the pixel value of the th class of images after standardization processing; represents the pixel value of the original th class of images; , are the mean and standard deviation of each channel respectively; takes the value of ; represents the image modality identifier, represents the ultraviolet image, represents an infrared image, represents a visible light image;

[0081] The CSP-YOLO detection module is used to construct an improved CSPDarknet53 network by using Equation (18), introduce deformable convolutions into the backbone network of CSPDarknet53 to obtain an improved CSPDarknet53 network, and use the improved CSPDarknet53 network for feature extraction:

[0082] (18);

[0083] where, represents the value of the output signal or feature at the position variable at, is the position offset within the neighborhood; is the neighborhood set; is the weight function; is the input signal; is the dynamically learned position offset;

[0084] The Transformer object extractor is used to receive the multi-modal features output by the CSP-YOLO detection module, and calculate the semantic association between any positions in the feature map through the multi-head self-attention mechanism to determine the global dependence relationship;

[0085] The multi-head attention mechanism is shown in Equation (19):

[0086] (19);

[0087] where, is the result of processing the ultraviolet, visible, and infrared spectral images by the multi-head attention mechanism, is the concatenation operation function, represents the output projection weight matrix;

[0088] Attention head is calculated using Equation (20):

[0089] (20);

[0090] where, represents the attention output of the th head; is the activation function used to convert the original scores of the model into a probability distribution; represents the query matrix of the attention head ; represents the key matrix of the attention head ; represents the transpose operator, represents the attention head value matrix; represents the key matrix dimension;

[0091] The dynamic fusion module is used to construct a two-stream feature fusion mechanism by using Equation (21), adaptively generate channel attention weights based on SENet, and combine with the deformable convolutional network to achieve cross-modal complementarity and spatial alignment of ultraviolet, infrared, and visible light multi-modal features:

[0092] (21);

[0093] where, is the dynamic fusion eigenvalue, is the dynamic coefficient, is the input YOLO network eigenvalue, is the input Transformer network eigenvalue, is the deformable convolutional network;

[0094] The multi-task classifier classifies defects and identifies faults in the high-voltage equipment area extracted by the Transformer object extractor; the multi-task classifier constructs a depthwise separable convolution parallel output branch by using Equation (22) to classify defects and identify faults in the equipment area output by the Transformer object extractor:

[0095] (22);

[0096] where, is the multi-dimensional classification and localization loss value, is the position loss weight value, is the position loss value, is the classification loss weight value, is the classification loss value;

[0097] The multi-scale output end detects defects of different scales based on the feature pyramid and outputs a three-channel detection result by using Equation (23):

[0098] (23);

[0099] where, represents the diagnosis result, represents operation, represents the discharge probability; is the activation function, represents the temperature anomaly probability, represents the appearance defect probability.

[0100] In the second aspect of the present invention, a high-voltage equipment defect detection device based on deep learning and multi-spectral image fusion is disclosed. The device includes a spectral splitting module, an image processing and fusion module, a model construction and training module, and a defect recognition module. The spectral splitting module is used to obtain the multi-spectral image of the high-voltage equipment. The image processing and fusion module is used to preprocess the multi-spectral image and perform cross-modal fusion processing on the preprocessed image data. The model construction and training module is used to construct a high-voltage equipment defect detection model based on the YOLO-STrans network, and use the fused image data to train the high-voltage equipment defect detection model to obtain a high-voltage equipment defect recognition model. The defect recognition module is used to obtain the multi-spectral image of the high-voltage equipment to be detected and use the high-voltage equipment defect recognition model to identify the defects in the multi-spectral image.

[0101] In the third aspect of the present invention, an electronic device is disclosed, including: at least one processor; and a memory that stores instructions, which when executed by the at least one processor, cause the at least one processor to execute the above-mentioned high-voltage equipment defect detection method based on deep learning and multi-spectral image fusion.

[0102] In the fourth aspect of the present invention, a machine-readable storage medium is disclosed, which stores executable instructions that, when executed, cause the machine to execute the above-mentioned high-voltage equipment defect detection method based on deep learning and multi-spectral image fusion.

[0103] Beneficial effects

[0104] Compared with the prior art, the high-voltage equipment defect recognition method of the present invention obtains the ultraviolet image, visible image, and infrared image of the high-voltage equipment, and performs fusion and superposition on the obtained ultraviolet image and visible light image to obtain a first fusion and superposition image, and performs fusion and superposition on the ultraviolet image and infrared image to obtain a second fusion and superposition image; uses the pre-trained high-voltage equipment defect detection model to process the first fusion and superposition image and the second fusion and superposition image, and finally obtains the recognition result of the equipment state. The present invention realizes the analysis of high-voltage equipment from multiple dimensions and levels by fusing and superposing the ultraviolet image and visible light image, and fusing and superposing the ultraviolet image and infrared image, avoiding the omission of image information; furthermore, inputting the result of image fusion and superposition into the pre-trained high-voltage equipment defect detection model realizes automatic processing, solves the difficulty of manual inspection, and ensures the accuracy of the inspection result, and finally realizes the accurate recognition of the state of high-voltage equipment. Description of the drawings

[0105] Figure 1It is a flowchart of the high-voltage equipment defect detection method based on deep learning and multi-spectral image fusion in the present invention;

[0106] Figure 2 It is a flowchart of the method for obtaining the discharge intensity, temperature, and appearance defects of the multi-spectral image in the present invention;

[0107] Figure 3 It is an architecture diagram of the high-voltage equipment defect detection and recognition model in the present invention;

[0108] Figure 4 It is an architecture diagram of the CSP-YOLO detection module in the high-voltage equipment defect detection model of the present invention;

[0109] Figure 5 It is an architecture diagram of the Transformer target extraction and classifier in the high-voltage equipment defect detection model of the present invention;

[0110] Figure 6 It is a structural block diagram of the high-voltage equipment defect detection device based on deep learning and multi-spectral image fusion in the present invention. Specific Embodiments

[0111] To have a further understanding and recognition of the structural features and achieved effects of the present invention, the following is a detailed description with reference to preferred embodiments and accompanying drawings:

[0112] Embodiment 1

[0113] This embodiment provides a high-voltage equipment defect detection method based on deep learning and multi-spectral image fusion. The method first obtains the spectral image information of three bands, namely ultraviolet, infrared, and visible light, on the surface of the target high-voltage equipment; then extracts the features of the spectral image information of the three bands, and obtains the ultraviolet corona intensity, the surface temperature of the equipment, and appearance defects through image analysis; next, the ultraviolet image and the visible light image are fused and superimposed to obtain a first fused and superimposed image; the ultraviolet image and the infrared image are fused and superimposed to obtain a second fused and superimposed image; a high-voltage equipment defect detection model is constructed, and the first fused and superimposed image and the second fused and superimposed image are input into the pre-trained high-voltage equipment defect detection model, and the high-voltage equipment defect detection model outputs the fault recognition result of the target line in the target section.

[0114] As Figure 1 to Figure 2 shown, the high-voltage equipment defect detection method based on deep learning and multi-spectral image fusion of the present invention includes the following steps:

[0115] S1. Obtain the multi-spectral image of the high-voltage equipment and perform preprocessing.

[0116] Obtain a multi-spectral image of the high-voltage equipment as the original image. The multi-spectral image includes an ultraviolet image, an infrared image, and a visible light image, and preprocess the ultraviolet image, the infrared image, and the visible light image respectively.

[0117] The multi-spectral image includes an ultraviolet image, an infrared image, and a visible light image. In this embodiment, professional imaging devices such as an ultraviolet imager, an infrared thermal imager, and a high-definition visible light camera are used to collect images of the high-voltage equipment. The high-voltage equipment in this embodiment includes ultra-high voltage equipment. Among them, the ultraviolet imager is used to capture the ultraviolet light signal generated by corona discharge on the surface of the equipment. The ultraviolet light signal can reflect the insulation state of the equipment, especially potential partial discharge phenomena; the infrared thermal imager obtains the temperature distribution image of the equipment by detecting the thermal radiation on the surface of the equipment to identify overheating defects caused by reasons such as poor contact and overload; the high-definition visible light camera is used to record the appearance details of the high-voltage equipment, such as cracks, deformations, foreign object attachments, etc. During the acquisition process, it is necessary to ensure that the shooting angles, distances, and times of the three devices are consistent to ensure that the collected images have good consistency in space and time, facilitating subsequent processing.

[0118] In step S1, for the step of obtaining a multi-spectral image of the high-voltage equipment as the original image, where the multi-spectral image includes an ultraviolet image, an infrared image, and a visible light image, and preprocessing the ultraviolet image, the infrared image, and the visible light image respectively, it specifically includes the following steps:

[0119] S1. Obtain a multi-spectral image of the high-voltage equipment and perform preprocessing.

[0120] Obtain a multi-spectral image of the high-voltage equipment as the original image. The multi-spectral image includes an ultraviolet image, an infrared image, and a visible light image, and preprocess the ultraviolet image, the infrared image, and the visible light image respectively.

[0121] The multi-spectral image includes an ultraviolet image, an infrared image, and a visible light image. In this embodiment, professional imaging devices such as an ultraviolet imager, an infrared thermal imager, and a high-definition visible light camera are used to collect images of the high-voltage equipment. The high-voltage equipment in this embodiment includes ultra-high voltage equipment. Among them, the ultraviolet imager is used to capture the ultraviolet light signal generated by corona discharge on the surface of the equipment. The ultraviolet light signal can reflect the insulation state of the equipment, especially potential partial discharge phenomena; the infrared thermal imager obtains the temperature distribution image of the equipment by detecting the thermal radiation on the surface of the equipment to identify overheating defects caused by reasons such as poor contact and overload; the high-definition visible light camera is used to record the appearance details of the high-voltage equipment, such as cracks, deformations, foreign object attachments, etc. During the acquisition process, it is necessary to ensure that the shooting angles, distances, and times of the three devices are consistent to ensure that the collected images have good consistency in space and time, facilitating subsequent processing.

[0122] In the step S1, obtaining a multi-spectral image of the high-voltage device as the original image, where the multi-spectral image includes an ultraviolet image, an infrared image, and a visible light image, and preprocessing the ultraviolet image, the infrared image, and the visible light image respectively, specifically including the following steps:

[0123] S11. The multi-spectral image includes an ultraviolet image, an infrared image, and a visible light image. Preprocess the ultraviolet image, the infrared image, and the visible light image respectively: Synchronously collect the ultraviolet image of the high-voltage device in the target area , the infrared image and the visible light image , and establish a three-channel feature space including the ultraviolet image, the infrared image, and the visible light image with the height and width of the image as dimensions; the three-channel feature space is shown in Equation (1):

[0124] (1);

[0125] Wherein, represents the three-channel feature quantity, represents the tensor real number field, represents the height of the image, represents the width of the image, is the pixel value of the ultraviolet channel, is the pixel value of the infrared channel, is the pixel value of the visible light channel.

[0126] In the detection of high-voltage devices, first synchronously collect the ultraviolet image, the infrared image, and the visible light image of the target area of the high-voltage device (such as insulator strings, transformer bushings, etc.). Assume that the spatial resolutions of the three types of images are the same, all being . To ensure the consistency of the image data, when collecting images, existing spatio-temporal registration techniques need to be used. Through geometric correction and time synchronization triggering devices, the three types of images are pixel-level aligned in the spatial and temporal dimensions, that is, the coordinates of the same physical position in different images correspond to the same. After completing the image collection and alignment, based on the height and width dimensions of the image, construct a three-channel feature space containing ultraviolet, infrared, and visible light information . Combine the ultraviolet image, the infrared image, and the visible light image along the channel dimension to obtain a tensor. In the three-channel feature space, the pixel value of the ultraviolet channel characterizes the corona discharge intensity at the corresponding position, and the higher the gray value, the more intense the discharge; the pixel value of the infrared channel corresponds to the temperature value in the infrared band, and this temperature value is obtained by inversion according to Planck's law; the pixel value is the visible light RGB pixel value, reflecting the appearance texture and structural details of the device. The subsequent three-channel feature space can be directly used as the input of the deep learning model, facilitating the model to utilize multi-modal information for defect detection and analysis of high-voltage equipment. In step S11, images are synchronously collected by ultraviolet, infrared, and visible light cameras to establish a three-channel feature space, and through hardware synchronous triggering and optical calibration, spatio-temporal alignment is ensured.

[0127] S12. Based on the three-channel feature space, preprocess each modal image in the three-channel feature space.

[0128] A channel refers to the dimensional division of data in the feature space. Each channel in the three-channel feature space corresponds to the image data of one modality. Modality refers to the differences in the physical characteristics or imaging principles of data. For example, ultraviolet, infrared, and visible light belong to different spectral modalities. Each channel carries the image information of one modality, and there is no channel mixing or modality missing. In this embodiment, each modal image in the three-channel feature space includes ultraviolet images, infrared images, and visible light images. Existing preprocessing techniques are used to perform noise processing, image enhancement, image geometric correction, and size normalization on the ultraviolet images, infrared images, and visible light images respectively.

[0129] (1) Noise processing

[0130] Due to factors such as the noise of the imaging device and environmental interference, there are various noises in the original images. The original ultraviolet images, original visible light images, and original infrared images in the three-channel feature space are respectively denoised to obtain high-quality ultraviolet images, visible light images, and infrared images. For multiplicative noise and salt-and-pepper noise in the images, existing Wiener filtering algorithms can be used for denoising. For Gaussian noise, existing median filtering algorithms can be used for denoising. Through denoising, clearer ultraviolet images, visible light images, and infrared images can be obtained. Preferably, for ultraviolet images and visible light images, the median filtering algorithm is used to remove salt-and-pepper noise. By replacing the value of each pixel point with the median of the pixel values in its neighborhood, isolated noise points can be effectively suppressed. For infrared images, due to their sensitivity to temperature, the Gaussian filtering algorithm is used to smooth the images, reducing the influence of thermal noise while retaining the edge information of temperature changes.

[0131] (2) Image enhancement

[0132] To highlight the key information in the image, image enhancement processing is required. For ultraviolet images, the gray dynamic range of the image is expanded through histogram equalization to make the contrast between the discharge area and the background more obvious. For infrared images, the linear stretching algorithm is used to map the temperature data to a more appropriate gray interval for easy observation of temperature differences. For visible light images, the adaptive local contrast enhancement algorithm is adopted to adjust the contrast according to the local characteristics in different regions, making the appearance details of the device clearer.

[0133] (3)Image geometric correction

[0134] Due to reasons such as the shooting angle and lens distortion, the original image may have geometric deformation. The perspective transformation and lens distortion correction algorithms are used to perform geometric correction on the visible light image and the infrared image to make them conform to the true geometric shape of the device. For ultraviolet images, since the main concern is the position and intensity of the discharge point, simple rotation and translation corrections are performed when necessary to ensure spatial alignment with other images.

[0135] (4)Size adjustment

[0136] The size of the image is uniformly adjusted to a fixed size, such as 512×512 pixels, for subsequent processing and input into the high-voltage equipment defect detection model. At the same time, the pixel values of the image are normalized and mapped to the interval of [0,1] or [-1,1] to eliminate the brightness difference and numerical scale difference between different images and improve the stability of subsequent feature extraction and model training.

[0137] S2. Perform cross-modal fusion processing on the preprocessed image data.

[0138] Perform cross-modal fusion processing on the preprocessed ultraviolet image, infrared image, and visible light image to generate a fused feature image. The fused feature image includes a first fused overlay image and a second fused overlay image.

[0139] In the step S2, the cross-modal fusion processing of the preprocessed ultraviolet image, infrared image, and visible light image specifically includes the following steps:

[0140] S21. Fuse and overlay the preprocessed ultraviolet image and visible light image to obtain a first fused overlay image.

[0141] In the step S21, the fusion and overlay of the preprocessed ultraviolet image and visible light image specifically includes the following steps:

[0142] S211. Perform edge detection on the preprocessed ultraviolet image and visible light image respectively to extract the ultraviolet discharge edge image and the high-voltage equipment structure edge image.

[0143] S212. Extract features from the preprocessed ultraviolet image and visible light image through a dual-channel attention mechanism to generate a first edge weight matrix .

[0144] The first edge weight matrix is generated using an improved YOLOv8_seg network. The edge weight matrix is as shown in Equation (2):

[0145] (2);

[0146] where represents the first edge weight matrix, represents the Sigmoid activation function, represents the convolution operation, represents the channel concatenation operation, is the pixel value of the ultraviolet channel, is the pixel value of the visible light channel.

[0147] S213. Use the first edge weight matrix to perform weighted fusion on the ultraviolet discharge edge image and the high-voltage equipment structure edge image to generate a first region to be recognized; the first region to be recognized is the overlapping region of the ultraviolet discharge edge image and the high-voltage equipment structure edge image.

[0148] S214. Perform multi-scale feature extraction on the first region to be recognized to obtain an ultraviolet light feature image and a visible light feature image, and perform magnification processing on the ultraviolet light feature image and the visible light feature image.

[0149] The multi-scale feature extraction of the first region to be recognized includes:

[0150] Construct a dual-branch feature pyramid through a residual network, and use the dual-branch feature pyramid to perform multi-scale feature extraction on the first region to be recognized;

[0151] The dual-branch feature pyramid is as shown in Equation (3):

[0152] (3);

[0153] where represents the ultraviolet image pixel feature, represents the output of the third layer feature extracted by the residual network, represents the visible light image pixel feature, is the pixel value of the ultraviolet channel, is the pixel value of the visible light channel.

[0154] S215. The pyramid fusion algorithm with dynamic weight allocation is adopted to perform cross-modal fusion on the magnified ultraviolet light feature image and visible light feature image by using Equation (4) to obtain the first fusion superposition image:

[0155] (4);

[0156] Among them, is the pixel value of the first fusion image, represents the dynamic weight, and respectively represent the layer Gaussian pyramid and Laplacian pyramid operations; represents the image modality identifier.

[0157] The dynamic weight is calculated through the attention mechanism shown in Equation (5):

[0158] (5);

[0159] Among them, represents the dynamic weight, represents the exponential operation, represents the ultraviolet image pixel feature, and respectively represent the layer Gaussian pyramid and Laplacian pyramid operations, represents the image modality identifier, represents the visible light image pixel feature.

[0160] S22. The preprocessed ultraviolet image and infrared image are fused and superimposed to obtain the second fusion superposition image.

[0161] In the step S22, the preprocessed ultraviolet image and infrared image are fused and superimposed to obtain the second fusion superposition image, which specifically includes the following steps:

[0162] S221. Edge detection is respectively performed on the preprocessed ultraviolet image and infrared image to extract the ultraviolet discharge edge image and infrared edge image.

[0163] S222. Feature extraction is performed on the preprocessed ultraviolet image and infrared image through a dual-channel attention mechanism to generate the second edge weight matrix .

[0164] The second edge weight matrix is generated by using the improved YOLOv8_seg network, and the second edge weight matrix is as shown in Equation (6):

[0165] (6);

[0166] Wherein, is a dynamically generated edge weight matrix, represents the real number field of the tensor, represents the height of the image, represents the width of the image; is an activation function for converting the original score of the model into a probability distribution; represents a convolution operation; represents tensor concatenation; is the pixel value of the ultraviolet channel; is the pixel value of the infrared channel.

[0167] S223. Use the second edge weight matrix to perform weighted fusion on the ultraviolet discharge edge image and the infrared edge image to generate a second region to be recognized. The second region to be recognized is the overlapping region of the ultraviolet discharge edge image and the infrared edge image.

[0168] S224. Perform texture analysis and magnification processing on the second region to be recognized to obtain an ultraviolet light detail image and an infrared light detail image, and respectively extract features from the ultraviolet light detail image and the infrared light detail image to generate a corona intensity feature image and a heat temperature feature image. The corona intensity feature image is the region where the ultraviolet light intensity is greater than the ultraviolet light intensity threshold extracted from the ultraviolet light detail image; the heat temperature feature image is the region where the infrared light intensity is greater than the infrared light intensity threshold extracted from the infrared light detail image.

[0169] In the step S224, the extracting features from the ultraviolet light detail image and the infrared light detail image respectively to generate a corona intensity feature image and a heat temperature feature image specifically includes the following steps:

[0170] Use formula (7) to construct a multi-branch residual network, determine a three-path feature extraction structure including a shallow detail branch, a deep semantic branch and a cross-modal interaction branch, and perform multi-scale feature extraction on the ultraviolet light detail image and the infrared light detail image through a channel attention module and a spatial-channel joint attention mechanism, and output a corona intensity feature image and a heat temperature feature image:

[0171] (7);

[0172] Wherein, represents the corona intensity feature image, represents the heat temperature feature image, represents the output of the fourth layer feature extracted by the residual network, is the channel attention module, is the spatial-channel joint attention mechanism, is the pixel value of the ultraviolet channel, is the pixel value of the infrared channel;

[0173] In the step S224, the corona intensity feature image is the area where the ultraviolet light intensity is greater than the ultraviolet light intensity threshold extracted from the ultraviolet light detail image, including:

[0174] Use Equation (8) to construct a dynamic threshold calculation model. Based on the dynamic threshold calculation model, segment the area where the ultraviolet light intensity is greater than the ultraviolet light intensity threshold from the ultraviolet detail image to generate a corona intensity feature image:

[0175] (8);

[0176] where, represents the ultraviolet intensity threshold, is the mean value of the ultraviolet image, is the standard deviation of the ultraviolet image, is the curvature adjustment coefficient; represents with respect to direction and direction of the sum of the second-order partial derivatives.

[0177] S225. Use the thermodynamic diffusion equation shown in Equation (9) to perform feature coupling across physical domains on the corona intensity feature image and the heat temperature feature image to generate a corona-temperature coupling feature:

[0178] (9);

[0179] where, is the heat temperature feature, represents time, is the heat conduction coefficient, is the heat temperature feature of the spatial gradient, is the corona-temperature coupling coefficient, is the corona intensity feature.

[0180] S226. Use Equation (10) to construct a feature importance evaluation function, and based on the feature importance evaluation function, use Equation (11) to perform adaptive amplification processing on the corona-temperature coupling feature:

[0181] (10);

[0182] (11);

[0183] Among them, represents the enhancement coefficient, is the image pixel coordinate, is the image pixel coordinate of the corona intensity feature at the location, is the image pixel coordinate of the heat temperature feature at the location, is the corona intensity feature before amplification, is the heat temperature feature before amplification, represents the global maximum corona intensity feature value, represents the global maximum heat temperature feature value, is the corona intensity feature after amplification, is the heat temperature feature after amplification.

[0184] S227. Perform multi-physical field modeling on the second area to be recognized using Equation (12): :

[0185] (12);

[0186] Among them, is the corona intensity feature before amplification, is the corona intensity diffusion function; represents the Laplace operator; represents the corona intensity feature image; represents the exponential operation; represents the hyperparameter, which is set manually or determined through training, and is used to control the attenuation degree of the exponential term and adjust the influence weight of this term in the formula; is the ultraviolet intensity threshold, is the infrared intensity threshold, is the spatial gradient of the heat temperature feature of the infrared channel .

[0187] S228. Determine the adaptive fusion weight of each physical field feature using the dynamic Alpha coefficient generation algorithm shown in Equation (13), and generate the second fusion superposition image using Equation (14):

[0188] (13);

[0189] (14);

[0190] Among them, is the dynamic coefficient, is the activation function, is the corona intensity feature after amplification, Indicates generating a second fused superposed image, is the magnified heat temperature feature, Indicates a function used to obtain the global maximum value from the entire image.

[0191] S229. Establish a temperature-corona intensity correlation model using Equation (15), and perform hot spot detection on the second coupled superposed image based on the temperature-corona intensity correlation model to determine the fault type in the second coupled superposed image;

[0192] (15);

[0193] Among them, represents the pixel value of the output corona intensity feature image or ultraviolet feature image at the image pixel coordinate location, represents the metric value of the ultraviolet image at the image pixel coordinate location, , is the anomaly determination coefficient, is the ultraviolet intensity threshold.

[0194] The anomaly determination coefficient , is calculated using the adaptive algorithm shown in Equation (16):

[0195] (16);

[0196] Among them, is the mean value of the ultraviolet image, is the standard deviation of the ultraviolet image.

[0197] S3. Construct a high-voltage equipment defect detection model based on the YOLO-STrans network.

[0198] The high-voltage equipment defect detection model is implemented based on the YOLO-STrans network, and the YOLO-STrans network is constructed based on the improved YOLO and Transformer networks. As Figure 3 shown, the high-voltage equipment defect detection model includes a multi-spectral input end, a CSP-YOLO detection module, a Transformer target extractor, a dynamic fusion module, a multi-task classifier, and a multi-scale output end.

[0199] (1) Multi-spectral input end

[0200] The multi-spectral input end is used to input multi-spectral images, that is, the collected ultraviolet images, infrared images, and visible light images, and preprocess the input images using the spectral normalization module shown in Equation (17):

[0201] (17);

[0202] Among them, represents the pixel value after the th type of image (ultraviolet, infrared, visible light) has been standardized, and is used for subsequent feature extraction or fusion, making different modality data comparable in numerical range; represents the pixel value of the original th type of image, such as the pixel intensity of ultraviolet images, infrared images, and visible light images; , are the mean and standard deviation of each channel respectively; The value of is to prevent numerical overflow; represents the image modality identifier, represents the ultraviolet image, represents the infrared image,

[0203] (2) CSP-YOLO Detection Module

[0204] The CSP-YOLO detection module is used to construct an improved CSPDarknet53 network using Equation (18). A deformable convolution is introduced into the backbone network of CSPDarknet53 to obtain an improved CSPDarknet53 network, and the improved CSPDarknet53 network is used for feature extraction:

[0205] (18);

[0206] Among them, represents the value of the output signal or feature at the position variable , representing the result after weighted processing, such as the filtered pixel value, the fused feature, etc.; is the position variable, representing the space of the processing object; is the position offset within the neighborhood; is the neighborhood set; is the weight function, which assigns weights to each position within the neighborhood, such as the coefficients of the Gaussian filter, the weights of the attention mechanism, and is used to control the contribution degree of different positions to the output ; is the input signal; is the dynamically learned position offset, which is used to improve the detection accuracy of irregular discharge regions.

[0207] In the backbone network of CSPDarknet53, some standard convolutional layers are replaced with deformable convolutional layers to form an improved CSPDarknet53 network. When the CSP-YOLO detection module detects small targets (such as faint corona spots) or defects with variable shapes, it dynamically adjusts the sampling points to match the target contour; during the feature fusion process, the spatial alignment ability of features in different stages is enhanced through deformable convolution. By dynamically adjusting the sampling positions of the convolution kernels, the ability to extract features of irregular defects in high-voltage equipment (such as non-rectangular discharge areas and complex-shaped hot spots) is improved, that is, the ability to extract defect features is improved.

[0208] The architecture diagram of the CSP-YOLO detection module in this embodiment is as Figure 4 shown, and the main structure includes an input end, a backbone network, a Neck network, and an output end. Among them, the input end is used for inputting picture data. The backbone network consists of a Focus module for downsampling, a Conv module, a C3 connection module for residual network feature extraction, and an SPP pyramid pooling module. The Neck network is the part used for further feature extraction and fusion, mainly composed of a C3 module, an Upsample upsampling module, a Conv convolutional module, and a feature fusion Concat module. The output end is used to output the prediction results of the picture.

[0209] In the high-voltage equipment defect detection model described in this embodiment, the CSP-YOLO detection module is an image segmentation model based on the improved YOLO network. First, a CBAM convolutional attention mechanism is added to the backbone network of the YOLOv8 network to effectively avoid the influence of equipment occlusion problems on the detection and recognition accuracy. Secondly, a BiFPN (Bidirectional Feature Pyramid Network) weighted bidirectional (top-down + bottom-up) feature pyramid network is used to replace the FPN (Feature Pyramid Network) and PAN (Path Aggregation Network) network structures used in the original YOLOv8 network, improving the feature extraction ability of the algorithm model for targets of different scales. Finally, at the output end of the model network, an SIOU prediction box position loss function is used to replace the CIOU (Complete Intersection over Union) loss function used in the original YOLOv8 network. On the basis of the YOLOv8 network, the direction position loss of the target is comprehensively considered, so as to effectively prevent the influence of the position deviation of devices such as cameras on the detection in actual applications.

[0210] (3)Transformer Object Extractor

[0211] The Transformer target extractor is used to receive the multi-modal features (ultraviolet / infrared feature maps) output by the CSP-YOLO detection module, and calculate the semantic associations between any positions in the feature maps through the multi-head self-attention mechanism to determine the global dependency relationship.

[0212] The multi-head attention mechanism is shown as follows:

[0213] The multi-head attention mechanism is shown as in Equation (19):

[0214] (19);

[0215] Where, is the result after the multi-head attention mechanism processes the images of the ultraviolet, visible, and infrared spectra, is the concatenation operation function, represents the output projection weight matrix;

[0216] Attention head is calculated using Equation (20):

[0217] (20);

[0218] Where, represents the attention output of the -th head. Each head calculates the attention through an independent linear transformation to capture the features of different subspaces (such as space, channel, modality, etc.); is the activation function used to convert the original scores of the model into a probability distribution; represents the query matrix of the attention head ; represents the key matrix of the attention head ; represents the transpose operator, represents the value matrix of the attention head ; represents the key matrix dimension for scaling.

[0219] Specifically, the Transformer target extractor performs target extraction on the area where the high-voltage equipment is located, and extracts the area where the high-voltage equipment is located from the segmented image according to the output result of the image segmentation detection of the CSP-YOLO detection module to filter out background interference and focus on the defect detection of the equipment area (such as insulators, conductors, etc.).

[0220] (4) Dynamic Fusion Module

[0221] The dynamic fusion module is used to construct a two-stream feature fusion mechanism using Equation (21), adaptively generate channel attention weights using SENet, and combine with a deformable convolutional network to achieve cross-modal complementarity and spatial alignment of ultraviolet, infrared, and visible light multi-modal features, thereby improving the accuracy and robustness of high-voltage equipment defect detection.

[0222] (21);

[0223] Where, is the dynamic fusion feature value; is the dynamic coefficient, adaptively generated by SENet; is the feature value input to the YOLO network; is the feature value input to the Transformer network; is the deformable convolutional network.

[0224] In the high-voltage equipment defect detection model described in this embodiment, the role of the fusion module is to be responsible for performing downsampling operations on the feature map before the start of each stage, reducing the resolution of the picture through convolutional operations with a stride of 2, and at the same time, adjusting the number of channels to achieve hierarchical processing of features, thereby realizing a hierarchical structure. In addition, the fusion module can also reduce computations and capture defect features at different scales through hierarchical feature extraction.

[0225] The dynamic fusion module can not only precisely align two kinds of defect features by adjusting the sampling points through deformable convolution, avoiding missed detections; but also suppress noise channels in visible light through channel attention and retain effective features of ultraviolet / infrared.

[0226] The dynamic fusion module solves the problems of information redundancy and conflict existing in traditional feature splicing through adaptive weight assignment and deformable convolution. The two-stream feature fusion mechanism means that the dynamic fusion module processes features of different modalities in two paths: one is the channel attention branch, which generates channel weights α through SENet to suppress irrelevant channels and enhance key features (such as ultraviolet features in the discharge area); the other is the spatial feature alignment branch, which uses deformable convolution (DCN) to dynamically adjust sampling points to compensate for spatial misalignment between different modalities (such as the position deviation between ultraviolet discharge points and infrared hot spots). After the two-path feature processing, fusion is achieved through weighted summation. The feature map after fusion by the dynamic fusion module is passed to a multi-task classifier (such as a classification branch based on depthwise separable convolution) for tasks such as defect type recognition and severity assessment.

[0227] Traditional methods (such as Early Fusion and Late Fusion) use preset weights and cannot adapt to the differences in feature distributions of different defect types (such as discharges and heat generation). In contrast, the dynamic fusion module in this embodiment dynamically learns weights through SENet, enhancing the weight of ultraviolet features in the discharge scenario and boosting the weight of infrared features in the heat generation scenario. To address the problem that the fixed sampling grid of standard convolution has difficulty handling spatial misalignments between different modalities, the dynamic fusion module in this embodiment utilizes the deformable convolutional network (DCN) to dynamically adjust the sampling positions of the convolutional kernels. The DCN enables the model to capture the precise correspondence between ultraviolet discharge points and infrared hotspots by learning the offsets.

[0228] (5) Multi-task classifier

[0229] The multi-task classifier classifies defects and identifies faults in the high-voltage equipment areas extracted by the Transformer object extractor, and outputs results such as defect types (such as discharges and overheating) and severity levels. The multi-task classifier is improved based on the Transformer network, introducing a sliding window mechanism to calculate self-attention within local windows, reducing the computational complexity while maintaining the global modeling ability.

[0230] The multi-task classifier constructs depthwise separable convolution parallel output branches using Equation (22) to classify defects and identify faults in the equipment areas output by the Transformer object extractor, thereby achieving multi-dimensional classification and localization of defects:

[0231] (22);

[0232] where is the multi-dimensional classification and localization loss value, is the position loss weight value, is the position loss value, is the classification loss weight value, is the classification loss value.

[0233] The multi-task classifier adds a sliding window mechanism to the Transformer network, calculates self-attention within local windows, reduces the computational complexity, and maintains the global modeling ability.

[0234] The network structure of the multi-task classifier is as shown in Figure 5As shown, the multi-task classifier has a hierarchical structure. The process of processing images is divided into 4 stages in total. In each stage, the resolution of the feature map input into the network is reduced, and this process is similar to the convolutional neural network that can gradually extract image features. Each stage contains a PatchMerging module for fusing strides and multiple Transformer Blocks. Before input, feature embedding is first performed. After the image is segmented, vectors are embedded, that is, the image is cut into small pieces and embedded into the Embedding. Before inputting into the Block, the input image needs to be cut into many patches of a fixed size, and then vectors are embedded. Through a two-dimensional convolutional layer, both the stride and the kernel size are set to the patch size to achieve position encoding. Finally, the height H and width W dimensions are unfolded and moved to the first dimension. Inside each Block, attention is calculated only within a window of a fixed size (such as 7×7), and information interaction is achieved through sliding window shifting when crossing windows.

[0235] The multi-task classifier is used to receive the multi-modal features output by the dynamic fusion module, process them, and output classification and localization results. The multi-task classifier constructs multiple parallel branches based on depthwise separable convolution to simultaneously process different detection tasks such as defect type classification, position regression, and severity assessment. The defect type classification refers to identifying types such as discharge, overheating, and mechanical damage; the position regression refers to predicting the bounding box coordinates of the defect; the severity assessment refers to judging the defect level (general, serious, critical). The multi-task classifier constructs parallel branches through DSConv. Different tasks (classification, localization, severity assessment) can share underlying features while maintaining task independence and avoiding interference between tasks. The dynamic class weights are automatically adjusted according to the sample distribution of each class, which is more adaptable than fixed weights.

[0236] (6)Multi-scale output end

[0237] The multi-scale output end, based on the feature pyramid, uses equation (23) to detect defects of different scales and outputs three-channel detection results to achieve three-dimensional diagnosis of discharge area localization, temperature anomaly recognition, and defect classification.

[0238] (23);

[0239] Among them, represents the diagnosis result, denotes operation, represents the discharge probability; is the activation function, represents the temperature anomaly probability, represents the appearance defect probability.

[0240] In the high-voltage equipment defect detection model, the multi-scale output end realizes the refined detection of defects at different scales through the feature pyramid, combines the three-channel output (ultraviolet, infrared, visible light) to complete three-dimensional diagnosis, and outputs the high-voltage equipment defect detection results.

[0241] Considering the significant differences in the sizes of high-voltage equipment defects, there are small defects such as initial corona points and micro-cracks, which require high-resolution feature maps to capture details; and large defects such as large-area heating regions and equipment structure deformations, which require low-resolution feature maps to extract global semantics. In this embodiment, the feature pyramid enables the model to simultaneously have high-resolution detail perception and low-resolution semantic understanding capabilities by fusing features at different levels. The three-channel output includes an ultraviolet channel, an infrared channel, and a visible light channel. Among them, the ultraviolet channel is used to locate the discharge area and output the spot coordinates and intensity; the infrared channel is used to identify temperature anomalies and mark the hot spot position and temperature value; the visible light channel is used to classify defect types (such as cracks, contamination) and provide physical structure context. The multi-scale output end realizes multi-dimensional verification of defects and reduces the false detection rate through cross-channel association (such as spatial alignment of discharge points and hot spots).

[0242] S4. Use the fused image data to train the high-voltage equipment defect detection model to obtain a high-voltage equipment defect recognition model.

[0243] The high-voltage equipment defect detection model in this embodiment is a model based on a deep neural network. A data set is constructed, and the high-voltage equipment defect detection model is trained using the data set. The trained high-voltage equipment defect detection model is the high-voltage equipment defect recognition model. The data set is collected by a detection device in a simulated laboratory and a real application scenario, and the high-voltage equipment defect detection model is trained using the training data in the data set. The training data includes, but is not limited to, ultraviolet images, visible light images, infrared images, first fusion superimposed images, and second fusion superimposed images.

[0244] S5. Obtain the multi-spectral image of the high-voltage equipment to be detected and input it into the high-voltage equipment defect recognition model, and use the high-voltage equipment defect recognition model to perform defect recognition. In step S5, the high-voltage equipment defect detection model is verified.

[0245] In this embodiment, through operations such as edge detection, feature extraction, and magnification processing on the ultraviolet image and the visible light image, the detailed processing of the discharge area in the ultraviolet image and the visible light image is realized, laying a foundation for accurately identifying line faults. Through operations such as edge detection, texture analysis, and magnification processing on the ultraviolet image and the infrared image, other useless infrared information is filtered out, and the detailed processing of the equipment in the ultraviolet image and the infrared image is realized, laying a foundation for accurately detecting equipment status faults.

[0246] The comparison of the method of the present invention with other detection algorithms in the prior art in the same dataset experiment is as follows: for YOLOv5, the precision P is 91.7%, the recall rate R is 80.5%, and the mean average precision mAP is 87.7%; for YOLOv8, the precision P is 92.4%, the recall rate R is 89.3%, and the mean average precision mAP is 91.9%; for YOLOv10, the precision P is 93.2%, the recall rate R is 91.6%, and the mean average precision mAP is 92.7%; for the present invention, the precision P is 96.7%, the recall rate R is 95.3%, and the mean average precision mAP is 98.7%. From the above data, it can be seen that the method of the present invention is superior to other algorithms in terms of precision, recall rate, and mean average precision in the experiment.

[0247] Example 2

[0248] This embodiment provides a high-voltage equipment defect detection device based on deep learning and multi-spectral image fusion, and this device can apply the above-mentioned high-voltage equipment defect detection method based on deep learning and multi-spectral image fusion.

[0249] As Figure 6 shown, the present invention also provides a high-voltage equipment defect detection device based on deep learning and multi-spectral image fusion, and this device includes a spectroscope module, an image processing and fusion module, a model construction and training module, and a defect recognition module.

[0250] The spectroscope module is used to obtain the multi-spectral image of the high-voltage equipment. Specifically, the spectroscope module obtains the full-band spectral information on the surface of the target high-voltage equipment, divides the full-band spectral information into three target optical paths: ultraviolet, infrared, and visible light; and performs optical path processing on the same target high-voltage equipment to obtain the original ultraviolet image, infrared image, and visible light image, and performs optical zoom and electronic zoom control on the original ultraviolet image, infrared image, and visible light image to obtain refined images.

[0251] The image processing and fusion module is used to preprocess the multi-spectral image and perform cross-modal fusion processing on the preprocessed image data. The preprocessing includes cleaning the original data, performing operations such as format conversion and feature engineering, merging multi-source data, and constructing a unified data set. The preprocessing also includes: processing the spectral information of the ultraviolet, infrared, and visible light bands to obtain the discharge intensity, temperature, and appearance defect information on the surface of the target high-voltage device; performing noise suppression and grayscale processing on the original ultraviolet image to obtain a low-noise image; and calculating the low-noise image according to the discharge intensity detection algorithm to obtain the ultraviolet spot image and ultraviolet coordinate information; performing noise reduction processing on the original infrared image to obtain a low-noise image, and calculating the low-noise image according to the infrared heat temperature detection algorithm to obtain the surface temperature information and infrared coordinate information of the target high-voltage device; performing edge detection on the original visible light image, and obtaining the appearance defect information and visible light coordinate information on the surface of the target high-voltage device through texture feature analysis. The temperature and appearance defect information includes: the surface temperature information, coordinate information, and appearance surface defect information of the target high-voltage device. The cross-modal fusion processing of the preprocessed image data includes obtaining the coordinate matching coefficient and offset according to the ultraviolet coordinate information, infrared coordinate information, and visible light coordinate information; comparing the coordinate matching coefficient and offset with a preset threshold. If the coordinate matching coefficient and offset are less than the first threshold, it is determined that the target high-voltage device is in the normal working area; if the coordinate matching coefficient and offset are greater than or equal to the first threshold and less than the second threshold, it is determined that the target high-voltage device is in the warning area; if the coordinate matching coefficient is greater than the second threshold, it is determined that the target high-voltage device is in the critical area.

[0252] The model construction and training module is used to construct a high-voltage device defect detection model based on the YOLO-STrans network and train the high-voltage device defect detection model using the fused image data.

[0253] The defect recognition module is used to obtain the multi-spectral image of the high-voltage device to be detected and identify the defects in the multi-spectral image using the high-voltage device defect detection model.

[0254] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the above division of each functional module is used as an example. In actual applications, the above functions can be allocated to different functional modules according to needs, that is, the internal structure of the system is divided into different functional modules to complete all or part of the functions described above.

[0255] Example 3

[0256] The present invention also provides a processing device, which mainly includes: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the method provided by the foregoing embodiments.

[0257] Further, the processing device further includes at least one input device and at least one output device; in the processing device, the processor, the memory, the input device, and the output device are connected through a bus.

[0258] In the embodiments of the present invention, the specific types of the memory, the input device, and the output device are not limited; for example: the input device may be a touch screen, an image acquisition device, a physical button, or a mouse, etc.; the output device may be a display terminal; the memory may be a Random Access Memory (RAM), or a non-volatile memory, such as a disk memory.

[0259] Embodiment 4

[0260] The present invention also provides a readable storage medium storing a computer program, which implements the method provided by the foregoing embodiments when the computer program is executed by a processor.

[0261] In the embodiments of the present invention, the readable storage medium as a computer-readable storage medium may be disposed in the foregoing processing device, for example, as the memory in the processing device. In addition, the readable storage medium may also be various media that can store program codes, such as a USB flash drive, a mobile hard disk, a Read-Only Memory (ROM), a magnetic disk, or an optical disc.

[0262] The above shows and describes the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments, and the principles described in the above embodiments and the specification are only the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention will have various changes and improvements, and these changes and improvements all fall within the scope of the present invention claimed. The scope of protection claimed by the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for detecting defects in high-voltage equipment based on deep learning and multi-spectral image fusion, characterized in that, It includes the following steps: S1. Obtain the multi-spectral image of the high-voltage equipment and perform preprocessing; S2. Perform cross-modal fusion processing on the preprocessed image data; S3. Construct a high-voltage equipment defect detection model based on the YOLO-STrans network; S4. Use the fused image data to train the high-voltage equipment defect detection model to obtain a high-voltage equipment defect recognition model; S5. Obtain the multi-spectral image of the high-voltage equipment to be detected, and use the high-voltage equipment defect recognition model to identify defects in the multi-spectral image.

2. The method for detecting defects in high-voltage equipment based on deep learning and multi-spectral image fusion according to claim 1, wherein in the step S1, the obtaining the multi-spectral image of the high-voltage equipment and performing preprocessing specifically includes the following steps: S11. The multispectral image includes an ultraviolet image, an infrared image, and a visible light image. Preprocess the ultraviolet image, the infrared image, and the visible light image respectively: synchronously collect the ultraviolet image of the high-voltage equipment in the target area , the infrared image , and the visible light image . Taking the height and width of the image as dimensions, establish a three-channel feature space containing the ultraviolet image, the infrared image, and the visible light image. The three-channel feature space is shown in Equation (1): （1）； Among them, represents the three-channel feature quantity, represents the real number field of the tensor, represents the height of the image, represents the width of the image, is the pixel value of the ultraviolet channel, is the pixel value of the infrared channel, is the pixel value of the visible light channel; S12. Based on the three-channel feature space, preprocess each modal image in the three-channel feature space.

3. The method for detecting defects in high-voltage equipment based on deep learning and multi-spectral image fusion according to claim 2, wherein in the step S2, the performing cross-modal fusion processing on the preprocessed image data specifically includes the following steps: S21. Fuse and superimpose the preprocessed ultraviolet image and visible light image to obtain a first fused and superimposed image; S22. Fuse and superimpose the preprocessed ultraviolet image and infrared image to obtain a second fused and superimposed image.

4. The method for detecting defects in high-voltage equipment based on deep learning and multi-spectral image fusion according to claim 3, wherein the step S21 specifically includes the following steps: S211. Perform edge detection on the preprocessed ultraviolet image and visible light image respectively, and extract the ultraviolet discharge edge image and the high-voltage equipment structure edge image; S212. Extract features from the preprocessed ultraviolet image and visible light image through a dual-channel attention mechanism to generate a first edge weight matrix ; The first edge weight matrix is generated by using an improved YOLOv8_seg network, and the edge weight matrix is as shown in Equation (2): （2）； Among them, represents the first edge weight matrix, represents the Sigmoid activation function, represents the convolution operation, represents the channel concatenation operation, is the pixel value of the ultraviolet channel, is the pixel value of the visible light channel; S213. Use the first edge weight matrix Perform weighted fusion on the ultraviolet discharge edge image and the high-voltage equipment structure edge image to generate a first region to be recognized; the first region to be recognized is the overlapping region of the ultraviolet discharge edge image and the high-voltage equipment structure edge image. S214. Perform multi-scale feature extraction on the first region to be recognized to obtain an ultraviolet light feature image and a visible light feature image, and magnify the ultraviolet light feature image and the visible light feature image; The performing multi-scale feature extraction on the first region to be recognized includes: Construct a dual-branch feature pyramid through a residual network, and use the dual-branch feature pyramid to perform multi-scale feature extraction on the first region to be recognized; The dual-branch feature pyramid is shown in Equation (3): （3）； Among them, represents the ultraviolet image pixel feature, represents the output of the third layer feature extracted by the residual network, represents the visible light image pixel feature, is the pixel value of the ultraviolet channel, is the pixel value of the visible light channel; S215. Adopt a pyramid fusion algorithm with dynamic weight allocation, and use Equation (4) to perform cross-modal fusion on the magnified ultraviolet light feature image and visible light feature image to obtain a first fused and superimposed image: （4）； Among them, is the first fused image pixel value, represents the dynamic weight, and respectively represent the layer Gaussian pyramid and Laplacian pyramid operations; represents the image modality identifier; The dynamic weight is calculated through the attention mechanism shown in formula (5): （5）； Among them, represents the dynamic weight, represents the exponential operation, represents the ultraviolet image pixel feature, and respectively represent the layer Gaussian pyramid and Laplacian pyramid operations, represents the image modality identifier, represents the visible light image pixel feature.

5. The method for detecting defects in high-voltage equipment based on deep learning and multi-spectral image fusion according to claim 4, wherein the step S22 specifically includes the following steps: S221. Perform edge detection on the preprocessed ultraviolet image and infrared image respectively, and extract the ultraviolet discharge edge image and the infrared edge image; S222. Extract features from the preprocessed ultraviolet image and infrared image through a dual-channel attention mechanism to generate a second edge weight matrix ; The second edge weight matrix is generated by using an improved YOLOv8_seg network. The second edge weight matrix is as shown in Equation (6): （6）； Among them, is a dynamically generated edge weight matrix, represents the real number field of the tensor, represents the height of the image, represents the width of the image; is an activation function, which is used to convert the original score of the model into a probability distribution; represents a convolution operation; represents a tensor concatenation; is the pixel value of the ultraviolet channel; is the pixel value of the infrared channel; S223. Use the second edge weight matrix Perform weighted fusion on the ultraviolet discharge edge image and the infrared edge image to generate a second area to be recognized; the second area to be recognized is the overlapping area of the ultraviolet discharge edge image and the infrared edge image; S224. Perform texture analysis and magnification processing on the second region to be recognized to obtain an ultraviolet light detail image and an infrared light detail image, perform feature extraction on the ultraviolet light detail image and the infrared light detail image respectively, and generate a corona intensity feature image and a heat temperature feature image; S225. Using the thermodynamic diffusion equation shown in Equation (9), perform feature coupling across physical domains on the corona intensity feature image and the heat temperature feature image to generate a corona-temperature coupling feature: （9）； Among them, is the heat temperature characteristic, represents time, is the heat conduction coefficient, is the spatial gradient of the heat temperature characteristic ; is the corona-temperature coupling coefficient, is the corona intensity characteristic; S226. Use Equation (10) to construct a feature importance evaluation function, and based on the feature importance evaluation function, use Equation (11) to perform an adaptive amplification process on the corona-temperature coupling feature: （10）； （11）； Among them, represents the enhancement coefficient, is the image pixel coordinate, is the image pixel coordinate at the corona intensity feature, is the image pixel coordinate at the heat temperature feature, is the corona intensity feature before magnification, is the heat temperature feature before magnification, represents the global maximum corona intensity feature value, represents the global maximum heat temperature feature value, is the corona intensity feature after magnification, is the heat temperature feature after magnification; S227. Use equation (12) to perform multi-physics field modeling on the second area to be recognized : （12）； Among them, is the corona intensity feature before amplification, is the corona intensity diffusion function; represents the Laplace operator; represents the corona intensity feature image; represents the exponential operation; represents a hyperparameter, which is set manually or determined through training, and is used to control the attenuation degree of the exponential term and adjust the influence weight of this term in the formula; is the ultraviolet intensity threshold, is the infrared intensity threshold, is the spatial gradient of the heat temperature feature of the infrared channel ; S228. Use the dynamic Alpha coefficient generation algorithm shown in Equation (13) to determine the adaptive fusion weights of the features of each physical field, and use Equation (14) to generate a second fusion superposition image: （13）； （14）； Among them, is the dynamic coefficient, is the activation function, is the magnified corona intensity feature, represents generating the second fusion and superposition image, is the magnified heat temperature feature, represents the function for taking the global maximum value from the entire image; S229. Use Equation (15) to establish a temperature-corona intensity correlation model, and based on the temperature-corona intensity correlation model, perform hot spot detection on the second coupling superposition image to determine the fault type in the second coupling superposition image; （15）； Among them, represents the pixel value of the output corona intensity characteristic image or ultraviolet characteristic image at the image pixel coordinates ; represents the metric value of the ultraviolet image at the image pixel coordinates ; , is the abnormal determination coefficient, is the ultraviolet intensity threshold; The abnormal determination coefficient , is calculated by using the adaptive algorithm shown in Equation (16): （16）； Among them, is the mean value of the ultraviolet image, is the standard deviation of the ultraviolet image.

6. The method for detecting defects in high-voltage equipment based on deep learning and multi-spectral image fusion according to claim 5, wherein in step S224, the feature extraction of the ultraviolet light detail image and the infrared light detail image respectively to generate a corona intensity feature image and a heat temperature feature image specifically includes the following steps: Construct a multi-branch residual network using Equation (7), determine a three-path feature extraction structure including a shallow detail branch, a deep semantic branch, and a cross-modal interaction branch, and perform multi-scale feature extraction on the ultraviolet light detail image and the infrared light detail image through a channel attention module and a spatial-channel joint attention mechanism, and output a corona intensity feature image and a heat temperature feature image: （7）； Among them, represents the corona intensity feature image, represents the heat temperature feature image, represents the output of the fourth layer of features extracted by the residual network, is the channel attention module, is the spatial-channel joint attention mechanism, is the pixel value of the ultraviolet channel, is the pixel value of the infrared channel; In step S224, the corona intensity feature image is an area where the ultraviolet light intensity extracted from the ultraviolet light detail image is greater than the ultraviolet light intensity threshold, including: Construct a dynamic threshold calculation model using Equation (8), and based on the dynamic threshold calculation model, segment the area where the ultraviolet light intensity in the ultraviolet detail image is greater than the ultraviolet light intensity threshold to generate a corona intensity feature image: （8）； Among them, represents the ultraviolet intensity threshold, is the mean value of the ultraviolet image, is the standard deviation of the ultraviolet image, is the curvature adjustment coefficient; represents with respect to direction and the sum of the second-order partial derivatives in the direction.

7. The method for detecting defects in high-voltage equipment based on deep learning and multi-spectral image fusion according to claim 1, wherein the high-voltage equipment defect detection model is implemented based on the YOLO-STrans network, and the YOLO-STrans network is constructed based on the improved YOLO and Transformer networks; the high-voltage equipment defect detection model includes a multi-spectral input end, a CSP-YOLO detection module, a Transformer target extractor, a dynamic fusion module, a multi-task classifier, and a multi-scale output end; the multi-spectral input end is used to input multi-spectral images, that is, the collected ultraviolet images, infrared images, and visible light images, and preprocess the input images using the spectral normalization module shown in Equation (17): （17）； Among them, represents the pixel value of the th class of images after normalization; represents the pixel value of the original th class of images; , are the mean and standard deviation of each channel respectively; takes the value of ; represents the image modality identifier, represents the ultraviolet image, represents the infrared image, represents the visible light image; The CSP-YOLO detection module is used to construct an improved CSPDarknet53 network by using Equation (18). A deformable convolution is introduced into the backbone network of CSPDarknet53 to obtain the improved CSPDarknet53 network, and the improved CSPDarknet53 network is used for feature extraction: （18）； wherein, represents the value of the output signal or feature at the position variable ; is the position offset within the neighborhood; is the neighborhood set; is the weight function; is the input signal; is the dynamically learned position offset; The Transformer object extractor is used to receive the multi-modal features output by the CSP-YOLO detection module, and calculate the semantic association between any positions in the feature map through the multi-head self-attention mechanism to determine the global dependence relationship; The multi-head attention mechanism is shown in Equation (19): （19）； Among them, is the result of processing the ultraviolet, visible, and infrared spectral images by the multi-head attention mechanism, is the connection operation function, represents the output projection weight matrix; Attention head Calculated using Equation (20): （20）； Among them, represents the attention output of the th head; is an activation function used to convert the raw scores of the model into a probability distribution; represents the query matrix of the attention head ; represents the key matrix of the attention head ; represents the transpose operator, represents the value matrix of the attention head ; represents the key matrix dimension; The dynamic fusion module is used to construct a two-stream feature fusion mechanism by using Equation (21), adaptively generate channel attention weights based on SENet, and combine with the deformable convolution network to achieve cross-modal complementarity and spatial alignment of multi-modal features of ultraviolet, infrared, and visible light: （21）； Among them, is the dynamic fusion eigenvalue, is the dynamic coefficient, is the input eigenvalue of the YOLO network, is the input eigenvalue of the Transformer network, is the deformable convolutional network; The multi-task classifier classifies defects and identifies faults in the high-voltage equipment area extracted by the Transformer object extractor; the multi-task classifier constructs a depthwise separable convolution parallel output branch by using Equation (22) to classify defects and identify faults in the equipment area output by the Transformer object extractor: （22）； Among them, is the multi-dimensional classification and localization loss value, is the position loss weight value, is the position loss value, is the classification loss weight value, is the classification loss value; The multi-scale output end is based on the feature pyramid and uses Equation (23) to detect defects at different scales and outputs a three-channel detection result: （23）； Among them, represents the diagnostic result, denotes operation, denotes the discharge probability; is the activation function, denotes the probability of abnormal temperature, denotes the probability of appearance defect.

8. A high-voltage equipment defect detection device based on deep learning and multispectral image fusion, characterized in that, The device includes a spectroscope module, an image processing and fusion module, a model construction and training module, and a defect identification module; The spectroscope module is used to obtain multi-spectral images of high-voltage equipment; The image processing and fusion module is used to preprocess the multi-spectral images and perform cross-modal fusion processing on the preprocessed image data; The model construction and training module is used to construct a high-voltage equipment defect detection model based on the YOLO-STrans network, and use the fused image data to train the high-voltage equipment defect detection model to obtain a high-voltage equipment defect identification model; The defect identification module is used to obtain the multi-spectral images of the high-voltage equipment to be detected and use the high-voltage equipment defect identification model to identify the defects in the multi-spectral images.

9. An electronic device, characterized in that, Comprising: At least one processor; And a memory that stores instructions, which when executed by the at least one processor, cause the at least one processor to execute the method for detecting high-voltage equipment defects based on deep learning and multi-spectral image fusion according to any one of claims 1 to 7.

10. A machine-readable storage medium, characterized in that, It stores executable instructions that, when executed, cause the machine to execute the method for detecting high-voltage equipment defects based on deep learning and multi-spectral image fusion according to any one of claims 1 to 7.

Images