High-voltage equipment defect detection method based on deep learning and multispectral image fusion

Abstract

The present invention relates to a high-voltage equipment defect detection method based on deep learning and multispectral image fusion, and relates to the technical field of electric high-voltage equipment status detection. The method includes: obtaining ultraviolet images, infrared images, and visible light images of the surface of the high-voltage equipment; performing fusion processing on the ultraviolet images, infrared images, and visible light images based on image pixel features through an image fusion method; establishing a high-voltage equipment defect detection model, using the fused image data to train the high-voltage equipment defect detection model, and obtaining a high-voltage equipment defect recognition model based on the YOLO‑STrans multispectral fusion network; inputting the ultraviolet image, infrared image, and visible light image of the outer surface of the electric high-voltage equipment into the high-voltage equipment defect recognition model, and obtaining the fault recognition result of the electric high-voltage equipment to be detected. The present invention can improve the recognition accuracy of extremely early insulation degradation and temperature anomaly defects on the surface of electric high-voltage equipment.

Description

Technical Field

[0001] The present invention relates to the technical field of electric high-voltage equipment status detection, and specifically to a high-voltage equipment defect detection method based on deep learning and multispectral image fusion. Background Art

[0002] High-voltage equipment, as a vital component of the power system, can experience ionization discharges during operation due to insulation degradation, moisture, and other factors, leading to equipment and system failures. To ensure stable power system operation, real-time monitoring of high-voltage equipment defects is essential.

[0003] Currently, conventional high-voltage equipment defect detection relies on visible light and infrared detection, along with a regular maintenance system. By the time visible light and infrared detection detect equipment defects, they are often already severely deteriorating, making it impossible to provide timely monitoring and early warning of ongoing deterioration trends.

[0004] Therefore, there is an urgent need for a high-voltage equipment defect detection technology that can solve the problem that existing detection methods are unable to monitor and warn of equipment defects and degradation in the early stages, so as to more accurately identify defects such as extremely early insulation degradation and temperature anomalies on the surface of high-voltage equipment. Summary of the Invention

[0005] The present invention is based on a high-voltage equipment defect detection method based on the fusion of deep learning and multispectral images. By fusing ultraviolet, infrared and visible light information, it can more accurately identify defects such as extremely early insulation degradation and temperature anomalies on the surface of high-voltage equipment. It provides a high-voltage equipment defect detection method based on the fusion of deep learning and multispectral images to solve the above problems.

[0006] In order to achieve the above object, the technical solution of the present invention is as follows:

[0007] In a first aspect of the present invention, a method for high-voltage equipment defect detection based on deep learning and multispectral image fusion is provided, the method comprising the following steps:

[0008] S1. Acquire multispectral images of high-voltage equipment and perform preprocessing;

[0009] S2, performing cross-modal fusion processing on the pre-processed image data;

[0010] S3. Build a high-voltage equipment defect detection model based on the YOLO-STrans network;

[0011] S4. Using the fused image data, a high-voltage equipment defect detection model is trained to obtain a high-voltage equipment defect recognition model;

[0012] S5. Obtain a multispectral image of the high-voltage equipment to be inspected, and perform defect recognition on the multispectral image using a high-voltage equipment defect recognition model.

[0013] Furthermore, in step S1, the multispectral image of the high-voltage equipment is obtained and preprocessed, specifically including the following steps:

[0014] S11, the multispectral image includes an ultraviolet image, an infrared image and a visible light image, and the ultraviolet image, infrared image and visible light image are preprocessed respectively: the ultraviolet image of the high-voltage equipment in the target area is synchronously collected. , infrared images and visible light images , and using the height and width of the image as dimensions, a three-channel feature space including ultraviolet image, infrared image and visible light image is established; the three-channel feature space is shown in formula (1):

[0015] (1);

[0016] in, represents the three-channel feature quantity, represents the tensor real field, Indicates the height of the image, Indicates the width of the image, is the pixel value of the UV channel, is the pixel value of the infrared channel, is the pixel value of the visible light channel;

[0017] S12. Preprocess each modality image in the three-channel feature space based on the three-channel feature space.

[0018] Furthermore, in step S2, the cross-modal fusion processing of the pre-processed image data specifically includes the following steps:

[0019] S21, fusing and superimposing the pre-processed ultraviolet image and the visible light image to obtain a first fused superimposed image;

[0020] S22: Fusing and superimposing the pre-processed ultraviolet image and infrared image to obtain a second fused superimposed image.

[0021] Furthermore, the step S21 specifically includes the following steps:

[0022] S211, performing edge detection on the pre-processed ultraviolet image and visible light image respectively, and extracting the ultraviolet discharge edge image and the high-voltage equipment structure edge image;

[0023] S212: Extract features from the preprocessed UV image and visible light image through a dual-channel attention mechanism to generate a first edge weight matrix ;

[0024] The first edge weight matrix The edge weight matrix is ​​generated using the improved YOLOv8_seg network. As shown in formula (2):

[0025] (2);

[0026] in, represents the first edge weight matrix, represents the Sigmoid activation function, represents the convolution operation, Indicates channel splicing operation, is the pixel value of the UV channel, is the pixel value of the visible light channel;

[0027] S213, using the first edge weight matrix Performing weighted fusion on the ultraviolet discharge edge image and the high-voltage equipment structure edge image to generate a first area to be identified; the first area to be identified is an area where the ultraviolet discharge edge image and the high-voltage equipment structure edge image overlap;

[0028] S214, performing multi-scale feature extraction on the first to-be-identified area to obtain an ultraviolet feature image and a visible light feature image, and performing a magnification process on the ultraviolet feature image and the visible light feature image;

[0029] The performing multi-scale feature extraction on the first area to be identified includes:

[0030] Constructing a dual-branch feature pyramid through a residual network, and using the dual-branch feature pyramid to extract multi-scale features of the first area to be identified;

[0031] The dual-branch feature pyramid is shown in formula (3):

[0032] (3);

[0033] in, represents the pixel features of the UV image, express The third layer feature output extracted by the residual network, Represents the pixel features of visible light images, is the pixel value of the UV channel, is the pixel value of the visible light channel;

[0034] S215, using a pyramid fusion algorithm with dynamic weight allocation, and using formula (4) to perform cross-modal fusion on the amplified ultraviolet feature image and the visible light feature image to obtain a first fused superimposed image:

[0035] (4);

[0036] in, is the pixel value of the first fused image, represents the dynamic weight, and Respectively represent Layer Gaussian pyramid and Laplacian pyramid operations; Indicates the image modality identifier;

[0037] The dynamic weight The attention mechanism shown in formula (5) is used to calculate:

[0038] (5);

[0039] in, represents the dynamic weight, represents exponential operation, represents the pixel features of the UV image, and Respectively represent Layer Gaussian pyramid and Laplacian pyramid operations, Indicates the image modality identifier, Represents the pixel features of visible light images.

[0040] Furthermore, the step S22 specifically includes the following steps:

[0041] S221, performing edge detection on the pre-processed ultraviolet image and infrared image respectively, and extracting the ultraviolet discharge edge image and the infrared edge image;

[0042] S222, extract features from the preprocessed ultraviolet image and infrared image through a dual-channel attention mechanism to generate a second edge weight matrix ;

[0043] The second edge weight matrix The second edge weight matrix is ​​generated using the improved YOLOv8_seg network As shown in formula (6):

[0044] (6);

[0045] in, is the dynamically generated edge weight matrix, represents the tensor real field, Indicates the height of the image, Indicates the width of the image; for Activation function, used to convert the model's raw scores into probability distributions; Represents the convolution operation; Represents tensor concatenation; is the pixel value of the UV channel; is the pixel value of the infrared channel;

[0046] S223, using the second edge weight matrix Performing weighted fusion on the ultraviolet discharge edge image and the infrared edge image to generate a second area to be identified; the second area to be identified is an area where the ultraviolet discharge edge image and the infrared edge image overlap;

[0047] S224, performing texture analysis and amplification processing on the second area to be identified to obtain an ultraviolet detail image and an infrared detail image, performing feature extraction on the ultraviolet detail image and the infrared detail image respectively to generate a corona intensity feature image and a heat temperature feature image;

[0048] S225. Using the thermodynamic diffusion equation shown in formula (9), perform feature coupling across physical domains on the corona intensity feature image and the heat temperature feature image to generate a corona-temperature coupling feature:

[0049] (9);

[0050] in, is the heat temperature characteristic, Indicates time, is the thermal conductivity coefficient, Thermal temperature characteristics The spatial gradient of is the corona-temperature coupling coefficient, is the corona intensity characteristic;

[0051] S226. Construct a feature importance evaluation function using formula (10), and based on the feature importance evaluation function, perform adaptive amplification processing on the corona-temperature coupling feature using formula (11):

[0052] (10);

[0053] (11);

[0054] in, represents the enhancement coefficient, is the image pixel coordinate, is the image pixel coordinate The corona intensity characteristics at is the image pixel coordinate The thermal temperature characteristics at is the corona intensity characteristic before amplification, is the heat temperature characteristic before amplification, represents the global maximum corona intensity eigenvalue, represents the global maximum heat temperature eigenvalue, is the amplified corona intensity characteristic, is the amplified heat temperature characteristic;

[0055] S227, using formula (12) to identify the second area Perform multiphysics modeling:

[0056] (12);

[0057] in, is the corona intensity characteristic before amplification, is the corona intensity diffusion function; represents the Laplace operator; Represents the corona intensity characteristic image; Represents exponential operation; Represents a hyperparameter, which is set manually or determined through training. It is used to control the attenuation of the exponential term and adjust the influence weight of the term in the formula. is the UV intensity threshold, is the infrared intensity threshold, Thermal temperature characteristics of the infrared channel spatial gradients;

[0058] S228. Determine the adaptive fusion weights of the physical field features using the dynamic Alpha coefficient generation algorithm shown in formula (13), and generate a second fused superposition image using formula (14):

[0059] (13);

[0060] (14);

[0061] in, is the dynamic coefficient, for activation function, is the amplified corona intensity characteristic, Indicates the generation of the second fused overlay image, is the amplified heat temperature characteristic, express Function, used to get the global maximum value from the entire image;

[0062] S229, using formula (15) to establish a temperature-corona intensity correlation model, performing hot spot detection on the second coupled superposition image based on the temperature-corona intensity correlation model, and determining the fault type in the second coupled superposition image;

[0063] (15);

[0064] in, Indicates the output corona intensity characteristic image or ultraviolet characteristic image at the image pixel coordinate The pixel value at Indicates the UV image at the image pixel coordinates The measurement value at , is the abnormality determination coefficient, is the UV intensity threshold;

[0065] The abnormality determination coefficient , calculated using the adaptive algorithm shown in formula (16):

[0066] (16);

[0067] in, is the UV image mean, is the standard deviation of the UV image.

[0068] Furthermore, in step S224, the feature extraction is performed on the ultraviolet detail image and the infrared detail image respectively to generate the corona intensity feature image and the heat temperature feature image, which specifically includes the following steps:

[0069] A multi-branch residual network is constructed using Equation (7), and a three-path feature extraction structure including a shallow detail branch, a deep semantic branch, and a cross-modal interaction branch is determined. Multi-scale feature extraction is performed on the ultraviolet detail image and the infrared detail image through the channel attention module and the spatial-channel joint attention mechanism, and the corona intensity feature image and the heat temperature feature image are output:

[0070] (7);

[0071] in, represents the corona intensity characteristic image, Represents the heat temperature feature image, express The fourth layer feature output extracted by the residual network, is the channel attention module, is the spatial-channel joint attention mechanism, is the pixel value of the UV channel, is the pixel value of the infrared channel;

[0072] In step S224, the corona intensity characteristic image is extracted from the ultraviolet detail image, where the ultraviolet intensity is greater than the ultraviolet intensity threshold, and includes:

[0073] Formula (8) is used to construct a dynamic threshold calculation model. Based on the dynamic threshold calculation model, the area with ultraviolet light intensity greater than the ultraviolet light intensity threshold is segmented from the ultraviolet detail image to generate the corona intensity feature image:

[0074] (8);

[0075] in, represents the UV intensity threshold, is the mean value of the UV image, is the standard deviation of the UV image, is the curvature adjustment coefficient; express about Direction and The sum of the second-order partial derivatives in the direction.

[0076] Furthermore, the high-voltage equipment defect detection model is implemented based on the YOLO-STrans network, which is built based on the improved YOLO and Transformer networks;

[0077] The high-voltage equipment defect detection model includes a multispectral input terminal, a CSP-YOLO detection module, a Transformer target extractor, a dynamic fusion module, a multi-task classifier, and a multi-scale output terminal;

[0078] The multispectral input terminal is used to input a multispectral image, i.e., the collected ultraviolet image, infrared image, and visible light image, and preprocess the input image using the spectral normalization module shown in formula (17):

[0079] (17);

[0080] in, Indicates the The pixel value of the class image after normalization; Represents the original Class image pixel value; 、 are the mean and standard deviation of each channel respectively; The value of ; Indicates the image modality identifier, represents the UV image, represents an infrared image, Represents a visible light image;

[0081] The CSP-YOLO detection module is used to construct an improved CSPDarknet53 network using formula (18), introduce deformable convolution into the backbone network of CSPDarknet53 to obtain an improved CSPDarknet53 network, and use the improved CSPDarknet53 network for feature extraction:

[0082] (18);

[0083] in, Indicates the output signal or feature at the position variable The value at is the position offset within the neighborhood; is the neighborhood set; is the weight function; is the input signal; is the position offset of dynamic learning;

[0084] The Transformer object extractor is used to receive the multimodal features output by the CSP-YOLO detection module and calculate the semantic association between any positions in the feature map through a multi-head self-attention mechanism to determine the global dependency relationship;

[0085] The multi-head attention mechanism is shown in formula (19):

[0086] (19);

[0087] in, This is the result of multi-head attention mechanism processing ultraviolet, visible and infrared spectral images. For the connection operation function, represents the output projection weight matrix;

[0088] Attention Head Calculate using formula (20):

[0089] (20);

[0090] in, Indicates the Attention output of size; is the activation function used to convert the raw scores of the model into a probability distribution; Represents the attention head The query matrix; Represents the attention head The bond matrix, represents the transpose operator, Represents the attention head The value matrix of represents the bond matrix dimension;

[0091] The dynamic fusion module is used to construct a dual-stream feature fusion mechanism using formula (21), adaptively generate channel attention weights based on SENet, and combine with deformable convolutional networks to achieve cross-modal complementarity and spatial alignment of ultraviolet, infrared, and visible light multimodal features:

[0092] (twenty one);

[0093] in, is the dynamic fusion eigenvalue, is the dynamic coefficient, To input the yolo network feature value, is the input Transformer network feature value, It is a deformable convolutional network;

[0094] The multi-task classifier performs defect classification and fault identification on the high-voltage equipment area extracted by the Transformer target extractor. The multi-task classifier uses formula (22) to construct a depth-separable convolution parallel output branch to perform defect classification and fault identification on the equipment area output by the Transformer target extractor:

[0095] (twenty two);

[0096] in, Positioning loss value for multi-dimensional classification, is the position loss weight value, is the position loss value, is the classification loss weight value, is the classification loss value;

[0097] The multi-scale output end detects defects of different scales based on the feature pyramid using formula (23) and outputs three-channel detection results:

[0098] (twenty three);

[0099] in, Represents the diagnosis result, express Operation, represents the discharge probability; is the activation function, represents the probability of temperature anomaly, Represents the probability of appearance defects.

[0100] In a second aspect of the present invention, a high-voltage equipment defect detection device based on deep learning and multispectral image fusion is disclosed, which includes a spectroscopic module, an image processing and fusion module, a model construction and training module, and a defect recognition module. The spectroscopic module is used to obtain multispectral images of high-voltage equipment. The image processing and fusion module is used to preprocess the multispectral images and perform cross-modal fusion processing on the preprocessed image data. The model construction and training module is used to construct a high-voltage equipment defect detection model based on the YOLO-STrans network, and use the fused image data to train the high-voltage equipment defect detection model to obtain a high-voltage equipment defect recognition model. The defect recognition module is used to obtain a multispectral image of the high-voltage equipment to be detected, and use the high-voltage equipment defect recognition model to perform defect recognition on the multispectral image.

[0101] In a third aspect of the present invention, an electronic device is disclosed, comprising: at least one processor; and a memory, wherein the memory stores instructions, and when the instructions are executed by the at least one processor, the at least one processor executes the above-mentioned high-voltage equipment defect detection method based on deep learning and multispectral image fusion.

[0102] In a fourth aspect of the present invention, a machine-readable storage medium is disclosed, which stores executable instructions. When the instructions are executed, the machine performs the above-mentioned high-voltage equipment defect detection method based on deep learning and multispectral image fusion.

[0103] Beneficial effects

[0104] Compared with the prior art, the high-voltage equipment defect recognition method described in the present invention obtains ultraviolet images, visible images, and infrared images of the high-voltage equipment, and fuses and superimposes the obtained ultraviolet images and visible light images to obtain a first fused superimposed image, and fuses and superimposes the ultraviolet images and infrared images to obtain a second fused superimposed image; the first fused superimposed image and the second fused superimposed image are processed using a pre-trained high-voltage equipment defect detection model to ultimately obtain an identification result of the equipment status. The present invention achieves multi-dimensional and multi-level analysis of high-voltage equipment by fusing and superimposing ultraviolet images and visible light images, as well as fusing and superimposing ultraviolet images and infrared images, thereby avoiding omission of image information; furthermore, the result of image fusion and superposition is input into a pre-trained high-voltage equipment defect detection model to achieve automated processing, thereby solving the difficulties of manual inspections while ensuring the accuracy of inspection results, and ultimately achieving accurate identification of the high-voltage equipment status. BRIEF DESCRIPTION OF THE DRAWINGS

[0105] Figure 1is a flow chart of a high-voltage equipment defect detection method based on deep learning and multispectral image fusion in the present invention;

[0106] Figure 2 This is a flow chart of the method for obtaining discharge intensity, temperature, and appearance defects from multispectral images in the present invention;

[0107] Figure 3 This is a diagram of the architecture of the high-voltage equipment defect detection and identification model of the present invention;

[0108] Figure 4 This is an architecture diagram of the CSP-YOLO detection module in the high-voltage equipment defect detection model of the present invention;

[0109] Figure 5 This is the architecture diagram of the Transformer target extraction classifier in the high-voltage equipment defect detection model of the present invention;

[0110] Figure 6 This is a structural block diagram of the high-voltage equipment defect detection device based on deep learning and multispectral image fusion in the present invention. DETAILED DESCRIPTION

[0111] In order to provide a further understanding and appreciation of the structural features and effects achieved by the present invention, a detailed description is provided with reference to preferred embodiments and accompanying drawings as follows:

[0112] Example 1

[0113] This embodiment provides a high-voltage equipment defect detection method based on deep learning and multispectral image fusion. The method first obtains spectral image information of the target high-voltage equipment in three bands: ultraviolet, infrared, and visible light. Feature extraction is then performed on the spectral image information of the three bands, and ultraviolet corona intensity, equipment surface temperature, and appearance defects are obtained through image analysis. Next, the ultraviolet image and the visible light image are fused and superimposed to obtain a first fused superimposed image. The ultraviolet image and the infrared image are fused and superimposed to obtain a second fused superimposed image. A high-voltage equipment defect detection model is constructed, and the first fused superimposed image and the second fused superimposed image are input into a pre-trained high-voltage equipment defect detection model. The high-voltage equipment defect detection model outputs a fault identification result of the target line in the target section.

[0114] like Figure 1~Figure 2 As shown, the high-voltage equipment defect detection method based on deep learning and multispectral image fusion described in the present invention includes the following steps:

[0115] S1. Acquire multispectral images of high-voltage equipment and perform preprocessing.

[0116] A multispectral image of a high-voltage device is obtained as an original image, wherein the multispectral image includes an ultraviolet image, an infrared image, and a visible light image, and the ultraviolet image, the infrared image, and the visible light image are preprocessed respectively.

[0117] The multispectral image includes ultraviolet images, infrared images, and visible light images. In this embodiment, professional imaging equipment such as ultraviolet imagers, infrared thermal imagers, and high-definition visible light cameras are used to capture images of high-voltage equipment. The high-voltage equipment in this embodiment includes ultra-high voltage equipment. The ultraviolet imager is used to capture ultraviolet light signals generated by corona discharge on the surface of the equipment. The ultraviolet light signal can reflect the insulation status of the equipment, especially potential partial discharge phenomena. The infrared thermal imager detects thermal radiation from the surface of the equipment to obtain a temperature distribution image of the equipment, thereby identifying overheating defects caused by poor contact, overload, etc. The high-definition visible light camera is used to record the appearance details of the high-voltage equipment, such as cracks, deformation, and foreign matter attachment. During the acquisition process, it is necessary to ensure that the shooting angle, distance, and time of the three devices are consistent to ensure that the acquired images are consistent in space and time, facilitating subsequent processing.

[0118] In step S1, a multispectral image of the high-voltage equipment is obtained as an original image, wherein the multispectral image includes an ultraviolet image, an infrared image, and a visible light image. Preprocessing the ultraviolet image, the infrared image, and the visible light image is performed respectively, specifically comprising the following steps:

[0119] S1. Acquire multispectral images of high-voltage equipment and perform preprocessing.

[0120] A multispectral image of a high-voltage device is obtained as an original image, wherein the multispectral image includes an ultraviolet image, an infrared image, and a visible light image, and the ultraviolet image, the infrared image, and the visible light image are preprocessed respectively.

[0121] The multispectral image includes ultraviolet images, infrared images, and visible light images. In this embodiment, professional imaging equipment such as ultraviolet imagers, infrared thermal imagers, and high-definition visible light cameras are used to capture images of high-voltage equipment. The high-voltage equipment in this embodiment includes ultra-high voltage equipment. The ultraviolet imager is used to capture ultraviolet light signals generated by corona discharge on the surface of the equipment. The ultraviolet light signal can reflect the insulation status of the equipment, especially potential partial discharge phenomena. The infrared thermal imager detects thermal radiation from the surface of the equipment to obtain a temperature distribution image of the equipment, thereby identifying overheating defects caused by poor contact, overload, etc. The high-definition visible light camera is used to record the appearance details of the high-voltage equipment, such as cracks, deformation, and foreign matter attachment. During the acquisition process, it is necessary to ensure that the shooting angle, distance, and time of the three devices are consistent to ensure that the acquired images are consistent in space and time, facilitating subsequent processing.

[0122] In step S1, a multispectral image of the high-voltage equipment is obtained as an original image, wherein the multispectral image includes an ultraviolet image, an infrared image, and a visible light image. Preprocessing the ultraviolet image, the infrared image, and the visible light image is performed respectively, specifically comprising the following steps:

[0123] S11, the multispectral image includes an ultraviolet image, an infrared image and a visible light image, and the ultraviolet image, infrared image and visible light image are preprocessed respectively: the ultraviolet image of the high-voltage equipment in the target area is synchronously collected. , infrared images and visible light images , and using the height and width of the image as dimensions, a three-channel feature space including ultraviolet image, infrared image and visible light image is established; the three-channel feature space is shown in formula (1):

[0124] (1);

[0125] in, represents the three-channel feature quantity, represents the tensor real field, Indicates the height of the image, Indicates the width of the image, is the pixel value of the UV channel, is the pixel value of the infrared channel, is the pixel value of the visible light channel.

[0126] In the detection of high-voltage equipment, firstly, ultraviolet images, infrared images and visible light images are collected synchronously for the target area of ​​high-voltage equipment (such as insulator strings, transformer bushings, etc.). Assume that the spatial resolution of the three types of images is the same. To ensure the consistency of image data, the existing spatiotemporal registration technology is used when collecting images. Through geometric correction and time synchronization triggering devices, the three types of images are aligned at the pixel level in space and time dimensions, that is, the coordinates of the same physical location in different images are the same. After completing image collection and alignment, based on the height of the image and width dimension, constructing a three-channel feature space containing ultraviolet, infrared and visible light information The ultraviolet image, infrared image and visible light image are combined along the channel dimension to obtain a In the three-channel feature space, the pixel value of the ultraviolet channel is Characterizes the corona discharge intensity at the corresponding position. The higher the grayscale value, the more intense the discharge. The pixel value of the infrared channel The temperature value corresponding to the infrared band is obtained by inversion of Planck's law; the pixel value of the visible light channel The three-channel feature space is a visible light RGB pixel value that reflects the appearance, texture, and structural details of the device. This three-channel feature space can subsequently be directly used as input for deep learning models, allowing the model to utilize multimodal information for defect detection and analysis of high-voltage equipment. In step S11, images are synchronously captured using ultraviolet, infrared, and visible light cameras to establish a three-channel feature space. Hardware synchronization triggering and optical calibration ensure spatiotemporal alignment.

[0127] S12. Preprocess each modality image in the three-channel feature space based on the three-channel feature space.

[0128] Channel refers to the dimensional division of data in the feature space. Each channel in the three-channel feature space corresponds to image data of one modality. Modality refers to the physical properties of the data or the differences in imaging principles. For example, ultraviolet, infrared, and visible light belong to different spectral modalities. Each channel carries image information of one modality, and there is no channel mixing or modality missing. In this embodiment, the modal images in the three-channel feature space include ultraviolet images, infrared images, and visible light images. Existing preprocessing technologies are used to perform noise processing, image enhancement, image geometric correction, and size normalization on the ultraviolet images, infrared images, and visible light images, respectively.

[0129] (1) Noise processing

[0130] Due to factors such as noise from the imaging device and environmental interference, various noises exist in the original image. The original ultraviolet image, original visible light image, and original infrared image in the three-channel feature space are denoised respectively to obtain high-quality ultraviolet images, visible light images, and infrared images. For multiplicative noise and salt and pepper noise in the image, the existing Wiener filtering algorithm can be used for denoising. For Gaussian noise, the existing median filtering algorithm can be used for denoising. Through denoising, clearer ultraviolet images, visible light images, and infrared images can be obtained. Preferably, for ultraviolet images and visible light images, the median filtering algorithm is used to remove salt and pepper noise. By replacing the value of each pixel with the median of the pixel values ​​in its neighborhood, isolated noise points are effectively suppressed. For infrared images, due to their sensitivity to temperature, the Gaussian filtering algorithm is used to smooth the image to reduce the influence of thermal noise while retaining the edge information of temperature changes.

[0131] (2) Image enhancement

[0132] To highlight key information within an image, image enhancement processing is required. For ultraviolet images, histogram equalization is used to expand the image's grayscale dynamic range, enhancing the contrast between the discharge area and the background. For infrared images, a linear stretching algorithm is used to map temperature data to a more appropriate grayscale range, making it easier to observe temperature differences. For visible light images, an adaptive local contrast enhancement algorithm is used to adjust the contrast of different areas based on their local characteristics, making the device's appearance details clearer.

[0133] (3) Image geometric correction

[0134] The original image may be geometrically distorted due to factors such as shooting angle and lens distortion. Visible and infrared images are geometrically corrected using perspective transformation and lens distortion correction algorithms to ensure they conform to the device's true geometry. For UV images, where the primary focus is on the location and intensity of discharge points, simple rotation and translation corrections are performed where necessary to ensure spatial alignment with the other images.

[0135] (4) Size adjustment

[0136] The images are resized to a fixed size, such as 512×512 pixels, for subsequent processing and input into the high-voltage equipment defect detection model. The pixel values ​​are also normalized, mapping them to the range [0, 1] or [-1, 1]. This eliminates differences in brightness and numerical scale between images and improves the stability of subsequent feature extraction and model training.

[0137] S2. Perform cross-modal fusion processing on the preprocessed image data.

[0138] The pre-processed ultraviolet image, infrared image and visible light image are cross-modally fused to generate a fused feature image, which includes a first fused overlay image and a second fused overlay image.

[0139] In step S2, the cross-modal fusion processing of the pre-processed ultraviolet image, infrared image and visible light image specifically includes the following steps:

[0140] S21 , fusing and superimposing the preprocessed ultraviolet image and the visible light image to obtain a first fused and superimposed image.

[0141] In step S21, the fusion and superposition of the pre-processed ultraviolet image and the visible light image specifically include the following steps:

[0142] S211 , performing edge detection on the pre-processed ultraviolet image and visible light image respectively, and extracting the ultraviolet discharge edge image and the high-voltage equipment structure edge image.

[0143] S212: Extract features from the preprocessed UV image and visible light image through a dual-channel attention mechanism to generate a first edge weight matrix .

[0144] The first edge weight matrix The edge weight matrix is ​​generated using the improved YOLOv8_seg network. As shown in formula (2):

[0145] (2);

[0146] in, represents the first edge weight matrix, represents the Sigmoid activation function, represents the convolution operation, Indicates channel splicing operation, is the pixel value of the UV channel, is the pixel value of the visible light channel.

[0147] S213, using the first edge weight matrix The ultraviolet discharge edge image and the high-voltage equipment structure edge image are weightedly fused to generate a first area to be identified; the first area to be identified is an area where the ultraviolet discharge edge image and the high-voltage equipment structure edge image overlap.

[0148] S214 , performing multi-scale feature extraction on the first area to be identified to obtain an ultraviolet light feature image and a visible light feature image, and performing a magnification process on the ultraviolet light feature image and the visible light feature image.

[0149] The performing multi-scale feature extraction on the first area to be identified includes:

[0150] Constructing a dual-branch feature pyramid through a residual network, and using the dual-branch feature pyramid to extract multi-scale features of the first area to be identified;

[0151] The dual-branch feature pyramid is shown in formula (3):

[0152] (3);

[0153] in, represents the pixel features of the UV image, express The third layer feature output extracted by the residual network, Represents the pixel features of visible light images, is the pixel value of the UV channel, is the pixel value of the visible light channel.

[0154] S215, using a pyramid fusion algorithm with dynamic weight allocation, and using formula (4) to perform cross-modal fusion on the amplified ultraviolet feature image and the visible light feature image to obtain a first fused superimposed image:

[0155] (4);

[0156] in, is the pixel value of the first fused image, represents the dynamic weight, and Respectively represent Layer Gaussian pyramid and Laplacian pyramid operations; Indicates the image modality identifier.

[0157] The dynamic weight The attention mechanism shown in formula (5) is used to calculate:

[0158] (5);

[0159] in, represents the dynamic weight, represents exponential operation, represents the pixel features of the UV image, and Respectively represent Layer Gaussian pyramid and Laplacian pyramid operations, Indicates the image modality identifier, Represents the pixel features of visible light images.

[0160] S22: Fusing and superimposing the pre-processed ultraviolet image and infrared image to obtain a second fused superimposed image.

[0161] In step S22, the pre-processed ultraviolet image and infrared image are fused and superimposed to obtain a second fused superimposed image, which specifically includes the following steps:

[0162] S221 , performing edge detection on the pre-processed ultraviolet image and infrared image respectively, and extracting an ultraviolet discharge edge image and an infrared edge image.

[0163] S222, extract features from the preprocessed ultraviolet image and infrared image through a dual-channel attention mechanism to generate a second edge weight matrix .

[0164] The second edge weight matrix The second edge weight matrix is ​​generated using the improved YOLOv8_seg network As shown in formula (6):

[0165] (6);

[0166] in, is the dynamically generated edge weight matrix, represents the tensor real field, Indicates the height of the image, Indicates the width of the image; for Activation function, used to convert the model's raw scores into probability distributions; Represents the convolution operation; Represents tensor concatenation; is the pixel value of the UV channel; is the pixel value of the infrared channel.

[0167] S223, using the second edge weight matrix The ultraviolet discharge edge image and the infrared edge image are weightedly fused to generate a second area to be identified. The second area to be identified is an area where the ultraviolet discharge edge image and the infrared edge image overlap.

[0168] S224: Perform texture analysis and amplification processing on the second area to be identified to obtain an ultraviolet detail image and an infrared detail image. Feature extraction is performed on the ultraviolet detail image and the infrared detail image to generate a corona intensity feature image and a heat and temperature feature image. The corona intensity feature image is extracted from the ultraviolet detail image, where the ultraviolet intensity is greater than the ultraviolet intensity threshold; the heat and temperature feature image is extracted from the infrared detail image, where the infrared intensity is greater than the infrared intensity threshold.

[0169] In step S224, the feature extraction is performed on the ultraviolet detail image and the infrared detail image respectively to generate the corona intensity feature image and the heat temperature feature image, which specifically includes the following steps:

[0170] A multi-branch residual network is constructed using Equation (7), and a three-path feature extraction structure including a shallow detail branch, a deep semantic branch, and a cross-modal interaction branch is determined. Multi-scale feature extraction is performed on the ultraviolet detail image and the infrared detail image through the channel attention module and the spatial-channel joint attention mechanism, and the corona intensity feature image and the heat temperature feature image are output:

[0171] (7);

[0172] in, represents the corona intensity characteristic image, Represents the heat temperature feature image, express The fourth layer feature output extracted by the residual network, is the channel attention module, is the spatial-channel joint attention mechanism, is the pixel value of the UV channel, is the pixel value of the infrared channel;

[0173] In step S224, the corona intensity characteristic image is extracted from the ultraviolet detail image, where the ultraviolet intensity is greater than the ultraviolet intensity threshold, and includes:

[0174] Formula (8) is used to construct a dynamic threshold calculation model. Based on the dynamic threshold calculation model, the area with ultraviolet light intensity greater than the ultraviolet light intensity threshold is segmented from the ultraviolet detail image to generate the corona intensity feature image:

[0175] (8);

[0176] in, represents the UV intensity threshold, is the mean value of the UV image, is the standard deviation of the UV image, is the curvature adjustment coefficient; express about Direction and The sum of the second-order partial derivatives in the direction.

[0177] S225. Using the thermodynamic diffusion equation shown in formula (9), perform feature coupling across physical domains on the corona intensity feature image and the heat temperature feature image to generate a corona-temperature coupling feature:

[0178] (9);

[0179] in, is the heat temperature characteristic, Indicates time, is the thermal conductivity coefficient, Thermal temperature characteristics The spatial gradient of is the corona-temperature coupling coefficient, is the corona intensity characteristic.

[0180] S226. Construct a feature importance evaluation function using formula (10), and based on the feature importance evaluation function, perform adaptive amplification processing on the corona-temperature coupling feature using formula (11):

[0181] (10);

[0182] (11);

[0183] in, represents the enhancement coefficient, is the image pixel coordinate, is the image pixel coordinate The corona intensity characteristics at is the image pixel coordinate The thermal temperature characteristics at is the corona intensity characteristic before amplification, is the heat temperature characteristic before amplification, represents the global maximum corona intensity eigenvalue, represents the global maximum heat temperature eigenvalue, is the amplified corona intensity characteristic, is the amplified heat temperature characteristic.

[0184] S227, using formula (12) to identify the second area Perform multiphysics modeling:

[0185] (12);

[0186] in, is the corona intensity characteristic before amplification, is the corona intensity diffusion function; represents the Laplace operator; Represents the corona intensity characteristic image; Represents exponential operation; Represents a hyperparameter, which is set manually or determined through training. It is used to control the attenuation of the exponential term and adjust the influence weight of the term in the formula. is the UV intensity threshold, is the infrared intensity threshold, Thermal temperature characteristics of the infrared channel spatial gradient.

[0187] S228. Determine the adaptive fusion weights of the physical field features using the dynamic Alpha coefficient generation algorithm shown in formula (13), and generate a second fused superposition image using formula (14):

[0188] (13);

[0189] (14);

[0190] in, is the dynamic coefficient, for activation function, is the amplified corona intensity characteristic, Indicates the generation of the second fused overlay image, is the amplified heat temperature characteristic, express Function, used to get the global maximum value from the entire image.

[0191] S229, using formula (15) to establish a temperature-corona intensity correlation model, performing hot spot detection on the second coupled superposition image based on the temperature-corona intensity correlation model, and determining the fault type in the second coupled superposition image;

[0192] (15);

[0193] in, Indicates the output corona intensity characteristic image or ultraviolet characteristic image at the image pixel coordinate The pixel value at Indicates the UV image at the image pixel coordinates The measurement value at , is the abnormality determination coefficient, is the UV intensity threshold.

[0194] The abnormality determination coefficient , calculated using the adaptive algorithm shown in formula (16):

[0195] (16);

[0196] in, is the UV image mean, is the standard deviation of the UV image.

[0197] S3. Build a high-voltage equipment defect detection model based on the YOLO-STrans network.

[0198] The high-voltage equipment defect detection model is implemented based on the YOLO-STrans network, which is built based on the improved YOLO and Transformer networks. Figure 3 As shown in the figure, the high-voltage equipment defect detection model includes a multi-spectral input terminal, a CSP-YOLO detection module, a Transformer target extractor, a dynamic fusion module, a multi-task classifier and a multi-scale output terminal.

[0199] (1) Multispectral input

[0200] The multispectral input terminal is used to input a multispectral image, i.e., the collected ultraviolet image, infrared image, and visible light image, and preprocess the input image using the spectral normalization module shown in formula (17):

[0201] (17);

[0202] in, Indicates the The pixel values ​​of the class images (ultraviolet, infrared, visible light) after standardization are used for subsequent feature extraction or fusion to make the data of different modalities comparable in value range; Represents the original Image pixel values, such as pixel intensities of ultraviolet images, infrared images, and visible light images; 、 are the mean and standard deviation of each channel respectively; The value of , used to prevent numerical overflow; Indicates the image modality identifier, represents the UV image, represents an infrared image, Represents a visible light image;

[0203] (2) CSP-YOLO detection module

[0204] The CSP-YOLO detection module is used to construct an improved CSPDarknet53 network using formula (18), introduce deformable convolution into the backbone network of CSPDarknet53 to obtain an improved CSPDarknet53 network, and use the improved CSPDarknet53 network for feature extraction:

[0205] (18);

[0206] in, Indicates the output signal or feature at the position variable The value at represents the result after weighted processing, such as the pixel value after filtering, the feature after fusion, etc. is a position variable, which represents the space of the processing object; is the position offset within the neighborhood; is the neighborhood set; is the weight function, for each location in the neighborhood The weights assigned, such as the coefficients of the Gaussian filter and the weights of the attention mechanism, are used to control the output of different positions the extent of contribution; is the input signal; It is the dynamically learned position offset, which is used to improve the detection accuracy of irregular discharge areas.

[0207] In the CSPDarknet53 backbone network, some standard convolutional layers are replaced with deformable convolutional layers, forming an improved CSPDarknet53 network. When the CSP-YOLO detection module detects small targets (such as faint corona spots) or defects with variable shapes, it dynamically adjusts sampling points to match the target contour. During feature fusion, deformable convolution is used to enhance the spatial alignment of features at different stages. By dynamically adjusting the convolution kernel sampling position, the ability to extract irregular defect features (such as non-rectangular discharge areas and complex hot spots) in high-voltage equipment is improved, thereby enhancing the ability to extract defect features.

[0208] The architecture diagram of the CSP-YOLO detection module in this embodiment is as follows Figure 4 As shown in the figure, the main structure includes an input, a backbone network, a neck network, and an output. The input is used to input image data. The backbone network consists of a Focus module for downsampling, a Conv module, a C3 connection module for residual network feature extraction, and an SPP pyramid pooling module. The neck network, which further performs feature extraction and fusion, mainly consists of a C3 module, an Upsample module, a Conv module, and a Concat module for feature fusion. The output is used to output the predicted image results.

[0209] In the high-voltage equipment defect detection model described in this embodiment, the CSP-YOLO detection module is an image segmentation model based on an improved YOLO network. First, the CBAM convolutional attention mechanism is added to the YOLOv8 network backbone to effectively mitigate the impact of device occlusion on detection and recognition accuracy. Second, a weighted bidirectional (top-down + bottom-up) feature pyramid network (BiFPN) is adopted, replacing the FPN (Feature Pyramid Network) and PAN (Path Aggregation Network) structures used in the original YOLOv8 network. This improves the algorithm's ability to extract features for objects of varying scales. Finally, at the model network's output, the SIOU prediction box position loss function is adopted, replacing the CIOU (Complete Intersection over Union) loss function used in the original YOLOv8 network. Building on the YOLOv8 network, this model comprehensively considers the target's directional position loss, effectively preventing the impact of device position offsets, such as camera positions, on detection in practical applications.

[0210] (3) Transformer target extractor

[0211] The Transformer target extractor is used to receive the multimodal features (ultraviolet / infrared feature maps) output by the CSP-YOLO detection module, and calculate the semantic association between any positions in the feature map through a multi-head self-attention mechanism to determine the global dependency.

[0212] The multi-head attention mechanism is shown as follows:

[0213] The multi-head attention mechanism is shown in formula (19):

[0214] (19);

[0215] in, This is the result of multi-head attention mechanism processing ultraviolet, visible and infrared spectral images. For the connection operation function, represents the output projection weight matrix;

[0216] Attention Head Calculate using formula (20):

[0217] (20);

[0218] in, Indicates the The attention output of each head, each head calculates attention through independent linear transformation to capture the characteristics of different subspaces (such as space, channel, modality, etc.); is the activation function used to convert the raw scores of the model into a probability distribution; Represents the attention head The query matrix; Represents the attention head The bond matrix, represents the transpose operator, Represents the attention head The value matrix of Indicates the key matrix dimension, used for scaling.

[0219] Specifically, the Transformer object extractor extracts objects in the area where the high-voltage equipment is located. Based on the output results of the image segmentation detection of the CSP-YOLO detection module, it extracts the area where the high-voltage equipment is located from the segmented image to filter out background interference and focus on defect detection in the equipment area (such as insulators and conductors).

[0220] (4) Dynamic fusion module

[0221] The dynamic fusion module is used to construct a dual-stream feature fusion mechanism using formula (21), use SENet to adaptively generate channel attention weights, and combine with a deformable convolutional network to achieve cross-modal complementarity and spatial alignment of ultraviolet, infrared, and visible light multimodal features, thereby improving the accuracy and robustness of high-voltage equipment defect detection.

[0222] (twenty one);

[0223] in, is the dynamic fusion eigenvalue; is a dynamic coefficient, adaptively generated by SENet; Input yolo network feature value; is the input Transformer network feature value; It is a deformable convolutional network.

[0224] In the high-voltage equipment defect detection model described in this embodiment, the role of the fusion module is to downsample the feature map before the start of each stage, reduce the resolution of the image through a convolution operation with a step size of 2, and at the same time, adjust the number of channels to achieve hierarchical processing of features, thereby realizing a hierarchical structure. In addition, the fusion module can also reduce calculations and capture defect features of different scales through hierarchical feature extraction.

[0225] The dynamic fusion module can not only adjust the sampling points through deformable convolution to accurately align the two defect features to avoid missed detection; it can also suppress the noise channel in the visible light through channel attention and retain the effective features of the ultraviolet / infrared.

[0226] The dynamic fusion module uses adaptive weight allocation and deformable convolution to address the information redundancy and conflict issues inherent in traditional feature splicing. The dual-stream feature fusion mechanism involves the dynamic fusion module processing features from different modalities in two separate processes: a channel attention branch, which generates channel weights α using SENet to suppress irrelevant channels and enhance key features (such as the UV signature of the discharge area); and a spatial feature alignment branch, which utilizes deformable convolution (DCN) to dynamically adjust sampling points to compensate for spatial misalignment between modalities (such as the positional deviation between UV discharge points and infrared hotspots). After processing the two feature streams, they are fused through a weighted summation. The fused feature maps from the dynamic fusion module are then passed to a multi-task classifier (such as a classification branch based on depthwise separable convolution) for tasks such as defect type identification and severity assessment.

[0227] Traditional methods (such as Early Fusion and Late Fusion) use preset weights and are unable to adapt to the differences in feature distributions between different defect types (such as discharge and heating). The dynamic fusion module in this embodiment, however, uses SENet to dynamically learn weights, increasing the weight of ultraviolet features in discharge scenarios and enhancing the weight of infrared features in heating scenarios. To address the problem that the fixed sampling grid of standard convolution cannot handle spatial misalignment between different modalities, the dynamic fusion module in this embodiment utilizes a deformable convolutional network (DCN) to dynamically adjust the sampling position of the convolution kernel. By learning offsets, the DCN enables the model to capture the precise correspondence between ultraviolet discharge points and infrared hotspots.

[0228] (5) Multi-task classifier

[0229] The multi-task classifier classifies defects and identifies faults in the high-voltage equipment regions extracted by the Transformer object extractor, outputting results such as defect type (such as discharge or overheating) and severity. Based on an improved Transformer network, the multi-task classifier introduces a sliding window mechanism that calculates self-attention within a local window, reducing computational complexity while maintaining global modeling capabilities.

[0230] The multi-task classifier uses Equation (22) to construct a depthwise separable convolution parallel output branch to perform defect classification and fault identification on the device area output by the Transformer target extractor, thereby achieving multi-dimensional classification and positioning of defects:

[0231] (twenty two);

[0232] in, Positioning loss value for multi-dimensional classification, is the position loss weight value, is the position loss value, is the classification loss weight value, is the classification loss value.

[0233] The multi-task classifier adds a sliding window mechanism to the Transformer network, calculating self-attention within a local window to reduce computational complexity while maintaining global modeling capabilities.

[0234] The network structure of the multi-task classifier is as follows Figure 5As shown, the multi-task classifier has a hierarchical structure. Image processing is divided into four stages. In each stage, the resolution of the feature map input to the network is reduced. This process is similar to the step-by-step feature extraction of convolutional neural networks. Each stage includes a stride-size fusion module, PatchMerging, and multiple Transformer blocks. Feature embedding is performed before input. The image is divided into blocks and then embedded into vectors. This means that the image is divided into small blocks and embedded into embedding blocks. Before the block is input, the input image is first divided into many fixed-size patches and then embedded into vectors. Positional encoding is achieved through a two-dimensional convolutional layer with both stride and kernelsize set to the patch size. Finally, the height H and width W dimensions are expanded and moved to the first dimension. Within each block, attention is calculated only within a fixed-size window (e.g., 7×7). Information exchange is achieved across windows using sliding window shifting.

[0235] The multi-task classifier receives the multimodal features output by the dynamic fusion module, processes them, and outputs classification and localization results. This multi-task classifier constructs multiple parallel branches based on depthwise separable convolution to simultaneously handle different detection tasks, such as defect type classification, location regression, and severity assessment. Defect type classification involves identifying types such as discharge, overheating, and mechanical damage; location regression involves predicting the bounding box coordinates of the defect; and severity assessment involves determining the defect level (normal, severe, or critical). The multi-task classifier uses DSConv to construct parallel branches, allowing different tasks (classification, localization, and severity assessment) to share underlying features while maintaining task independence and avoiding interference between them. Dynamic class weights automatically adjust based on the sample distribution of each class, making them more adaptable than fixed weights.

[0236] (6) Multi-scale output

[0237] The multi-scale output end detects defects of different scales based on the feature pyramid using formula (23) and outputs three-channel detection results to achieve three-dimensional diagnosis of discharge area positioning, temperature anomaly identification, and defect classification.

[0238] (twenty three);

[0239] in, Represents the diagnosis result, express Operation, represents the discharge probability; is the activation function, represents the probability of temperature anomaly, Represents the probability of appearance defects.

[0240] In the high-voltage equipment defect detection model, the multi-scale output end uses a feature pyramid to achieve refined detection of defects of different scales, combines three-channel output (ultraviolet, infrared, and visible light) to complete three-dimensional diagnosis, and outputs the high-voltage equipment defect detection results.

[0241] Considering the significant differences in defect sizes in high-voltage equipment, small defects such as initial corona spots and tiny cracks require high-resolution feature maps to capture details, while large defects such as large heating areas and equipment structural deformation require low-resolution feature maps to extract global semantics. In this embodiment, the feature pyramid integrates features from different levels to enable the model to have both high-resolution detail perception and low-resolution semantic understanding capabilities. The three-channel output includes an ultraviolet channel, an infrared channel, and a visible light channel. The ultraviolet channel is used to locate the discharge area and output the coordinates and intensity of the light spot; the infrared channel is used to identify temperature anomalies and mark the location and temperature of the hot spot; and the visible light channel is used to classify defect types (such as cracks and contamination) and provide physical structural context. The multi-scale output terminal achieves multi-dimensional verification of defects and reduces the false detection rate through cross-channel correlation (such as spatial alignment of discharge points and hot spots).

[0242] S4. Use the fused image data to train a high-voltage equipment defect detection model to obtain a high-voltage equipment defect recognition model.

[0243] The high-voltage equipment defect detection model in this embodiment is based on a deep neural network. A dataset is constructed and used to train the high-voltage equipment defect detection model. The trained high-voltage equipment defect detection model serves as the high-voltage equipment defect recognition model. The dataset is collected using a detection device in a simulated laboratory and real-world application scenarios. The high-voltage equipment defect detection model is trained using training data from the dataset. The training data includes, but is not limited to, ultraviolet images, visible light images, infrared images, first fused overlay images, and second fused overlay images.

[0244] S5. Obtain a multispectral image of the high-voltage equipment to be inspected and input it into a high-voltage equipment defect recognition model, and perform defect recognition using the high-voltage equipment defect recognition model. In step S5, the high-voltage equipment defect detection model is verified.

[0245] In this embodiment, edge detection, feature extraction, and amplification are performed on UV and visible light images, enabling detailed processing of discharge areas in these images. This lays the foundation for accurate line fault identification. Edge detection, texture analysis, and amplification are performed on UV and infrared images, filtering out unnecessary infrared information. This allows detailed processing of devices in these images, paving the way for accurate device status fault detection.

[0246] In an experimental comparison of the method of the present invention with other detection algorithms in the prior art on the same dataset, the following results were obtained: YOLOv5 achieved a precision of 91.7%, a recall of 80.5%, and a mean average precision (mAP) of 87.7%; YOLOv8 achieved a precision of 92.4%, a recall of 89.3%, and a mean average precision (mAP) of 91.9%; YOLOv10 achieved a precision of 93.2%, a recall of 91.6%, and a mean average precision (mAP) of 92.7%; and the method of the present invention achieved a precision of 96.7%, a recall of 95.3%, and a mean average precision (mAP) of 98.7%. As can be seen from the above data, the method of the present invention outperformed other algorithms in terms of precision, recall, and mean average precision.

[0247] Example 2

[0248] This embodiment provides a high-voltage equipment defect detection device based on deep learning and multispectral image fusion, which can apply the above-mentioned high-voltage equipment defect detection method based on deep learning and multispectral image fusion.

[0249] like Figure 6 As shown, the present invention also includes providing a high-voltage equipment defect detection device based on deep learning and multispectral image fusion, which includes a spectroscopic module, an image processing and fusion module, a model building and training module, and a defect recognition module.

[0250] The spectroscopic module is used to acquire multispectral images of high-voltage equipment. Specifically, the spectroscopic module acquires full-band spectral information from the surface of the target high-voltage equipment and divides the full-band spectral information into three target optical paths: ultraviolet, infrared, and visible light. The spectroscopic module then processes the optical paths of the target high-voltage equipment to acquire raw ultraviolet, infrared, and visible light images. These images are then optically and electronically zoomed to produce a refined image.

[0251] The image processing and fusion module is used to preprocess multispectral images and perform cross-modal fusion on the preprocessed image data. This preprocessing includes cleaning the raw data, performing format conversion, feature engineering, and merging multi-source data to construct a unified dataset. This preprocessing also includes: processing spectral information in the ultraviolet, infrared, and visible light bands to obtain discharge intensity, temperature, and surface defect information on the target high-voltage equipment surface; performing noise suppression and grayscale processing on the raw ultraviolet image to obtain a low-noise image; and calculating the low-noise image based on a discharge intensity detection algorithm to obtain an ultraviolet spot image and ultraviolet coordinate information; performing noise reduction on the raw infrared image to obtain a low-noise image; and calculating the low-noise image based on an infrared heat temperature detection algorithm to obtain surface temperature information and infrared coordinate information on the target high-voltage equipment surface; and performing edge detection on the raw visible light image and obtaining surface defect information and visible light coordinate information on the target high-voltage equipment surface through texture feature analysis. The temperature and surface defect information includes surface temperature information, coordinate information, and surface defect information on the target high-voltage equipment surface. The cross-modal fusion processing of the pre-processed image data includes obtaining a coordinate matching coefficient and an offset based on the ultraviolet coordinate information, the infrared coordinate information, and the visible light coordinate information; comparing the coordinate matching coefficient and the offset with a preset threshold value; if the coordinate matching coefficient and the offset are less than a first threshold value, it is determined that the target high-voltage equipment is in a normal working area; if the coordinate matching coefficient and the offset are greater than or equal to the first threshold value and less than a second threshold value, it is determined that the target high-voltage equipment is in a warning area; if the coordinate matching coefficient is greater than the second threshold value, it is determined that the target high-voltage equipment is in a critical area.

[0252] The model building and training module is used to build a high-voltage equipment defect detection model based on the YOLO-STrans network, and use the fused image data to train the high-voltage equipment defect detection model.

[0253] The defect recognition module is used to obtain a multispectral image of the high-voltage equipment to be detected, and perform defect recognition on the multispectral image using a high-voltage equipment defect detection model.

[0254] Those skilled in the art will clearly understand that for the convenience and brevity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.

[0255] Example 3

[0256] The present invention also provides a processing device, which mainly includes: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the method provided in the aforementioned embodiment.

[0257] Furthermore, the processing device further includes at least one input device and at least one output device; in the processing device, the processor, memory, input device, and output device are connected via a bus.

[0258] In the embodiment of the present invention, the specific types of the memory, input device and output device are not limited; for example, the input device can be a touch screen, an image acquisition device, a physical button or a mouse, etc.; the output device can be a display terminal; the memory can be a random access memory (RAM) or a non-volatile memory (non-volatile memory), such as a disk storage.

[0259] Example 4

[0260] The present invention also provides a readable storage medium storing a computer program, which implements the method provided in the above embodiment when the computer program is executed by a processor.

[0261] In the embodiments of the present invention, the computer-readable storage medium may be provided in the aforementioned processing device, for example, as a memory in the processing device. Alternatively, the computer-readable storage medium may be a USB flash drive, a removable hard drive, a read-only memory (ROM), a magnetic disk, or an optical disk, among other media capable of storing program code.

[0262] The above shows and describes the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions merely illustrate the principles of the present invention. Various changes and modifications may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and modifications are intended to fall within the scope of the present invention. The scope of protection claimed by the present invention is defined by the appended claims and their equivalents.

Claims

1. A high-voltage equipment defect detection method based on deep learning and multispectral image fusion, characterized in that: The following steps are involved: S1. Acquire multispectral images of high-voltage equipment and perform preprocessing; S2, performing cross-modal fusion processing on the pre-processed image data; S3. Build a high-voltage equipment defect detection model based on the YOLO-STrans network; S4. Using the fused image data, a high-voltage equipment defect detection model is trained to obtain a high-voltage equipment defect recognition model; S5. Acquire a multispectral image of the high-voltage equipment to be inspected, and perform defect recognition on the multispectral image using a high-voltage equipment defect recognition model; The step S2 specifically includes the following steps: S21, fusing and superimposing the pre-processed ultraviolet image and the visible light image to obtain a first fused superimposed image; S22, fusing and superimposing the pre-processed ultraviolet image and infrared image to obtain a second fused superimposed image; The step S22 specifically includes the following steps: S221, performing edge detection on the pre-processed ultraviolet image and infrared image respectively, and extracting the ultraviolet discharge edge image and the infrared edge image; S222, extract features from the preprocessed ultraviolet image and infrared image through a dual-channel attention mechanism to generate a second edge weight matrix ; S223, using the second edge weight matrix Performing weighted fusion on the ultraviolet discharge edge image and the infrared edge image to generate a second area to be identified; the second area to be identified is an area where the ultraviolet discharge edge image and the infrared edge image overlap; S224, performing texture analysis and amplification processing on the second area to be identified to obtain an ultraviolet detail image and an infrared detail image, performing feature extraction on the ultraviolet detail image and the infrared detail image respectively to generate a corona intensity feature image and a heat temperature feature image; S225. Using a thermodynamic diffusion equation, perform feature coupling across physical domains on the corona intensity feature image and the heat temperature feature image to generate a corona-temperature coupling feature. S226. Construct a feature importance evaluation function using formula (10), and based on the feature importance evaluation function, perform adaptive amplification processing on the corona-temperature coupling feature using formula (11): （10）； （11）； in, represents the enhancement coefficient, is the image pixel coordinate, is the image pixel coordinate The corona intensity characteristics at is the image pixel coordinate The thermal temperature characteristics at is the corona intensity characteristic before amplification, is the heat temperature characteristic before amplification, represents the global maximum corona intensity eigenvalue, represents the global maximum heat temperature eigenvalue, is the amplified corona intensity characteristic, is the amplified heat temperature characteristic; S227, the second area to be identified Perform multiphysics modeling; S228. Determine the adaptive fusion weights of the physical field features using the dynamic Alpha coefficient generation algorithm shown in formula (13), and generate a second fused superposition image using formula (14): （13）； （14）； in, is the dynamic coefficient, for activation function, is the amplified corona intensity characteristic, Indicates the generation of the second fused overlay image, is the amplified heat temperature characteristic, express Function, used to get the global maximum value from the entire image; S229 , establishing a temperature-corona intensity correlation model, performing hot spot detection on the second coupled superimposed image based on the temperature-corona intensity correlation model, and determining a fault type in the second coupled superimposed image.

2. The high-voltage equipment defect detection method based on deep learning and multispectral image fusion according to claim 1 is characterized in that: The step S1 specifically includes the following steps: S11, the multispectral image includes an ultraviolet image, an infrared image and a visible light image, and the ultraviolet image, infrared image and visible light image are preprocessed respectively: the ultraviolet image of the high-voltage equipment in the target area is synchronously collected. , infrared images and visible light images And with the height and width of the image as the dimensions, a three-channel feature space including ultraviolet image, infrared image and visible light image is established; S12. Preprocess each modality image in the three-channel feature space based on the three-channel feature space.

3. The high-voltage equipment defect detection method based on deep learning and multispectral image fusion according to claim 1 is characterized in that: The step S21 specifically includes the following steps: S211, performing edge detection on the pre-processed ultraviolet image and visible light image respectively, and extracting the ultraviolet discharge edge image and the high-voltage equipment structure edge image; S212: Extract features from the preprocessed UV image and visible light image through a dual-channel attention mechanism to generate a first edge weight matrix ; S213, using the first edge weight matrix Performing weighted fusion on the ultraviolet discharge edge image and the high-voltage equipment structure edge image to generate a first area to be identified; the first area to be identified is an area where the ultraviolet discharge edge image and the high-voltage equipment structure edge image overlap; S214, performing multi-scale feature extraction on the first to-be-identified area to obtain an ultraviolet feature image and a visible light feature image, and performing a magnification process on the ultraviolet feature image and the visible light feature image; S215 , using a pyramid fusion algorithm with dynamic weight allocation to perform cross-modal fusion on the amplified ultraviolet feature image and the visible light feature image to obtain a first fused superimposed image.

4. The high-voltage equipment defect detection method based on deep learning and multispectral image fusion according to claim 1 is characterized in that: In step S224, the feature extraction of the ultraviolet detail image and the infrared detail image is performed to generate a corona intensity feature image and a heat temperature feature image, including: A multi-branch residual network is constructed to determine a three-path feature extraction structure consisting of a shallow detail branch, a deep semantic branch, and a cross-modal interaction branch. Multi-scale feature extraction is performed on ultraviolet and infrared detail images through a channel attention module and a spatial-channel joint attention mechanism, outputting corona intensity feature images and thermal temperature feature images. In step S224, the corona intensity characteristic image is extracted from the ultraviolet detail image, where the ultraviolet intensity is greater than the ultraviolet intensity threshold, and includes: A dynamic threshold calculation model is constructed, and based on the dynamic threshold calculation model, a region where the ultraviolet light intensity is greater than the ultraviolet light intensity threshold is segmented from the ultraviolet detail image to generate a corona intensity characteristic image.

5. The high-voltage equipment defect detection method based on deep learning and multispectral image fusion according to claim 1 is characterized in that: The high-voltage equipment defect detection model is implemented based on the YOLO-STrans network, which is built based on the improved YOLO and Transformer networks; The high-voltage equipment defect detection model includes a multispectral input terminal, a CSP-YOLO detection module, a Transformer target extractor, a dynamic fusion module, a multi-task classifier and a multi-scale output terminal.

6. A high-voltage equipment defect detection device based on deep learning and multispectral image fusion, characterized in that: The device includes a spectroscopic module, an image processing and fusion module, a model building and training module, and a defect recognition module; The spectroscopic module is used to obtain multispectral images of high-voltage equipment; The image processing and fusion module is used to preprocess the multispectral image and perform cross-modal fusion processing on the preprocessed image data; The model building and training module is used to build a high-voltage equipment defect detection model based on the YOLO-STrans network, and use the fused image data to train the high-voltage equipment defect detection model to obtain a high-voltage equipment defect recognition model; The defect recognition module is used to obtain a multispectral image of the high-voltage equipment to be inspected and perform defect recognition on the multispectral image using a high-voltage equipment defect recognition model; The method for preprocessing the multispectral image and performing cross-modal fusion processing on the preprocessed image data specifically includes the following steps: S21, fusing and superimposing the pre-processed ultraviolet image and the visible light image to obtain a first fused superimposed image; S22, fusing and superimposing the pre-processed ultraviolet image and infrared image to obtain a second fused superimposed image; The step S22 specifically includes the following steps: S221, performing edge detection on the pre-processed ultraviolet image and infrared image respectively, and extracting the ultraviolet discharge edge image and the infrared edge image; S222, extract features from the preprocessed ultraviolet image and infrared image through a dual-channel attention mechanism to generate a second edge weight matrix ; S223, using the second edge weight matrix Performing weighted fusion on the ultraviolet discharge edge image and the infrared edge image to generate a second area to be identified; the second area to be identified is an area where the ultraviolet discharge edge image and the infrared edge image overlap; S224, performing texture analysis and amplification processing on the second area to be identified to obtain an ultraviolet detail image and an infrared detail image, performing feature extraction on the ultraviolet detail image and the infrared detail image respectively to generate a corona intensity feature image and a heat temperature feature image; S225. Using a thermodynamic diffusion equation, perform feature coupling across physical domains on the corona intensity feature image and the heat temperature feature image to generate a corona-temperature coupling feature. S226. Construct a feature importance evaluation function using formula (10), and based on the feature importance evaluation function, perform adaptive amplification processing on the corona-temperature coupling feature using formula (11): （10）； （11）； in, represents the enhancement coefficient, is the image pixel coordinate, is the image pixel coordinate The corona intensity characteristics at is the image pixel coordinate The thermal temperature characteristics at is the corona intensity characteristic before amplification, is the heat temperature characteristic before amplification, represents the global maximum corona intensity eigenvalue, represents the global maximum heat temperature eigenvalue, is the amplified corona intensity characteristic, is the amplified heat temperature characteristic; S227, the second area to be identified Perform multiphysics modeling; S228. Determine the adaptive fusion weights of the physical field features using the dynamic Alpha coefficient generation algorithm shown in formula (13), and generate a second fused superposition image using formula (14): （13）； （14）； in, is the dynamic coefficient, for activation function, is the amplified corona intensity characteristic, Indicates the generation of the second fused overlay image, is the amplified heat temperature characteristic, express Function, used to get the global maximum value from the entire image; S229 , establishing a temperature-corona intensity correlation model, performing hot spot detection on the second coupled superimposed image based on the temperature-corona intensity correlation model, and determining a fault type in the second coupled superimposed image.

7. An electronic device, characterized in that: include: at least one processor; and a memory storing instructions, which, when executed by the at least one processor, enable the at least one processor to execute the high-voltage equipment defect detection method based on deep learning and multispectral image fusion as described in any one of claims 1 to 5.

8. A machine-readable storage medium, characterized in that It stores executable instructions, which, when executed, enable the machine to perform the high-voltage equipment defect detection method based on deep learning and multispectral image fusion as described in any one of claims 1 to 5.

Images