Transformer substation oil leakage defect detection method and system based on multi-technology fusion

Abstract

The invention relates to a transformer substation oil leakage defect detection method and system based on multi-technology fusion. The detection method comprises the following steps: S1, synchronously acquiring an image of to-be-detected equipment through image acquisition equipment deployed in a substation equipment area; the image acquisition equipment comprises a visible light camera, an infrared thermal imager and an ultraviolet imager and is used for correspondingly acquiring a visible light image, an infrared thermal image and an ultraviolet image; s2, carrying out space registration on the collected visible light image, infrared thermal imaging and ultraviolet imaging; s3, inputting the registered image group into a cross-modal alignment guided feature pyramid encoder, and outputting three high-dimensional feature vectors; s4, fusing the three high-dimensional feature vectors based on a multi-modal dynamic fusion network to generate a defect judgment score; and S5, comparing the defect judgment score with a preset threshold value, if the defect judgment score is higher than the threshold value, judging that an oil leakage defect exists, otherwise, judging that no defect exists.

Description

TECHNICAL FIELD

[0001] The present application relates to a substation leakage oil defect detection method and system based on multi-technology fusion. BACKGROUND

[0002] In the power system, the substation as the key hub of power conversion and distribution, its safe and stable operation is crucial. Among them, many devices in the substation, such as transformers, bushings, etc. are oil-filled devices. Under the influence of installation process, equipment aging, environmental changes and mechanical vibration and other complex factors, it is easy to appear leakage oil phenomenon under long-term high load operation state.

[0003] The traditional substation leakage oil detection method has many limitations. The manual inspection mainly depends on the visual inspection and nose smell of the operation and maintenance personnel. This method not only has very low efficiency, but also the accuracy of detection is seriously affected by the experience level, mental state and working environment of the operation and maintenance personnel, and it is difficult to accurately find early or hidden leakage oil defects. For example, in the dimly lit area or the complex structure of the equipment, manual inspection may miss the tiny oil leakage point, so that the potential safety hazard of the equipment cannot be detected in time.

[0004] Part of the method that uses oil level gauge or oil level observation window to read the oil level to indirectly judge the leakage oil situation also has significant drawbacks. On the one hand, in actual operation, when observing the oil level gauge or observation window, there are often problems such as blocked view and difficult reading. On the other hand, there may also be false oil level phenomenon, that is, due to abnormal internal structure of the equipment or existence of bubbles in the oil, the displayed oil level is not the real oil level state, thereby causing misjudgment of the equipment leakage oil situation.

[0005] In order to overcome the defects of the above-mentioned traditional detection method, in recent years, some new technologies have been gradually applied to the substation leakage oil detection field. For example, using infrared thermal imaging technology, by analyzing the temperature distribution difference of the equipment surface to judge whether there is leakage oil. However, in actual application, this technology is easily disturbed by environmental temperature fluctuations, equipment self-heating and sunlight, etc. external factors, resulting in a big discount in the accuracy of the detection result. In addition, for some slight leakage oil situation, the infrared thermal imaging technology often cannot effectively identify due to the unobvious temperature change.

[0006] For example, using ultraviolet fluorescence detection technology, using the characteristic that insulating oil emits fluorescence under ultraviolet light to detect leakage oil. However, this method requires high precision of the detection equipment, and in the complex substation site environment, other substances or impurities with fluorescence characteristics may interfere with the detection result, causing misjudgment. At the same time, this technology has poor detection effect for large area and low concentration of leakage oil, and has a detection blind area.

[0007] In addition, image processing-based target detection algorithms have been attempted for substation oil leakage detection. However, due to the complex surface conditions of substation equipment, numerous background interference factors, and the high uncertainty of the shape, color, and location of leaking oil, existing target detection algorithms struggle to meet practical engineering requirements in terms of detection accuracy and robustness. Moreover, these algorithms typically require a large amount of high-quality labeled data for training, but obtaining rich and accurately labeled oil leakage image data in actual substation scenarios is quite difficult, severely limiting the application effectiveness of this technology.

[0008] In summary, existing methods for detecting oil leaks in substations have varying degrees of shortcomings in terms of detection efficiency, accuracy, reliability, and anti-interference capabilities, making it difficult to meet the stringent requirements of current power systems for the safe operation of substation equipment. Therefore, there is an urgent need to develop a multi-technology integrated method for detecting oil leaks in substations to achieve efficient, accurate, and reliable detection, thereby ensuring the safe and stable operation of the power system. Summary of the Invention

[0009] Existing methods for detecting oil leaks in substations have limitations in terms of detection efficiency, accuracy, reliability, and anti-interference capabilities, making it difficult to meet the stringent requirements of current power systems for the safe operation of substation equipment. Therefore, this invention achieves efficient, accurate, and reliable detection of oil leaks in substations, ensuring the safe and stable operation of the power system.

[0010] The present invention provides a method for detecting oil leakage defects in substations based on the integration of multiple technologies:

[0011] S1. Simultaneously acquire images of the equipment to be inspected by image acquisition devices deployed in the substation equipment area; the image acquisition devices are visible light cameras, infrared thermal imagers and ultraviolet imagers, which acquire visible light images, infrared thermal images and ultraviolet images respectively.

[0012] S2. Spatial registration of the acquired visible light images, infrared thermal images, and ultraviolet images;

[0013] S3. Input the registered image group into the cross-modal alignment-guided feature pyramid encoder and output three high-dimensional feature vectors;

[0014] S4. Based on a multimodal dynamic fusion network, three high-dimensional feature vectors are fused to generate a defect judgment score;

[0015] S5. Compare the defect judgment score with the preset threshold. If it is higher than the threshold, it is judged that there is an oil leakage defect; otherwise, it is judged that there is no defect.

[0016] In step S1, the deployment and data acquisition process for each acquisition device is as follows:

[0017] Step S11: Configure the parameters of the visible light camera, infrared thermal imager, and ultraviolet imager to ensure clear imaging;

[0018] For visible light cameras, set parameters according to the lighting conditions at the substation site;

[0019] Step S12: Send a synchronous acquisition command through the central control system to trigger the visible light camera, infrared thermal imager, and ultraviolet imager to start acquiring data simultaneously.

[0020] The acquisition duration is set according to the equipment inspection requirements to generate a continuous image sequence;

[0021] Step S13: Align the three types of images acquired by timestamp to form three sets of parallel data corresponding to the same detection time period.

[0022] Step S2: Spatial registration of the acquired visible light image, infrared thermal image, and ultraviolet image;

[0023] Step S21: Using the visible light image as the reference mode, extract the feature point pair set corresponding to the reference mode from the other two types of images;

[0024] Step S22: Calculate the spatial transformation matrix from the modal space to the reference modal space based on the feature point pair set;

[0025] Step S23: Perform spatial geometric transformation and resampling on the infrared and ultraviolet images using the spatial transformation matrix to complete pixel-level spatial registration;

[0026] Step S21: Using the visible light image as the reference mode, extract the feature point pair set corresponding to the reference mode from the other two types of images;

[0027] For the baseline mode, the SIFT algorithm with scale-invariant feature transformation is used to extract the set of strong feature points; regions with significant corner or spot features on the device surface that are not easily deformed are preferentially selected;

[0028] On the infrared and ultraviolet images to be registered, an automated script extracts corresponding feature points near the physical locations corresponding to the feature points in the reference image, forming feature point pairs, which are then organized to obtain a set of feature point pairs.

[0029] Before generating the first set of registration parameters, at least four pairs of identical feature points, evenly distributed across the four quadrants of the image, are precisely clicked on the reference image and the image to be registered through the human-computer interaction interface.

[0030] Step S22: Calculate the spatial transformation matrix from the modal space to the reference modal space based on the feature point pair set;

[0031] For registration from the modal space to be registered to the reference modal space, the transformation relationship is as follows:

[0032] Where x and y are the coordinates in the modal space to be registered. , These are the coordinates in the reference modal space, and H is a 3x3 matrix;

[0033] Based on the above transformation relationship, the optimal transformation matrix H1 is estimated from the reference mode-infrared image feature point pair using the RANSAC algorithm; then the optimal transformation matrix H2 is estimated from the reference mode-ultraviolet image feature point pair using the same algorithm; these two transformation matrices are the spatial transformation matrices from the mode space to be registered to the reference mode space.

[0034] Step S23: Perform spatial geometric transformation and resampling on the infrared and ultraviolet images using the spatial transformation matrix to complete pixel-level spatial registration;

[0035] To avoid holes and overlaps, a reverse mapping method is used: For each pixel in the reference modal coordinate system, denoted as the source pixel, its corresponding position (x', y') in the image to be registered is calculated using the inverse of the transformation matrix. Since the calculated corresponding position (x', y') is usually a sub-pixel precision non-integer coordinate, a bicubic interpolation algorithm is also needed to calculate the accurate pixel coordinates at this position. Then, these pixel coordinates are correlated with the source pixel coordinates to obtain the registered image. Infrared and ultraviolet images are spatially registered using the above method, and the registered visible light image, infrared image, and ultraviolet image are stored as a single image group.

[0036] Step S3: Input the registered image group into the cross-modal alignment-guided feature pyramid encoder and output three high-dimensional feature vectors;

[0037] The cross-modal alignment-guided feature pyramid encoder consists of a shared bottom layer, multiple modality-specific network branches, and a fusion layer. Its data processing flow is as follows:

[0038] Step S31: Extract common low-level features from all input images using a shared underlying layer;

[0039] Step S32: Input the extracted low-level features into the network branches corresponding to each image modality to extract the high-level features of each input image;

[0040] Step S33: Upsample and fuse the low-level and high-level features to obtain the final feature map;

[0041] Step S34: Introduce translation loss during training so that the encoder can automatically learn to align feature maps of different modalities in the abstract feature space;

[0042] Step S35: Convert the final fused feature map into a fixed-length high-dimensional feature vector;

[0043] Step S31: Extract common low-level features from all input images using a shared underlying layer;

[0044] The shared-weight convolutional network at the bottom layer extracts common low-level features such as edges and freckles; the output feature map size is H*W*Clow, where H*W is the visible light image resolution and Clow is the number of channels (64 in this embodiment). The low-level features extracted from the visible light, infrared, and ultraviolet images are represented as follows: , , .

[0045] Step S32: Input the extracted low-level features into the network branches corresponding to each image modality to extract the high-level features of each input image;

[0046] High-level features are further extracted from the low-level features of each modality. Each modality uses two layers of dedicated convolution with independent weights (3*3 kernels, stride 2, padding=1) + ReLU activation, with the following parameters:

[0047] Visible light exclusive: Infrared Exclusive: UV exclusive: ;

[0048] The specific convolution representations for the three modalities are as follows:

[0049] , among them Indicates the convolution operation;

[0050] The output size is (H / 4)*(W / 4)*Chigh. Because the convolution with stride 2 is performed twice, the size is reduced to 1 / 4, and Chigh=256. The output high-level features are represented as follows: , as well as .

[0051] Step S33: Upsample and fuse the low-level and high-level features to obtain the final feature map;

[0052] The extracted low- and high-level features are fed into a fusion layer, and the high-level feature map is upsampled: bilinear interpolation is used to enlarge the high-level feature Fhigh,j to H*W of the visible light image resolution, denoted as... , j is the modality index, with values ​​vis, ir, uv; the low-level feature flow is transformed by a 1*1 convolution, and the number of channels j is adjusted to 256, denoted as The element-wise high-level features after upsampling are fused with the low-level features after channel adjustment to output the final feature map. .

[0053] The fusion layer is represented as follows:

[0054] , in In response to 1*1 convolution kernel (adjust channels). These are the bias parameters used in the convolution operation. Indicating targeting Four times the upsampling.

[0055] Step S34: Introduce translation loss during training so that the encoder can automatically learn to align feature maps of different modalities in the abstract feature space;

[0056] In order to To ensure consistent representation of the same physical region in the feature space, a translation loss function is introduced during encoder training. A translator G, essentially a small MLP, is set after the fusion layer to achieve mutual translation of features from different modalities. Under the constraint of the translation loss function, the "translation" from visible light features should resemble infrared features, and vice versa. By minimizing this loss, the final feature maps of the two modalities will spontaneously have highly correlated representations at corresponding spatial locations, thus achieving precise alignment at the feature level, not just pixel-level registration. The translation loss function is expressed as follows:

[0057] ;

[0058] Where E() is a function that calculates the mean on the training data, j and i are modality indices, and j ≠ i. and These are the translator network branches from mode j to mode i and from mode i to mode j, respectively.

[0059] Step S35: Convert the final fused feature map into a fixed-length high-dimensional feature vector;

[0060] The final feature map is subjected to global average pooling (GAP) and flattening operations to obtain a fixed-length high-dimensional feature vector. .

[0061] Step S4: Based on the multimodal dynamic fusion network, the three high-dimensional feature vectors are fused to generate a defect judgment score;

[0062] The multimodal dynamic fusion network incorporates a dynamic feature selection unit that simulates the "doubt" and "acceptance" process in human expert decision-making. It learns to prioritize the most reliable evidence in specific scenarios. Its specific data processing logic is as follows:

[0063] Step S41: Acquire and encode ambient light, humidity, and noise data in real time to form environmental context variables;

[0064] Step S42: Calculate the doubt score for each modality by combining the generated environmental context quantity;

[0065] Step S43: Input the three high-dimensional feature vectors and the doubt score into the dynamic feature selection unit to generate the fused feature vector;

[0066] Step S44: Feed the fused feature vector into the classifier to obtain the defect determination score.

[0067] Step S41: Acquire and encode ambient light, humidity, and noise data in real time to form environmental context variables;

[0068] By deploying a sensor array in the substation detection area, key environmental parameters affecting the accuracy of multimodal detection are collected synchronously, including: illumination, humidity, and noise data. The acquisition frequency is kept consistent with the image acquisition frame rate to ensure spatiotemporal synchronization. The collected data is normalized and encoded to form a low-dimensional environmental context vector R.

[0069] Step S42: Calculate the doubt score for each modality by combining the generated environmental context quantity;

[0070] The formula for calculating the doubtful score is as follows: Where j is the modal index, with values ​​vis, ir, and uv. Let R represent the doubt score for the j-th modality, and let R be the context vector. It is the attention weight matrix, learned through training, used for the interaction between modal features (i.e., each high-dimensional feature vector) and the environmental context. Let j be the high-dimensional eigenvector of the j-th mode. Let be the characteristic variance of the j-th mode. The result within the parentheses is the maximum value obtained after applying Softmax. , These are globally questionable parameters, learned through training.

[0071] Step S43: Input the three high-dimensional feature vectors and the doubt score into the dynamic feature selection unit to generate the fused feature vector;

[0072] The dynamic feature selection unit is expressed by the mathematical formula as follows:

[0073] ,in This is the generated fusion feature vector.

[0074] Step S44: Feed the fused feature vector into the classifier to obtain the defect determination score;

[0075] fuse feature vectors Input a lightweight classifier and output a defect judgment score between 0 and 1. The higher the score, the greater the probability of oil leakage. The classifier uses a 2-layer fully connected network (hidden layer dimension 128, output layer dimension 1) + Sigmoid activation.

[0076] Step S5: Compare the defect judgment score with the preset threshold. If it is higher than the threshold, it is judged that there is an oil leakage defect; otherwise, it is judged that there is no defect.

[0077] A substation oil leakage defect detection system based on multi-technology fusion, used to execute the substation oil leakage defect detection method based on multi-technology fusion, includes a multi-modal image acquisition module, a spatial configuration module, a high-dimensional feature extraction module, a high-dimensional feature fusion module and an oil leakage defect determination module that are sequentially connected by signals.

[0078] The multimodal image acquisition module is used to simultaneously acquire images of the equipment under test through image acquisition devices deployed in the substation equipment area. The image acquisition devices are visible light cameras, infrared thermal imagers, and ultraviolet imagers, which acquire visible light images, infrared thermal images, and ultraviolet images respectively. The acquired images are transmitted to the spatial configuration module.

[0079] The spatial configuration module is used to perform spatial registration of visible light images, infrared thermal imaging, and ultraviolet imaging. The registered images are then transmitted to the high-dimensional feature extraction module.

[0080] The high-dimensional feature extraction module is used to input the registered image group into the cross-modal alignment-guided feature pyramid encoder, output three high-dimensional feature vectors, and transmit them to the high-dimensional feature fusion module.

[0081] The high-dimensional feature fusion module is used to fuse three high-dimensional feature vectors using a multimodal dynamic fusion network to generate a defect judgment score, which is then transmitted to the oil leakage defect judgment module.

[0082] The oil leakage defect determination module is used to compare the defect determination score with a preset threshold. If the score is higher than the threshold, it is determined that there is an oil leakage defect; otherwise, it is determined that there is no defect.

[0083] The multimodal image acquisition module specifically includes: an imaging device configuration submodule for sequential signal transmission, an acquisition command triggering submodule, and a timestamp alignment submodule; the imaging device configuration submodule is used to configure the parameters of the visible light camera, infrared thermal imager, and ultraviolet imager to ensure clear imaging, and after the configuration is completed, it jumps to the acquisition command triggering submodule;

[0084] The acquisition command triggering submodule is used to send a synchronous acquisition command through the central control system to trigger three devices to start acquiring data simultaneously. The acquired images are then transmitted to the timestamp alignment submodule.

[0085] The timestamp alignment submodule is used to align the three types of images acquired according to their timestamps, forming three sets of parallel data corresponding to the same detection period.

[0086] The spatial configuration module specifically includes: a feature point pair extraction submodule, a spatial transformation matrix calculation submodule, and a registered image group output submodule;

[0087] The feature point pair extraction submodule is used to extract the set of feature point pairs corresponding to the reference mode from the other two types of images, using the visible light image as the reference mode.

[0088] The spatial transformation matrix calculation submodule is used to calculate the spatial transformation matrix from the modal space to the reference modal space based on the feature point pair set;

[0089] The image group registration output submodule is used to perform spatial geometric transformation and resampling on infrared and ultraviolet images using a spatial transformation matrix, and output the registered image group.

[0090] A computer storage medium, comprising: at least one memory and at least one processor;

[0091] Memory, used to store one or more program instructions;

[0092] A processor for running one or more program instructions to execute the substation oil leakage defect detection method based on multi-technology integration as described in any one of claims 1-5.

[0093] The beneficial effects of the invention are as follows: It integrates multimodal technology with dynamic fusion network to solve the problems of single-modal detection being susceptible to environmental interference and having low accuracy, and accurately identifies oil leakage defects in substations; it does not require a lot of manual intervention, improves detection efficiency, ensures the safe and stable operation of the power system, and is suitable for complex substation scenarios. Attached Figure Description

[0094] The present invention will be further described below with reference to the accompanying drawings.

[0095] Figure 1This is a flowchart illustrating a substation oil leakage defect detection method based on multi-technology integration according to the present invention.

[0096] Figure 2 This is a schematic diagram of the processing flow of a substation oil leakage defect detection system based on multi-technology integration according to the present invention.

[0097] Figure 3 This diagram illustrates the connection between the multimodal image acquisition module, spatial configuration module, high-dimensional feature extraction module, high-dimensional feature fusion module, and oil leakage defect determination module in a substation oil leakage defect detection system based on multi-technology fusion according to the present invention. Detailed Implementation

[0098] See attached document Figures 1-3 The present invention will be further described below.

[0099] like Figure 1 As shown, Embodiment 1 of this application provides a substation oil leakage defect detection method based on multi-technology fusion, including:

[0100] Step S1: Simultaneously acquire images of the equipment under test using image acquisition devices deployed in the substation equipment area; the image acquisition devices are visible light cameras, infrared thermal imagers, and ultraviolet imagers, which acquire visible light images, infrared thermal images, and ultraviolet images respectively.

[0101] The deployment and data acquisition process for each data acquisition device is as follows:

[0102] Step S11: Configure the parameters of the visible light camera, infrared thermal imager, and ultraviolet imager to ensure clear imaging;

[0103] For visible light cameras, parameters are set according to the lighting conditions at the substation site, including: adjusting the resolution to 1920×1080 pixels, setting the frame rate to 25fps, automatically adapting the exposure time to the ambient light (range 1 / 1000s-1 / 50s), and enabling the autofocus function to ensure clear details on the device surface.

[0104] For the infrared thermal imager, the following parameters are set: resolution of 640×512 pixels, temperature measurement range adjusted to cover the temperature range of normal equipment operation and possible oil leakage, emissivity set according to the surface material of the equipment (0.9 for metal surfaces, 0.85 for insulating surfaces), and frame rate consistent with that of the visible light camera.

[0105] For the ultraviolet imager, the configuration parameters include: setting the ultraviolet band filtering range to 240nm-280nm (to match the fluorescent characteristics of insulating oil), adjusting the gain to a medium level (to avoid excessive environmental ultraviolet interference), synchronizing the frame rate to 25fps, and enabling the overlay mode with visible light images.

[0106] Step S12: Send a synchronous acquisition command through the central control system to trigger the three devices to start acquiring data simultaneously;

[0107] The acquisition duration is set according to the equipment inspection requirements. In this embodiment, it is set to last 3-5 minutes for a single acquisition to generate a continuous image sequence.

[0108] Each device adds a timestamp (accurate to milliseconds) to the image data it collects in real time, and stores it in the local cache according to the naming rule of "device type-collection time-frame number" (e.g., visible light images are named "VIS_20240520103000_0001.jpg").

[0109] Step S13: Align the three types of images acquired by timestamp to form three sets of parallel data corresponding to the same detection time period.

[0110] Step S2: Spatial registration of the acquired visible light image, infrared thermal image, and ultraviolet image;

[0111] This step aims to address pixel deviation issues in visible light, infrared, and ultraviolet images caused by differences in shooting angle, resolution, and sensor physical location, achieving precise pixel-level alignment of the same device region in images of different modalities. The specific implementation process is as follows:

[0112] Step S21: Using the visible light image as the reference mode, extract the feature point pair set corresponding to the reference mode from the other two types of images;

[0113] For the baseline mode (visible light image), the Scale Invariant Feature Transform (SIFT) algorithm is used to extract a set of strong feature points. Priority is given to selecting areas on the equipment surface that have significant corner or blob features and are not easily deformed, such as equipment edges, bolts, and nameplate corners.

[0114] On the infrared and ultraviolet images to be registered, an automated script extracts corresponding feature points near the physical locations corresponding to the feature points in the reference image, forming feature point pairs. After processing, a set of feature point pairs is obtained. Due to the significant spectral differences between multimodal images, the automatic feature matching algorithm has limited effectiveness. Therefore, this embodiment recommends a semi-automatic manual calibration method to ensure accuracy: before generating the first set of registration parameters, at least four sets (8-12 sets recommended) of evenly distributed feature point pairs spanning the four quadrants of the image are precisely clicked on the reference image and the image to be registered through a human-computer interaction interface.

[0115] Step S22: Calculate the spatial transformation matrix from the modal space to the reference modal space based on the feature point pair set;

[0116] For registration from the modal space to be registered to the reference modal space, the transformation relationship is as follows:

[0117] Where x and y are the coordinates in the modal space to be registered. , H represents the coordinates in the reference modal space, and H is a 3x3 matrix.

[0118] Based on the above transformation relationship, the RANSAC (Random Sample Consensus) algorithm is used to estimate the optimal transformation matrix H1 from the reference mode (visible light image) - infrared image feature point pair; then the same algorithm is used to estimate the optimal transformation matrix H2 from the reference mode - ultraviolet image feature point pair; these two transformation matrices are the spatial transformation matrices from the mode space to be registered to the reference mode space.

[0119] Step S23: Perform spatial geometric transformation and resampling on the infrared and ultraviolet images using the spatial transformation matrix to complete pixel-level spatial registration;

[0120] To avoid holes and overlaps, a reverse mapping method is used: For each pixel in the reference mode (visible light image) coordinate system, denoted as the source pixel, its corresponding position (x', y') in the image to be registered is calculated using the inverse of the transformation matrix. Since the calculated corresponding position (x', y') is usually a sub-pixel precision non-integer coordinate, a bicubic interpolation algorithm is also needed to calculate the accurate pixel coordinates at this position. Then, these pixel coordinates are correlated with the source pixel coordinates to obtain the registered image. Infrared and ultraviolet images are spatially registered using the above method. The registered visible light image, infrared image, and ultraviolet image are stored as a single image group.

[0121] Step S3: Input the registered image group into the cross-modal alignment-guided feature pyramid encoder and output three high-dimensional feature vectors;

[0122] The cross-modal alignment-guided feature pyramid encoder consists of a shared bottom layer, multiple modality-specific network branches, and a fusion layer. Its data processing flow is as follows:

[0123] Step S31: Extract common low-level features from all input images using a shared underlying layer;

[0124] The shared-weight convolutional network at the bottom layer extracts common low-level features such as edges and freckles; the output feature map size is H*W*Clow, where H*W is the visible light image resolution and Clow is the number of channels (64 in this embodiment). The low-level features extracted from the visible light, infrared, and ultraviolet images are represented as follows: , , .

[0125] Step S32: Input the extracted low-level features into the network branches corresponding to each image modality to extract the high-level features of each input image;

[0126] High-level features are further extracted from the low-level features of each modality. Each modality uses two layers of dedicated convolution with independent weights (3*3 kernels, stride 2, padding=1) + ReLU activation, with the following parameters:

[0127] Visible light exclusive: Infrared Exclusive: UV exclusive: ;

[0128] The specific convolution representations for the three modalities are as follows:

[0129] , among them Indicates the convolution operation;

[0130] The output size is (H / 4)*(W / 4)*Chigh. Because the convolution with stride 2 is performed twice, the size is reduced to 1 / 4, and Chigh=256. The output high-level features are represented as follows: , , .

[0131] Step S33: Upsample and fuse the low-level and high-level features to obtain the final feature map;

[0132] The extracted low-level and high-level features are fed into the fusion layer, and the high-level feature map is upsampled: bilinear interpolation is used to enlarge the high-level feature Fhigh,j to H*W (visible light image resolution), denoted as . , j is the modality index, with values ​​vis, ir, uv; the low-level feature flow is transformed by a 1*1 convolution, and the number of channels j is adjusted to 256, denoted as The element-wise high-level features after upsampling are fused with the low-level features after channel adjustment to output the final feature map. .

[0133] The fusion layer described above is represented as follows:

[0134] , in In response to 1*1 convolution kernel (adjust channels). These are the bias parameters used in the convolution operation. Indicating targeting Four times the upsampling.

[0135] Step S34: Introduce translation loss during training so that the encoder can automatically learn to align feature maps of different modalities in the abstract feature space;

[0136] In order to To ensure consistent representation of the same physical region in the feature space, a translation loss function is introduced during encoder training. A translator G (a small MLP) is set after the fusion layer to achieve mutual translation of features from different modalities. Under the constraint of the translation loss function, the "translation" from visible light features should resemble infrared features, and vice versa. By minimizing this loss, the model will spontaneously ensure that the final feature maps of the two modalities have highly correlated representations at corresponding spatial locations, thereby achieving precise alignment at the feature level, not just pixel-level registration. The translation loss function is expressed as:

[0137] Where E() is a function that calculates the mean on the training data, j and i are modality indices, and j ≠ i. and These are the translator network branches from mode j to mode i and from mode i to mode j, respectively.

[0138] Step S35: Convert the final fused feature map into a fixed-length high-dimensional feature vector;

[0139] The final feature map is subjected to global average pooling (GAP) and flattening operations to obtain a fixed-length high-dimensional feature vector. .

[0140] Step S4: Based on the multimodal dynamic fusion network, the three high-dimensional feature vectors are fused to generate a defect judgment score;

[0141] The multimodal dynamic fusion network incorporates a dynamic feature selection unit that simulates the "doubt" and "acceptance" process in human expert decision-making. This allows the model to learn to prioritize the most reliable evidence in specific scenarios. Its specific data processing logic is as follows:

[0142] Step S41: Acquire and encode ambient light, humidity, and noise data in real time to form environmental context variables;

[0143] By deploying a sensor array in the substation detection area, key environmental parameters affecting the accuracy of multimodal detection are collected synchronously, including: illumination, humidity, and noise data. The acquisition frequency is kept consistent with the image acquisition frame rate to ensure spatiotemporal synchronization. The collected data is normalized and encoded to form a low-dimensional environmental context vector R.

[0144] Step S42: Calculate the doubt score for each modality by combining the generated environmental context quantity;

[0145] The formula for calculating the doubtful score is as follows: Where j is the modal index, with values ​​vis, ir, and uv. Let R represent the doubt score for the j-th modality, and let R be the context vector. It is the attention weight matrix, learned through training, used for the interaction between modal features (i.e., each high-dimensional feature vector) and the environmental context. Let j be the high-dimensional eigenvector of the j-th mode. Let be the characteristic variance of the j-th mode. The result within the parentheses is the maximum value obtained after applying Softmax. , These are globally questionable parameters, learned through training.

[0146] Step S43: Input the three high-dimensional feature vectors and the doubt score into the dynamic feature selection unit to generate the fused feature vector;

[0147] The dynamic feature selection unit is expressed by the mathematical formula as follows:

[0148] ,in This is the generated fusion feature vector.

[0149] Step S144: Feed the fused feature vector into the classifier to obtain the defect determination score;

[0150] fuse feature vectors Input a lightweight classifier and output a defect judgment score between 0 and 1. The higher the score, the greater the probability of oil leakage. The classifier uses a 2-layer fully connected network (hidden layer dimension 128, output layer dimension 1) + Sigmoid activation.

[0151] Step S5: Compare the defect judgment score with the preset threshold. If it is higher than the threshold, it is judged that there is an oil leakage defect; otherwise, it is judged that there is no defect.

[0152] like Figure 2 As shown, this application provides a substation oil leakage defect detection system based on multi-technology fusion, including: a multimodal image acquisition module 21, a spatial configuration module 22, a high-dimensional feature extraction module 23, a high-dimensional feature fusion module 24, and an oil leakage defect determination module 25.

[0153] The multimodal image acquisition module 21 is used to simultaneously acquire images of the equipment under test through image acquisition devices deployed in the substation equipment area; the image acquisition devices are visible light cameras, infrared thermal imagers and ultraviolet imagers, which acquire visible light images, infrared thermal images and ultraviolet images respectively; specifically, it includes: an imaging device configuration submodule, an acquisition command triggering submodule and a timestamp alignment submodule;

[0154] 1. Imaging equipment configuration submodule, used to configure the parameters of visible light camera, infrared thermal imager, and ultraviolet imager to ensure clear imaging;

[0155] 2. Acquisition command triggering submodule, used to send synchronous acquisition commands through the central control system to trigger the three devices to start acquiring data simultaneously;

[0156] 3. The timestamp alignment submodule is used to align the three types of images acquired according to their timestamps, forming three sets of parallel data corresponding to the same detection period.

[0157] Spatial configuration module 22 is used for spatial registration of visible light images, infrared thermal imaging and ultraviolet imaging; specifically it includes: a feature point pair extraction submodule, a spatial transformation matrix calculation submodule, and a registered image group output submodule.

[0158] 1. Feature point pair extraction submodule, which uses the visible light image as the reference mode and extracts the feature point pair set corresponding to the reference mode from the other two types of images;

[0159] 2. Spatial transformation matrix calculation submodule, used to calculate the spatial transformation matrix from the modal space to be registered to the reference modal space based on the feature point pair set;

[0160] 3. The image group registration output submodule is used to perform spatial geometric transformation and resampling on infrared and ultraviolet images using a spatial transformation matrix, and output the registered image group.

[0161] The high-dimensional feature extraction module 23 is used to input the registered image group into the cross-modal alignment-guided feature pyramid encoder and output three high-dimensional feature vectors.

[0162] The high-dimensional feature fusion module 24 is used to fuse three high-dimensional feature vectors using a multimodal dynamic fusion network to generate a defect judgment score.

[0163] The oil leakage defect determination module 25 is used to compare the defect determination score with a preset threshold. If the score is higher than the threshold, it is determined that there is an oil leakage defect; otherwise, it is determined that there is no defect.

[0164] Corresponding to the above embodiments, the present invention provides a computer storage medium, including: at least one memory and at least one processor;

[0165] The memory is used to store one or more program instructions;

[0166] A processor is used to run one or more program instructions to execute a substation oil leakage defect detection method based on multi-technology integration.

[0167] Corresponding to the above embodiments, this embodiment of the invention provides a computer-readable storage medium containing one or more program instructions, which are executed by a processor to provide a substation oil leakage defect detection method based on multi-technology fusion.

[0168] The embodiments disclosed in this invention provide a computer-readable storage medium storing computer program instructions. When the computer program instructions are executed on a computer, the computer performs the aforementioned method for detecting oil leakage defects in substations based on multi-technology integration.

[0169] In this embodiment of the invention, the processor can be an integrated circuit chip with signal processing capabilities. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0170] The various methods, steps, and logic diagrams disclosed in the embodiments of this invention can be implemented or executed. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this invention can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The processor reads information from the storage medium and, in conjunction with its hardware, completes the steps of the above methods.

[0171] The storage medium can be memory, such as volatile memory or non-volatile memory, or may include both volatile and non-volatile memory.

[0172] Among them, non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory.

[0173] Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (Synchlink DRAM, SLDRAM), and direct memory bus RAM (DRRAM).

[0174] The storage media described in the embodiments of the present invention are intended to include, but are not limited to, these and any other suitable types of memory.

[0175] Those skilled in the art will recognize that, in one or more of the examples above, the functions described in this invention can be implemented using a combination of hardware and software. When applied as software, the corresponding functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any medium that facilitates the transmission of computer programs from one place to another. Storage media can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0176] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for detecting oil leakage defects in substations based on multi-technology integration, characterized in that, include: S1. Simultaneously acquire images of the equipment under test using image acquisition devices deployed in the substation equipment area; The image acquisition equipment consists of a visible light camera, an infrared thermal imager, and an ultraviolet imager, which respectively acquire visible light images, infrared thermal images, and ultraviolet images. S2. Spatial registration of the acquired visible light images, infrared thermal images, and ultraviolet images; S3. Input the registered image group into the cross-modal alignment-guided feature pyramid encoder and output three high-dimensional feature vectors; S4. Based on a multimodal dynamic fusion network, three high-dimensional feature vectors are fused to generate a defect judgment score; S5. Compare the defect judgment score with the preset threshold. If it is higher than the threshold, it is judged that there is an oil leakage defect; otherwise, it is judged that there is no defect.

2. The method for detecting oil leakage defects in substations based on multi-technology integration as described in claim 1, characterized in that, In step S1, the deployment and data acquisition process for each acquisition device is as follows: Step S11: Configure the parameters of the visible light camera, infrared thermal imager, and ultraviolet imager to ensure clear imaging; For visible light cameras, set parameters according to the lighting conditions at the substation site; Step S12: Send a synchronous acquisition command through the central control system to trigger the visible light camera, infrared thermal imager, and ultraviolet imager to start acquiring data simultaneously. The acquisition duration is set according to the equipment inspection requirements to generate a continuous image sequence; Step S13: Align the three types of images acquired by timestamp to form three sets of parallel data corresponding to the same detection time period.

3. The method for detecting oil leakage defects in substations based on multi-technology integration as described in claim 1, characterized in that, Step S2: Spatial registration of the acquired visible light image, infrared thermal image, and ultraviolet image; Step S21: Using the visible light image as the reference mode, extract the feature point pair set corresponding to the reference mode from the other two types of images; Step S22: Calculate the spatial transformation matrix from the modal space to the reference modal space based on the feature point pair set; Step S23: Perform spatial geometric transformation and resampling on the infrared and ultraviolet images using the spatial transformation matrix to complete pixel-level spatial registration.

4. The substation oil leakage defect detection method based on multi-technology integration according to claim 3, characterized in that, Step S21: Using the visible light image as the reference mode, extract the feature point pair set corresponding to the reference mode from the other two types of images; For the baseline mode, the SIFT algorithm with scale-invariant feature transformation is used to extract the set of strong feature points; Prioritize areas on the equipment surface that have prominent corner or spot features and are not easily deformed; On the infrared and ultraviolet images to be registered, the corresponding feature points are extracted near the physical locations corresponding to the feature points of the reference image by an automated script, forming feature point pairs, and then the feature point pair set is obtained after sorting. Before generating the first set of registration parameters, at least four pairs of identical feature points, evenly distributed and spanning the four quadrants of the image, are precisely clicked on the reference image and the image to be registered through the human-computer interaction interface. Step S22: Calculate the spatial transformation matrix from the modal space to the reference modal space based on the feature point pair set; For registration from the modal space to be registered to the reference modal space, the transformation relationship is as follows: Where x and y are the coordinates in the modal space to be registered. , These are the coordinates in the reference modal space, and H is a 3x3 matrix; Based on the above transformation relationship, the optimal transformation matrix H1 is estimated from the reference mode-infrared image feature point pair using the RANSAC algorithm; then the optimal transformation matrix H2 is estimated from the reference mode-ultraviolet image feature point pair using the same algorithm; these two transformation matrices are the spatial transformation matrices from the mode space to be registered to the reference mode space.

5. The substation oil leakage defect detection method based on multi-technology integration according to claim 3, characterized in that, Step S23: Perform spatial geometric transformation and resampling on the infrared and ultraviolet images using the spatial transformation matrix to complete pixel-level spatial registration; To avoid holes and overlaps, a reverse mapping method is used: For each pixel in the reference modal coordinate system, denoted as the source pixel, its corresponding position (x', y') in the image to be registered is calculated using the inverse of the transformation matrix. Since the calculated corresponding position (x', y') is a sub-pixel precision non-integer coordinate, a bicubic interpolation algorithm is also needed to calculate the accurate pixel coordinates at this position. Then, these pixel coordinates are correlated with the source pixel coordinates to obtain the registered image. Infrared and ultraviolet images are spatially registered using the above method, and the registered visible light image, infrared image, and ultraviolet image are stored as a single image group.

6. The method for detecting oil leakage defects in substations based on multi-technology integration according to claim 1, characterized in that, Step S3: Input the registered image group into the cross-modal alignment-guided feature pyramid encoder and output three high-dimensional feature vectors; The cross-modal alignment-guided feature pyramid encoder consists of a shared bottom layer, multiple modality-specific network branches, and a fusion layer. Its data processing flow is as follows: Step S31: Extract common low-level features from all input images using a shared underlying layer; Step S32: Input the extracted low-level features into the network branches corresponding to each image modality to extract the high-level features of each input image; Step S33: Upsample and fuse the low-level and high-level features to obtain the final feature map; Step S34: Introduce translation loss during training so that the encoder can automatically learn to align feature maps of different modalities in the abstract feature space; Step S35: Convert the final fused feature map into a fixed-length high-dimensional feature vector.

7. The method for detecting oil leakage defects in substations based on multi-technology integration according to claim 6, characterized in that, Step S31: Extract common low-level features from all input images using a shared underlying layer; The shared underlying layer uses convolutional networks with shared weights to extract common low-level features; The output feature map size is H*W*Clow, where H*W is the visible light image resolution and Clow is the number of channels. The low-level features extracted from the visible light image, infrared image, and ultraviolet image are represented as follows: , and .

8. The substation oil leakage defect detection method based on multi-technology integration according to claim 6, characterized in that, Step S32: Input the extracted low-level features into the network branches corresponding to each image modality to extract the high-level features of each input image; High-level features are further extracted from the low-level features of each modality. Each modality uses a dedicated convolutional layer with two independent weights and ReLU activation. The convolutional kernel is 3*3, the stride is 2, and the padding is 1. The parameters are as follows: Visible light exclusive: Infrared Exclusive: UV exclusive: ; The specific convolution representations for the three modalities are as follows: , among them Indicates the convolution operation; The output size is (H / 4)*(W / 4)*Chigh. Because the convolution with stride 2 is performed twice, the size is reduced to 1 / 4, and Chigh=256. The output high-level features are represented as follows: , as well as .

9. The method for detecting oil leakage defects in substations based on multi-technology integration according to claim 6, characterized in that, Step S33: Upsample and fuse the low-level and high-level features to obtain the final feature map; The extracted low- and high-level features are fed into a fusion layer, and the high-level feature map is upsampled: bilinear interpolation is used to enlarge the high-level feature Fhigh,j to H*W of the visible light image resolution, denoted as... , j is the modality index, with values ​​vis, ir, uv; the low-level feature flow is transformed by a 1*1 convolution, and the number of channels j is adjusted to 256, denoted as ; Element-wise addition and fusion of upsampled high-level features and channel-adjusted low-level features yields the final feature map. ; The fusion layer is represented as follows: ,in In response to 1*1 convolution kernel (adjust channels). These are the bias parameters used in the convolution operation. Indicating targeting Four times the upsampling.

10. The method for detecting oil leakage defects in substations based on multi-technology integration according to claim 6, characterized in that, Step S34: Introduce translation loss during training so that the encoder can automatically learn to align feature maps of different modalities in the abstract feature space; In order to To ensure consistent representation of the same physical region in the feature space, a translation loss function is introduced during encoder training. A translator G is set after the fusion layer. The translator G is a small MLP used to realize the mutual translation of features from different modalities. Under the forced constraint of the translation loss function, the "translation" from visible light features should be like infrared features, and vice versa. By minimizing this loss, the final feature maps of the two modalities will spontaneously have highly correlated representations at corresponding spatial locations, thereby achieving precise alignment at the feature level, rather than just pixel-level registration; the translation loss function is expressed as: ； Where E() is a function that calculates the mean on the training data, j and i are modality indices, and j ≠ i. and These are the translator network branches from mode j to mode i and from mode i to mode j, respectively. Step S35: Convert the final fused feature map into a fixed-length high-dimensional feature vector; The final feature map is subjected to global average pooling (GAP) and flattening operations to obtain a fixed-length high-dimensional feature vector. .

11. The method for detecting oil leakage defects in substations based on multi-technology integration according to claim 1, characterized in that, Step S4: Based on the multimodal dynamic fusion network, the three high-dimensional feature vectors are fused to generate a defect judgment score; The multimodal dynamic fusion network incorporates a dynamic feature selection unit that simulates the "doubt" and "acceptance" process in human expert decision-making. It learns to prioritize the most reliable evidence in specific scenarios. Its specific data processing logic is as follows: Step S41: Acquire and encode ambient light, humidity, and noise data in real time to form environmental context variables; Step S42: Calculate the doubt score for each modality by combining the generated environmental context quantity; Step S43: Input the three high-dimensional feature vectors and the doubt score into the dynamic feature selection unit to generate the fused feature vector; Step S44: Feed the fused feature vector into the classifier to obtain the defect determination score.

12. The method for detecting oil leakage defects in substations based on multi-technology integration according to claim 11, characterized in that, Step S41: Acquire and encode ambient light, humidity, and noise data in real time to form environmental context variables; By deploying a sensor array in the substation detection area, key environmental parameters affecting the accuracy of multimodal detection are collected synchronously, including: illumination, humidity and noise data. The acquisition frequency is kept consistent with the image acquisition frame rate to ensure spatiotemporal synchronization. The collected data is normalized and encoded to form a low-dimensional environmental context vector R.

13. The method for detecting oil leakage defects in substations based on multi-technology fusion according to claim 11, characterized in that, Step S42: Calculate the doubt score for each modality by combining the generated environmental context quantity; The formula for calculating the doubtful score is as follows: Where j is the modal index, with values ​​vis, ir, and uv. Let R represent the doubt score for the j-th modality, and let R be the context vector. It is the attention weight matrix, learned through training, used for the interaction between modal features (i.e., each high-dimensional feature vector) and the environmental context. Let j be the high-dimensional eigenvector of the j-th mode. Let be the characteristic variance of the j-th mode. The result within the parentheses is the maximum value obtained after applying Softmax. , These are globally questionable parameters, learned through training.

14. The method for detecting oil leakage defects in substations based on multi-technology integration according to claim 11, characterized in that, Step S43: Input the three high-dimensional feature vectors and the doubt score into the dynamic feature selection unit to generate the fused feature vector; The dynamic feature selection unit is expressed by the mathematical formula as follows: ,in This is the generated fusion feature vector.

15. The method for detecting oil leakage defects in substations based on multi-technology integration according to claim 11, characterized in that, Step S44: Feed the fused feature vector into the classifier to obtain the defect determination score; fuse feature vectors The input is a lightweight classifier, which outputs a defect judgment score between 0 and 1. The higher the score, the greater the probability of oil leakage. The classifier uses a 2-layer fully connected network (hidden layer dimension 128, output layer dimension 1) + Sigmoid activation.

16. A substation oil leakage defect detection system based on multi-technology integration, characterized in that, The method for detecting substation oil leakage defects based on multi-technology fusion as described in any one of claims 1-15 includes a multi-modal image acquisition module, a spatial configuration module, a high-dimensional feature extraction module, a high-dimensional feature fusion module, and an oil leakage defect determination module that are sequentially connected by signals. The multimodal image acquisition module is used to simultaneously acquire images of the equipment under test through image acquisition devices deployed in the substation equipment area; the image acquisition devices are visible light cameras, infrared thermal imagers, and ultraviolet imagers, which acquire visible light images, infrared thermal images, and ultraviolet images respectively. The spatial configuration module is used for spatial registration of visible light images, infrared thermal imaging, and ultraviolet imaging. The high-dimensional feature extraction module is used to input the registered image group into the cross-modal alignment-guided feature pyramid encoder and output three high-dimensional feature vectors. The high-dimensional feature fusion module is used to fuse three high-dimensional feature vectors using a multimodal dynamic fusion network to generate a defect judgment score. The oil leakage defect determination module is used to compare the defect determination score with a preset threshold. If the score is higher than the threshold, it is determined that there is an oil leakage defect; otherwise, it is determined that there is no defect.

17. The substation oil leakage defect detection system based on multi-technology integration according to claim 16, characterized in that, The multimodal image acquisition module specifically includes: an imaging device configuration submodule for sequential signal transmission, an acquisition command triggering submodule, and a timestamp alignment submodule; The imaging equipment configuration submodule is used to configure the parameters of the visible light camera, infrared thermal imager, and ultraviolet imager to ensure clear imaging. The data acquisition command triggering submodule is used to send a synchronous data acquisition command through the central control system to trigger the three devices to start data acquisition simultaneously. The timestamp alignment submodule is used to align the three types of images acquired according to their timestamps, forming three sets of parallel data corresponding to the same detection period.

18. The substation oil leakage defect detection system based on multi-technology integration according to claim 16, characterized in that, The spatial configuration module specifically includes: a feature point pair extraction submodule, a spatial transformation matrix calculation submodule, and a registered image group output submodule; The feature point pair extraction submodule is used to extract the set of feature point pairs corresponding to the reference mode from the other two types of images, using the visible light image as the reference mode. The spatial transformation matrix calculation submodule is used to calculate the spatial transformation matrix from the modal space to the reference modal space based on the feature point pair set; The image group registration output submodule is used to perform spatial geometric transformation and resampling on infrared and ultraviolet images using a spatial transformation matrix, and output the registered image group.

Images