SF6 gas leakage detection method based on multi-modal image fusion

Abstract

The invention discloses an SF6 gas leakage detection method based on multi-modal image fusion. The SF6 gas leakage detection method comprises the following steps: acquiring a visible light image and a corresponding infrared image of electrical equipment with SF6 gas leakage in a current environment; inputting the visible light image and the infrared image into an image fusion model to obtain a fused image; and inputting the fused image into an SF6 gas leakage detection model to obtain a prediction frame of an actual SF6 gas leakage area output by the SF6 gas leakage detection model. By means of the method, it can be ensured that even under the condition of tiny leakage, relatively accurate leakage point position information can be obtained.

Description

Technical Field

[0001] The present invention relates to the technical field of SF6 gas leakage detection, and in particular to a SF6 gas leakage detection method based on multimodal image fusion. Background Art

[0002] Sulfur hexafluoride SF6 is a colorless, odorless, non-toxic and non-flammable inert gas with excellent insulation and arc extinguishing properties. Therefore, it is widely used in the power industry, especially in high-voltage switchgear such as circuit breakers, disconnectors and GIS systems as an insulating medium and arc extinguishing medium. However, the global warming potential of SF6 gas is very high, about 23,500 times that of carbon dioxide, and its lifetime in the atmosphere can reach thousands of years. This means that even a small amount of SF6 gas leaking into the atmosphere can have a significant impact on the environment. Therefore, SF6 gas leak detection has become a very important issue in the power industry.

[0003] There are various methods for detecting SF6 gas leakage, including scientific methods such as electrochemical technology, electrical breakdown technology, infrared spectroscopy absorption technology, and traditional methods such as soap bubble detection and wrapping collection detection. Among them, infrared imaging technology has been widely used in modern power equipment maintenance due to its advantages of long-distance, non-contact, high sensitivity and intuitive imaging. However, infrared imaging is greatly affected by environmental factors, and the precise positioning of tiny leakage sources may result in poor image quality and low accuracy.

[0004] Infrared imaging technology generates images based on the infrared radiation emitted by the object itself, and can detect and visualize the temperature differences between different objects or parts of the same object. This technology is particularly suitable for nighttime or low-light conditions to meet the imaging needs in these environments.

[0005] Visible light imaging technology is a method of image acquisition and processing using light within the spectrum that can be recognized by the human eye. This technology captures the visible light reflected or emitted in the scene through an optical lens and converts these light signals into electronic signals, ultimately generating a clear image.

[0006] The VGG19 network model is a deep convolutional neural network known for its simplicity and effectiveness. It was developed by the Visual Geometry Group of the University of Oxford and made its mark in the ImageNet competition in 2014. The network contains 19 weight layers (16 convolutional layers plus 3 fully connected layers), all of which use fixed-size 3x3 convolution kernels and are downsampled with 5 maximum pooling layers. The design emphasizes deepening the network depth to enhance feature representation capabilities. It is widely used in image recognition tasks and is fine-tuned as a basic model in various applications due to the availability of pre-trained models.

[0007] Faster R-CNN is a deep learning framework for object detection. It replaces the selective search mechanism with a region proposal network (RPN), which implements an end-to-end detection process and greatly improves the detection speed. RPN uses a convolutional neural network to generate candidate regions, which are then refined and classified by a detector and regressed by bounding boxes, enabling the model to achieve real-time detection while ensuring accuracy.

[0008] Object detection is an important branch of computer vision research. Its main task is to identify and accurately locate one or more specific target objects within a provided image or video frame. This technology uses algorithms to automatically determine the location of the target object and classify it into categories. The object detection process usually includes two key steps: first, bounding box positioning, which is to define the specific location of the object in the image; second, classification recognition, which is to determine the specific category to which the object belongs.

[0009] Visible light and infrared image fusion is a multimodal image processing technology that aims to integrate the rich texture and color information in visible light images and the thermal radiation characteristics in infrared images to generate new images that combine the advantages of both. This technology can enhance image details and improve the ability to understand the scene. With the help of various image fusion algorithms, such as wavelet transform-based methods, frequency domain fusion technology, or deep learning technology, data from visible light and infrared sensors can be effectively integrated to achieve more accurate and robust target recognition and environmental perception. Summary of the invention

[0010] In order to solve the problem of low accuracy of existing SF6 leakage identification technology in the case of small leakage, the present invention provides a SF6 gas leakage detection method based on multimodal image fusion, which can ensure that relatively accurate leakage point location information can be obtained even in the case of small leakage.

[0011] The technical solution adopted by the present invention is:

[0012] The SF6 gas leakage detection method based on multimodal image fusion includes the following steps:

[0013] Step 1: Obtain visible light images and corresponding infrared images of electrical equipment with SF6 gas leakage in the current environment;

[0014] Step 2: Input the visible light image and the infrared image into the image fusion model to obtain a fused image;

[0015] Step 3: Input the fused image into the SF6 gas leakage detection model to obtain the prediction box of the actual SF6 gas leakage area output by the SF6 gas leakage detection model.

[0016] In step 2, the image fusion model is obtained based on the sample visible light image and infrared image in combination with the VGG19 network model.

[0017] The step 2 comprises the following steps:

[0018] S2.1: The image fusion model first performs convolution on the visible light image and the infrared image respectively, and then retains the output of the five convolution layers;

[0019] S2.2: After extracting the required features and targets, the activity level map is obtained by L1 norm and average operator;

[0020] S2.3: Use the softmax function and upsampling operator to obtain a weight map, and perform five different weight mapping convolutions on the input visible light image and infrared image to obtain a candidate fusion image;

[0021] S2.4: Finally, the maximum fusion strategy is used to generate the final fused image from the five candidate fused images to complete the image reconstruction. In S2.1, the first convolutional layer conv1_1 and the second convolutional layer conv1_2 of the VGG19 network model extract details and edges from the source image, and these two convolutional layers are retained;

[0022] The 3rd convolutional layer conv2_1 and the 4th convolutional layer conv2_2 of the VGG19 network model extract the edges in the image. By comparison, the 3rd convolutional layer conv2_1 is retained, and the edge extraction of this layer is more sufficient than that of the 4th convolutional layer conv2_2.

[0023] The 5th convolutional layer conv3_1, the 6th convolutional layer conv3_2, the 7th convolutional layer conv3_3, and the 8th convolutional layer conv3_4 of the VGG19 network model extract prominent targets in the image. By comparison, the 5th convolutional layer conv3_1 is retained, and the extraction effect of this layer is better than that of the other 3 layers;

[0024] The 9th convolutional layer conv4_1, the 10th convolutional layer conv4_2, the 11th convolutional layer conv4_3, and the 12th convolutional layer conv4_4 of the VGG19 network model also extract salient targets in the image. Compared with the four layers of the third convolutional layer group, the targets extracted by these four layers are more blurred, but the weights of the targets are also higher, which helps to highlight the salient targets in the infrared image in the subsequent fusion process. In contrast, the 9th convolutional layer conv4_1 is retained.

[0025] The 13th convolutional layer conv5_1, the 14th convolutional layer conv5_2, the 15th convolutional layer conv5_3, and the 16th convolutional layer conv5_4 of the VGG19 network model are no longer retained and are all removed.

[0026] In summary, the first convolutional layer conv1_1, the second convolutional layer conv1_2, the third convolutional layer conv2_1, the fifth convolutional layer conv3_1 and the ninth convolutional layer conv4_1 in VGG-19 are applied to the fusion method as the basis of transfer learning.

[0027] To facilitate subsequent calculations, the above five convolutional layers are reordered from Conv1 to Conv5. Extracting feature maps from convolutional layers represents I extracted from the kth convolutional layer n The feature map of , m is the number of channels of the kth layer.

[0028]

[0029] Where, k∈{1,2,3,4,5} represents Conv1, Conv2, Conv3, Conv4, Conv5, respectively, m∈{1,2,...,M}, where M of Conv1, Conv2, Conv3, Conv4, Conv5 is 64, 64, 128, 256, 512 respectively; Ψ k (·) represents the convolutional layer in the VGG19 network model.

[0030] In S2.2, the intuitive image is converted into an objective data distribution through the L1 norm, so that the amount of information contained in different images can be reflected through the data. It is obtained from the following formula:

[0031]

[0032] In the formula, The measure of the amount of information in the source image, that is, the measure of the activity level of the source image, can transform the intuitive image into an objective data distribution, thereby reflecting the amount of information contained in different images through data. is an M-dimensional vector, representing the feature map (x, y) The content of; (x, y) represents the position coordinates in the feature map.

[0033] The average operator calculation formula is used to make the fusion method robust to image mismatches, and the final activity level map is shown in the following formula:

[0034]

[0035] Where q represents the block size, q = 1; Represents the final activity level map after smoothing using the average operator. It represents the activity level value at a certain offset position in the image; γ and θ represent the offset values ​​in the horizontal and vertical directions respectively.

[0036] In S2.3, after obtaining the final activity level map, the softmax function is used to calculate the initial weight map, and the calculation formula is as follows:

[0037]

[0038] In the formula, let N be 2, represents the initial weight map; Represents the activity level map; all weight map values ​​are in the range of [0,1]. After obtaining the initial weight map, the size of the initial weight map is adjusted by upsampling to adapt to the size of the source image;

[0039]

[0040] In the formula, Represents the final weight map after upsampling The value at position (x+w,y+v); represents the initial weight map; w represents the displacement value of the horizontal axis; v represents the displacement value of the vertical axis.

[0041] The upsampling process is: each point in the preliminary weight map will be expanded into a block in the final weight map, and all element values ​​in this block are equal to the value of the point in the preliminary weight map, that is, when the preliminary weight map is enlarged, for each newly inserted (x+w,y+v) position, its value is equal to the value of the original position (x,y), where w,v belongs to {0,1,...,2^(a)-1}, and a is the number of convolutional layer groups corresponding to the weight map. For Conv1 and Conv2, since they are both in the first convolutional layer group, a=1; for Conv3, since it is in the second convolutional layer group, a=2; for Conv4, since it is in the third convolutional layer group, a=3; for Conv5, since it is in the fourth convolutional layer group, a=4.

[0042] Finally, the source images are weighted and five different candidate fusion images are obtained by the following formula:

[0043]

[0044] Where Fk(x,y) represents the candidate fusion image, represents the final weight graph, I i (x,y) represents the source infrared image and visible light image.

[0045] In S2.4, finally, the maximum value of each pixel is selected from the five candidate fused images as the final result according to the following formula to obtain the final fused image:

[0046] F(x,y)=max|F k (x,y)

[0047] Where F(x,y) represents the final fused image.

[0048] In step 3, the SF6 gas leakage detection model is trained based on the infrared image and the location label of the corresponding SF6 gas leakage area.

[0049] The SF6 gas leakage detection model first uses the ResNet101 network to extract feature information from the image, uses RPN to use the feature map of the shared convolution layer to generate a series of candidate regions, and then uses RoIPooling to crop the features corresponding to each candidate region from the convolution feature map and resize it to a fixed size to be sent to the subsequent fully connected layer. Finally, the fully connected layer performs multi-task learning, and simultaneously performs object classification and precise regression of the bounding box to obtain the predicted box coordinate information of the SF6 gas leakage area.

[0050] SF6 gas leakage detection system based on multimodal image fusion, the system includes:

[0051] An image acquisition module is used to acquire a visible light image and a corresponding infrared image of the electrical equipment with SF6 gas leakage in the current environment;

[0052] An image fusion module, used for inputting the visible light image and the infrared image into an image fusion model to obtain the fused image; the image fusion model is obtained based on the sample visible light image and the sample infrared image in combination with the VGG19 network model;

[0053] The SF6 gas leakage detection module is used to input the fused image into the SF6 gas leakage detection model to obtain the predicted frame coordinate information of the actual SF6 leakage area output by the SF6 gas leakage detection model; the SF6 gas leakage detection model of the electrical equipment is trained based on the sample infrared image and the position label of the corresponding SF6 leakage area.

[0054] The present invention provides a SF6 gas leakage detection method and system based on multimodal image fusion, and the technical effects are as follows: by obtaining visible light images and infrared images of electrical equipment with SF6 gas leakage in the current environment, the infrared images and visible light images are fused through an image fusion model to obtain a high-quality fused image, and then the SF6 gas leakage detection model fully utilizes the advantages of dual-modal image data through the effective information in the fused image, and can accurately locate the SF6 gas leakage area in the GIS equipment. This method ensures that even in the case of a small leak, relatively accurate leakage point location information can be obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The present invention will be further described below in conjunction with the accompanying drawings and embodiments:

[0056] Figure 1 It is a flow chart of the method for locating and identifying gas leakage in GIS equipment of the present invention.

[0057] Figure 2 It is a schematic diagram of the network structure of the image fusion model of the present invention;

[0058] Figure 3 It is a schematic diagram of the network structure of image feature extraction provided by the present invention;

[0059] Figure 4 It is a detection flow chart of the GIS equipment gas leakage detection model provided by the present invention.

[0060] Figure 5 It is an architecture diagram of the GIS equipment gas leakage positioning and identification system provided by the present invention.

[0061] Figure 6 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0062] In order to obtain the precise location of the leakage point, the present invention adopts an image fusion model to fuse the infrared image and the visible light image, and then locates the SF6 gas leakage area in the GIS equipment. Figure 1 shown.

[0063] The SF6 gas leakage detection method based on multimodal image fusion includes the following steps:

[0064] Step 1: Create SF6 gas leakage datasets of infrared images and visible light images.

[0065] Step 1.1: Obtain visible light images and corresponding infrared images of electrical equipment with SF6 gas leakage in the current environment. In some specific embodiments, a FLIR E85 thermal imager is selected to capture infrared and visible light images simultaneously. In order to ensure that the image contains rich effective feature point information, three different relative positions of the left, middle and right should be selected within the effective field of view of the camera, and images of various angles should be recorded at different distances. The angles of the image should include vertical, tilted to the right, tilted to the left, looking up and looking down, and the tilt angle should be about 15°. In addition, data needs to be collected under different weather conditions (such as sunny days, foggy days and nights) to ensure the diversity and comprehensiveness of the data set.

[0066] Step 1.2: Use the LabelImg tool to manually label the SF6 gas leakage target to mark the SF6 gas leakage area in the infrared image and generate the corresponding label file.

[0067] Step 1.2.1: Label the infrared image. The labels are stored in a dictionary format:

[0068] {frame:Number,imgHeight:H,imgWidth:W,object:[{boundingbox:xmin,ymin,xmax,ymax,label:SF6}]}. Frame corresponds to the image frame, which is consistent with the image name and serves as the image identifier; Number represents the index value of the image frame; imgHeight and imgWidth represent the height and width of the image respectively; object contains the annotation information; boundingbox refers to making a label in the shape of a rectangle; xmin, ymin, xmax, ymax refer to the upper left corner x, y coordinates and lower right corner x, y coordinates of the annotation box respectively; label indicates the annotation category, and the annotation object is SF6.

[0069] Step 1.2.2: In order to ensure the richness of the samples, the data set is expanded by cutting, rotating and other operations on the existing images through data augmentation methods.

[0070] Step 1.3: In the dataset, infrared image public label files are randomly selected in a ratio of 8:2 to form training and test sets.

[0071] Step 2: Input the visible light image and the infrared image into the image fusion model to obtain a fused image, such as Figure 2 shown.

[0072] Step 2.1: Extract the feature map after convolution of the VGG19 network model, such as Figure 3 shown.

[0073] The first convolutional layer conv1_1 and the second convolutional layer conv1_2 of the VGG19 network model extract details and edges from the source image, and these two convolutional layers are retained;

[0074] The 3rd convolutional layer conv2_1 and the 4th convolutional layer conv2_2 of the VGG19 network model extract the edges in the image. By comparison, the 3rd convolutional layer conv2_1 is retained, and the edge extraction of this layer is more sufficient than that of the 4th convolutional layer conv2_2.

[0075] The 5th convolutional layer conv3_1, the 6th convolutional layer conv3_2, the 7th convolutional layer conv3_3, and the 8th convolutional layer conv3_4 of the VGG19 network model extract prominent targets in the image. By comparison, the 5th convolutional layer conv3_1 is retained, and the extraction effect of this layer is better than that of the other 3 layers;

[0076] The 9th convolutional layer conv4_1, the 10th convolutional layer conv4_2, the 11th convolutional layer conv4_3, and the 12th convolutional layer conv4_4 of the VGG19 network model also extract salient targets in the image. Compared with the four layers of the third convolutional layer group, the targets extracted by these four layers are more blurred, but the weights of the targets are also higher, which helps to highlight the salient targets in the infrared image in the subsequent fusion process. In contrast, the 9th convolutional layer conv4_1 is retained.

[0077] The 13th convolutional layer conv5_1, the 14th convolutional layer conv5_2, the 15th convolutional layer conv5_3, and the 16th convolutional layer conv5_4 of the VGG19 network model are no longer retained and are all removed.

[0078] In summary, the first convolutional layer conv1_1, the second convolutional layer conv1_2, the third convolutional layer conv2_1, the fifth convolutional layer conv3_1 and the ninth convolutional layer conv4_1 in VGG-19 are applied to the fusion method as the basis of transfer learning.

[0079] To facilitate subsequent calculations, the above five convolutional layers are reordered from Conv1 to Conv5. Extracting feature maps from convolutional layers represents I extracted from the kth convolutional layer n The feature map of , m is the number of channels of the kth layer.

[0080]

[0081] Where, k∈{1,2,3,4,5}, respectively represents Conv1, Conv2, Conv3, Conv4, Conv5, m∈{1,2,...,M}; Ψ k (·) represents the convolutional layer in the VGG19 network model.

[0082] Step 2.2: Use L1 norm combined with feature map to obtain the amount of source image information:

[0083] Through the L1 norm, the intuitive image is transformed into an objective data distribution, so that the amount of information contained in different images can be reflected through the data. It is obtained by the following formula:

[0084]

[0085] In the formula, It represents the measure of the amount of information in the source image, that is, the measure of the activity level of the source image. We can transform the intuitive image into an objective data distribution, thereby reflecting the amount of information contained in different images through data. is an M-dimensional vector, representing the feature map (x, y) The content of; (x, y) represents the position coordinates in the feature map.

[0086] The average operator calculation formula is used to make the fusion method robust to image mismatches, and the final activity level map is shown in the following formula:

[0087]

[0088] Where q represents the block size, q = 1; Represents the final activity level map after smoothing using the average operator. Represents the activity level at a certain offset position in the image. γ and θ are the offset values ​​in the horizontal and vertical directions respectively.

[0089] Step 2.3: Use the softmax function and upsampling to obtain the weight map.

[0090] After obtaining the final activity level map, the softmax function is used to calculate the initial weight map. The calculation formula is as follows:

[0091]

[0092] All weight map values ​​are in the range of [0, 1]. After obtaining the initial weight map, the size of the initial weight map is adjusted by upsampling to fit the size of the source image.

[0093]

[0094] In the upsampling process, the newly added positions are filled with the elements in the upper left corner, that is, when the initial weight map is enlarged, for each newly inserted position (w, v), its value is equal to the value of the original position (x, y), where w, v belongs to {0, 1, ..., 2^(a-1)-1}, and a is the number of convolutional layer groups corresponding to the weight map.

[0095] Finally, the source images are weighted and five different candidate fusion images are obtained by the following formula:

[0096]

[0097] Step 2.4: Select the maximum value of each pixel from the five candidate fusion images as the final result according to the following formula to obtain the final fusion image:

[0098] F(x,y)=max|F k (x,y)|

[0099] Step 3: Input the fused image into the SF6 gas leakage detection model to obtain the prediction box of the actual SF6 gas leakage area output by the SF6 gas leakage detection model, such as Figure 4 shown.

[0100] Step 3.1: Use the ResNet101 network to extract the feature map of the infrared image.

[0101] First, the input image is normalized to a size of 224×224×3, and then convolution is performed layer by layer to extract the final image feature map. Through the residual network, the problem of gradient disappearance is solved while maintaining a deeper network.

[0102] Specifically:

[0103] 3.1.1: First, the input image is normalized to a size of 224×224×3, and features are extracted from the image through a convolution layer with a convolution kernel size of 7×7×64, and then the maximum pooling is performed to obtain an output of 56×56×64;

[0104] 3.1.2: Start the first residual convolution stage conv1_x, which contains 3 residual blocks, each with 3 convolutional layers, and the convolution kernel sizes are 1×1×64, 3×3×64, and 1×1×256, respectively, to obtain an output of 56×56×256.

[0105] 3.1.3: Start the second residual convolution stage conv2_x, which contains 4 residual blocks, each with 3 convolutional layers, and the convolution kernel sizes are 1×1×128, 3×3×128, and 1×1×512, respectively, to obtain an output of 28×28×512.

[0106] 3.1.4: Start the third residual convolution stage conv3_x, which contains 23 residual blocks, each with 3 convolutional layers, and the convolution kernel sizes are 1×1×256, 3×3×256, and 1×1×1024, respectively, to obtain an output of 14×14×1024.

[0107] 3.1.5: Start the fourth residual convolution stage conv4_x, which contains 3 residual blocks, each with 3 convolutional layers, and the convolution kernel sizes are 1×1×512, 3×3×512, and 1×1×2048, respectively, to obtain an output of 7×7×2048.

[0108] The overall extracted network structure is shown in Table 1.

[0109] Table 1 Overall extracted network structure

[0110]

[0111] In some specific embodiments, a relu activation function is used in each convolution process, and BatchNormalization normalization is performed.

[0112] Step 3.2: Use the RPN network to generate candidate regions in the feature map.

[0113] By designing three different scale ratios, the ratio of the area of ​​the prior frame to the area of ​​the feature map [0.3, 0.5, 0.8] and the aspect ratio [1:1, 1:2, 2:1], 9 prior frames are generated at each feature point of the feature map. The above-mentioned final output feature map is further fused with features between different channels through a 3×3 convolution. Then 1×1×36 and 1×1×18 are selected for convolution respectively. The 1×1×36 convolution result is regressed to the target frame to obtain preliminary target frame information. The 1×1×18 convolution result is used to distinguish the foreground and background images using the softmax function to obtain the binary classification results of each prior frame. The prior frame identified as the foreground is regarded as a candidate, and the prior frame identified as the background is regarded as a negative sample and directly discarded. Finally, the Proposal layer is used to combine the foreground image and the target frame regression results to generate the final feature map containing ROI information, that is, the ROI information of the fused image is obtained.

[0114] Step 3.3: SF6 gas leakage area detection process:

[0115] Step 3.3.1: Use RoIPooling to crop the features corresponding to each candidate region from the convolutional feature map and resize it to a fixed size so that it can be fed into the subsequent fully connected layer.

[0116] Step 3.3.2: The feature vector dimension of the candidate box after RoI pooling is [7,7,256]. A total of 300 proposals are output, so the dimension of all candidate boxes combined is [300,7,7,256]. After the last fully connected layer (there will be a flattening operation), the dimension is [300,10]. This vector can reflect the probability of each candidate box belonging to each category. Finally, it is determined that the area contains the target object of SF6 gas.

[0117] Step 3.3.2: Perform bounding box regression on the candidate box again and adjust the bounding box of the candidate area so that it surrounds the target object more tightly. Finally, the target category and recognition box are extracted.

[0118] Based on any of the above embodiments, an embodiment of the present invention provides a SF6 gas leakage detection method based on multimodal image fusion technology. Figure 5 : is a schematic diagram of the SF6 gas leakage detection system provided by an embodiment of the present invention, such as Figure 5 As shown, the system includes:

[0119] An image acquisition module 501 is used to acquire an infrared image and a corresponding visible light image of the GIS device with SF6 leakage in the current environment;

[0120] The image fusion module 502 is used to input the infrared image and the visible light image into an image fusion model to obtain the fused image; the image fusion model is obtained based on the sample visible light image and the sample infrared image in combination with the VGG19 model.

[0121] The SF6 gas leakage detection module 503 is used to input the fused image into the SF6 gas leakage detection model to obtain the prediction box of the actual SF6 leakage area output by the SF6 gas leakage detection model; the SF6 gas leakage detection model is trained based on the infrared image and the position label of the corresponding SF6 gas leakage area.

[0122] It is understandable that the detailed functional implementation of each of the above modules can be found in the introduction of the aforementioned method embodiment, and will not be repeated here.

[0123] Based on the method in the above embodiment, an embodiment of the present invention provides an electronic device. Figure 6 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention, such as Figure 6As shown, the electronic device may include: an online monitoring IED (Intelligent Electronic Device) 601, a network switch 602, a processor 603, a memory 604, a communication bus 605 and an alarm device 606, wherein the processor 603 and the memory 604 communicate with each other through the communication bus 605. The processor 603 may call the logic instructions in the memory 604 to execute the method in the above embodiment.

[0124] Online monitoring IED (Intelligent Electronic Device) 601 is an intelligent electronic device installed in the power system for real-time monitoring of equipment status. This type of equipment is usually used in smart substations and can continuously monitor various key equipment in the power system such as transformers and switch cabinets online, thereby improving the reliability, safety and maintenance efficiency of the power system.

[0125] The network switch 602 is mainly responsible for receiving, processing and forwarding data packets in the network. The network switch performs data packet processing tasks through its built-in circuits and processors, including identifying the destination address of the data packet, finding the corresponding port information and sending the data packet to the correct destination.

[0126] Alarm device 606 is a device designed to detect abnormal conditions and send out warning signals when these conditions are detected. This device is widely used in the fields of security and monitoring, including home security, industrial production, fire safety, etc. Alarm devices can be classified according to different application scenarios and technical means.

[0127] In addition, the logic instructions contained in the memory 604 can be implemented in the form of software function modules, and when sold or deployed as independent products, these instructions can be stored on a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or more precisely, the part of it that improves the prior art, or the key part of the technical solution, can be presented in the form of a software product. This computer software product is stored in a storage medium and contains multiple instructions that enable a computer device (such as a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention.

Claims

1. SF6 gas leakage detection method based on multimodal image fusion, characterized in that The following steps are involved: Step 1: Obtain visible light images and corresponding infrared images of electrical equipment with SF6 gas leakage in the current environment; Step 2: Input the visible light image and the infrared image into the image fusion model to obtain a fused image; Step 3: Input the fused image into the SF6 gas leakage detection model to obtain the prediction box of the actual SF6 gas leakage area output by the SF6 gas leakage detection model.

2. The SF6 gas leakage detection method based on multimodal image fusion according to claim 1 is characterized in that: In step 2, the image fusion model is obtained based on the sample visible light image and infrared image in combination with the VGG19 network model.

3. The SF6 gas leakage detection method based on multimodal image fusion according to claim 2 is characterized in that: The step 2 comprises the following steps: S2.1: The image fusion model first performs convolution on the visible light image and the infrared image respectively, and then retains the output of the five convolution layers; S2.2: After extracting the required features and targets, the activity level map is obtained by L1 norm and average operator; S2.3: Use the softmax function and upsampling operator to obtain a weight map, and perform five different weight mapping convolutions on the input visible light image and infrared image to obtain a candidate fusion image; S2.4: Finally, the maximum fusion strategy is used to generate the final fused image from the five candidate fused images to complete the image reconstruction.

4. The SF6 gas leakage detection method based on multimodal image fusion according to claim 3 is characterized in that: In S2.1, the first convolution layer conv1_1 and the second convolution layer conv1_2 of the VGG19 network model extract details and edges from the source image, and these two convolution layers are retained; The 3rd convolutional layer conv2_1 and the 4th convolutional layer conv2_2 of the VGG19 network model extract the edges in the image. By comparison, the 3rd convolutional layer conv2_1 is retained, and the edge extraction of this layer is more sufficient than that of the 4th convolutional layer conv2_2. The 5th convolutional layer conv3_1, the 6th convolutional layer conv3_2, the 7th convolutional layer conv3_3, and the 8th convolutional layer conv3_4 of the VGG19 network model extract prominent targets in the image. By comparison, the 5th convolutional layer conv3_1 is retained, and the extraction effect of this layer is better than that of the other 3 layers; The 9th convolutional layer conv4_1, the 10th convolutional layer conv4_2, the 11th convolutional layer conv4_3, and the 12th convolutional layer conv4_4 of the VGG19 network model also extract salient objects in the image, and the 9th convolutional layer conv4_1 is retained for comparison; The 13th convolutional layer conv5_1, the 14th convolutional layer conv5_2, the 15th convolutional layer conv5_3, and the 16th convolutional layer conv5_4 of the VGG19 network model are no longer retained and are all removed; In summary, the first convolutional layer conv1_1, the second convolutional layer conv1_2, the third convolutional layer conv2_1, the fifth convolutional layer conv3_1, and the ninth convolutional layer conv4_1 in VGG-19 are applied to the fusion method as the basis of transfer learning; In order to facilitate subsequent calculations, the above five convolutional layers are reordered from Conv1 to Conv5; Extracting feature maps from convolutional layers represents I extracted from the kth convolutional layer n feature map, m is the number of channels in the kth layer; Where, k∈{1,2,3,4,5} represents Conv1, Conv2, Conv3, Conv4, Conv5, respectively, m∈{1,2,...,M}, where M of Conv1, Conv2, Conv3, Conv4, Conv5 is 64, 64, 128, 256, 512 respectively; Ψ k (·) represents the convolutional layer in the VGG19 network model.

5. The SF6 gas leakage detection method based on multimodal image fusion according to claim 3 is characterized in that: In S2.2, the intuitive image is converted into an objective data distribution through the L1 norm, so that the amount of information contained in different images is reflected through the data; the initial activity level map It is obtained from the following formula: In the formula, A measure of the amount of information in the source image, that is, a measure of the activity level of the source image; is an M-dimensional vector, representing the feature map (x, y) content; (x, y) represents the position coordinates in the feature map; The average operator calculation formula is used to make the fusion method robust to image mismatches, and the final activity level map is shown in the following formula: Where q represents the block size, q = 1; represents the final activity level map after smoothing using the average operator; It represents the activity level value at a certain offset position in the image; γ and θ represent the offset values ​​in the horizontal and vertical directions respectively.

6. The SF6 gas leakage detection method based on multimodal image fusion according to claim 3 is characterized in that: In S2.3, after obtaining the final activity level map, the softmax function is used to calculate the initial weight map, and the calculation formula is as follows: In the formula, represents the initial weight map; Represents the activity level map; all weight map values ​​are in the range of [0,1]. After obtaining the initial weight map, the size of the initial weight map is adjusted by upsampling to adapt to the size of the source image; In the formula, Represents the final weight map after upsampling The value at position (x+w,y+v); represents the initial weight map; w represents the displacement value of the horizontal axis; v represents the displacement value of the vertical axis; The upsampling process is: each point in the preliminary weight map will be expanded into a block in the final weight map, and all element values ​​in this block are equal to the value of the point in the preliminary weight map, that is, when the preliminary weight map is enlarged, for each newly inserted (x+w,y+v) position, its value is equal to the value of the original position (x,y), where w,v belongs to {0,1,...,2^(a)-1}, a is the number of convolutional layer groups corresponding to the weight map, for Conv1 and Conv2, since they are both in the first convolutional layer group, a=1; for Conv3, since it is in the second convolutional layer group, a=2; for Conv4, since it is in the third convolutional layer group, a=3; for Conv5, since it is in the fourth convolutional layer group, a=4; Finally, the source images are weighted and five different candidate fusion images are obtained by the following formula: Where Fk(x,y) represents the candidate fusion image, represents the final weight graph, I i (x,y) represents the source infrared image and visible light image.

7. The SF6 gas leakage detection method based on multimodal image fusion according to claim 3 is characterized in that: In S2.4, finally, the maximum value of each pixel is selected from the five candidate fused images as the final result according to the following formula to obtain the final fused image: F(x,y)=max|F k (x,y) Where F(x,y) represents the final fused image.

8. The SF6 gas leakage detection method based on multimodal image fusion according to claim 1, characterized in that: In step 3, the SF6 gas leakage detection model is trained based on the infrared image and the location label of the corresponding SF6 gas leakage area.

9. The SF6 gas leakage detection method based on multimodal image fusion according to claim 8, characterized in that: In step 3, the SF6 gas leakage detection model first uses the ResNet101 network to extract feature information in the image, uses RPN to use the feature map of the shared convolution layer to generate a series of candidate regions, and then uses RoIPooling to crop the features corresponding to each candidate region from the convolution feature map and adjust it to a fixed size so as to be sent to the subsequent fully connected layer; finally, the fully connected layer performs multi-task learning, and simultaneously performs object classification and precise regression of the bounding box to obtain the predicted box coordinate information of the SF6 gas leakage area.

10. SF6 gas leakage detection system based on multimodal image fusion, characterized in that The system includes: An image acquisition module is used to acquire a visible light image and a corresponding infrared image of the electrical equipment with SF6 gas leakage in the current environment; An image fusion module, used for inputting the visible light image and the infrared image into an image fusion model to obtain the fused image; the image fusion model is obtained based on the sample visible light image and the sample infrared image in combination with the VGG19 network model; The SF6 gas leakage detection module is used to input the fused image into the SF6 gas leakage detection model to obtain the predicted frame coordinate information of the actual SF6 leakage area output by the SF6 gas leakage detection model; the SF6 gas leakage detection model of the electrical equipment is trained based on the sample infrared image and the position label of the corresponding SF6 leakage area.

Images