DETR-based multispectral methane gas leakage detection method

Abstract

The invention discloses a DETR-based multispectral methane gas leakage detection method, the method can effectively detect and position methane gas leakage, a DETR framework is adopted, an end-to-end method for detecting methane leakage gas by using a hyperspectral image is designed, a spectral feature generation and query optimization module in the method improves the performance of the traditional Transform, and the performance of the hyperspectral image is improved. A spectrum sensing linear filter is used for positioning potential methane hot spots of a hyperspectral image, improved query representation enables coding to be more effective, and by strategically selecting related pixels in a spectral domain, background distribution is whitened better, and a methane gas area is amplified better. The method provided by the invention has certain significance for accurately detecting methane gas leakage.

Description

Technical Field

[0001] The invention relates to a multi-spectrum methane gas leakage detection method based on DETR, belonging to the technical field of combination of computer vision and artificial intelligence. Background Art

[0002] Methane CH 4 It is one of the main gases that cause global warming, and its greenhouse effect is more than twenty times that of carbon dioxide. At the same time, methane is a flammable and explosive gas, and its leakage is extremely dangerous. Methane is a colorless and odorless gas and is difficult to detect. Deep learning provides a new method for methane gas leak detection. Existing methods based on hyperspectral images are used to analyze methane gas leaks. These methods are very sensitive to the image background environment and land type conditions, are prone to major errors, and require long-term manual investigation by field experts, so they are not scalable. Existing methods are also unable to effectively eliminate minor influencing factors. The spectral characteristics of methane are similar to those of white roofs, asphalt, etc., and such objects can also cause false alarms.

[0003] The DETR (Detection Transformer) model is an end-to-end object detection network based on Transformer. This model abandons the traditional region proposal or anchor generation strategy and directly predicts the location and category of all objects in the input image through the self-attention mechanism. The DETR model has attracted much attention due to its simplicity and efficiency. DETR first extracts image features through CNN, and then passes the feature values ​​into the encoder-decoder module of Transformer. The output results include the category class and bounding box box, and the loss is calculated through the binary matching algorithm to optimize the network parameters.

[0004] The airborne visible near-infrared imaging spectrometer AVIRIS-NG is a new generation of airborne visible / infrared imaging spectrometer. As a cutting-edge representative of aerial remote sensing technology, it has unprecedented spectral resolution and spatial coverage, providing strong data support for earth science research. Through its 598 sensor arrays, AVIRIS-NG can record spectral information from visible light to infrared bands at an ultra-high spatial resolution of 1.5 meters per pixel. The airborne AVIRIS-NG can acquire data along the route, and the data has been orthogonally corrected. The acquired multi-band data is presented in a data cube at each moment. The dimension of the data cube is 25000×1500×432. The data acquisition process is as follows: Figure 1 As shown in the figure. As a special type of remote sensing image data, the most notable feature of hyperspectral images is spectrum fusion, which can not only provide spatial distribution information of target objects, but also record the reflection or radiation intensity of objects at different wavelengths to form continuous spectral data. Rich multispectral information has broad application prospects in geological exploration, environmental monitoring, agricultural assessment and other fields.

[0005] To solve the problem that methane gas leakage is not easy to detect and locate, based on the hyperspectral image data obtained by the DETR model and AVIRIS-NG, the present invention proposes a novel end-to-end spectral wavelength-aware Transformer network to detect and locate methane gas leakage. The method provided by this patent is of certain significance for improving methane gas leakage detection. Summary of the Invention

[0006] The purpose of the present invention is to provide a multi-spectral methane gas leakage detection method based on DETR to detect and locate methane gas leakage.

[0007] To achieve the above object, the technical solution adopted by the present invention is as follows: A multi-spectral methane gas leakage detection method based on DETR, which includes the following steps:

[0008] Step 1: Obtain hyperspectral image data, perform orthogonal correction on the hyperspectral image, and then extract the RGB channel subset and the SWIR channel subset through an RGB band-pass filter (400 - 700 nm) and a short-wave infrared SWIR band-pass filter (2000 - 2500 nm) respectively;

[0009] Step 2: Input the RGB channel subset and the SWIR channel subset into the CNN backbone network (ResNet-50) respectively to extract spatial features, and combine the features of the two parts along the channel dimension to form a fused feature; compress the fused feature through a feed-forward network (FFN) to obtain a compressed feature, and input the compressed feature into the Transformer encoder to generate an encoded feature map;

[0010] Step 3: Process all channels of the hyperspectral image using a spectral linear filter (SLF) to extract candidate regions matching the spectral absorption characteristics of methane, and generate a methane candidate feature map ;

[0011] Step 4: Process the methane candidate feature map through a query refiner (QR) module. The query refiner module uses a cross-attention mechanism to optimize the learnable query and generate a refined query feature;

[0012] Step 5: Input the encoded feature map and the refined query feature into the Transformer decoder together, and generate an output embedding through a multi-head cross-attention layer; the output embedding is respectively input into a bounding box prediction module and a mask prediction module to output the bounding box coordinates, class confidence, and heat map mask of the methane leakage area;

[0013] Step 6: Adopt a two-stage training strategy. In the first stage, use the Hungarian algorithm to match the predicted bounding boxes with the ground truth annotation boxes, and calculate the L1 loss, GIoU loss, and cross-entropy loss. In the second stage, freeze the parameters of the bounding box detection network and train the mask prediction module to optimize the mask accuracy.

[0014] Further, in step 1, hyperspectral image data is obtained through the airborne visible and near-infrared imaging spectrometer AVIRIS-NG. The hyperspectral image contains 432 channels, and the wavelength coverage range is 200 - 2500 nm.

[0015] Further, in step 1, the RGB bandpass filter generates a 3-channel output corresponding to the normal red, green, and blue wavelengths; the SWIR bandpass filter generates channels at 5 nm intervals.

[0016] Further, in step 3, the design of the spectral linear filter (SLF) includes the following steps:

[0017] (1) Based on the local context whitening method, classify and segment the ground terrain into 20 categories including vegetation, water body, bare soil, rock, mountain, city, and road.

[0018] (2) Calculate the covariance matrix of each land cover class. The specific formula is:

[0019]

[0020] where, is the number of pixels in the class, is the mean of the class, T is the matrix transpose operator; i and j are the horizontal and vertical coordinates of the k-class object respectively; x i represents reading multi-column image data; k represents the k-th class object on the ground;

[0021] (3) Perform linear projection on the hyperspectral image according to the methane spectral absorption eigenvector to suppress the ground terrain interference. The formula is:

[0022]

[0023] where, represents the pixel at the k-th class index of the input hyperspectral image, i and j represent the horizontal and vertical coordinates of the image, k represents the k-th class object on the ground, the hyperspectral data is divided into 20 categories such as vegetation, water body, bare soil, rock, mountain, city, and road, and class refers to any one of them.

[0024] Even further, the classification and segmentation of the ground terrain are achieved through the following steps:

[0025] (1) Calculate the Normalized Difference Vegetation Index (NDVI) and the Normalized Difference Water Index (NDWI), and the formulas are as follows:

[0026]

[0027]

[0028] where, represents the normalized near-infrared band near the 880 nm wavelength, represents the normalized mid-infrared band near the 1240 nm wavelength, represents the normalized red band near the 660 nm wavelength;

[0029] (2) Divide the vegetation coverage area and the open water area according to the thresholds of and , and combine with the spectral reflectance characteristics of the hyperspectral image to complete the classification and mask generation of 20 types of ground terrain.

[0030] Furthermore, in the step 4, the input of the Query Refiner (QR) module includes the randomly initialized learnable query and the methane candidate feature map. The implementation of the Query Refiner (QR) module includes:

[0031] (1) Interact 100 randomly initialized learnable queries with the methane candidate feature map through the self-attention layer;

[0032] (2) Correlate the learnable query with the methane candidate feature map through the cross-attention layer to generate the refined query feature .

[0033] Furthermore, in the step 5, the mask prediction module generates the heat map mask through the following steps:

[0034] (1) Perform multi-head attention calculation on the output embedding and the encoded feature map to generate a low-resolution heat map;

[0035] (2) Upsample the low-resolution heat map and perform threshold screening, and retain the regions with confidence higher than the preset value as the final mask output of methane leakage.

[0036] The beneficial effects of the present invention are as follows: This method can effectively detect and locate methane plumes. By using the DetectionTransformer (DETR) framework, an end-to-end method for methane leakage gas detection using hyperspectral images is proposed. The spectral feature generation and query optimization module improves the performance of the traditional Transformer. The spectral-aware linear filter is applied to locate potential methane hotspots in the hyperspectral image. The improved query representation makes the encoding more effective. By strategically selecting relevant pixels in the spectral domain, the background distribution is better whitened and the methane gas region is amplified. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 is the AVIRIS-NG data collection process.

[0038] Figure 2 is the algorithm structure diagram of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0039] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0040] As Figure 2 shown, a multi-spectral methane gas leakage detection method based on DETR mainly includes the following steps:

[0041] Step 1: The hyperspectral image with 432 channels obtained by AVIRIS-NG is used as the input, and the wavelength range is 200 - 2500 nm. In addition to having characteristics in the visible light region, methane has very weak characteristics in the wavelength range of 2100 - 2400 nm. The hyperspectral image passes through the set RGB (400 - 700 nm) and SWIR (2000 - 2500 nm) band-pass filters to obtain a subset of channels within the required wavelength range, and then is respectively fed into the CNN backbone network (ResNet-50) to extract the feature map (Feature Map), and the feature maps are merged into . The feature map compressed by the feed-forward network (FFN) is fed into the encoder, and the output value is .

[0042] Step 2: Use the spectral linear filter (SLF) to process all channels of the hyperspectral image, extract candidate regions matching the spectral absorption characteristics of methane, and generate a methane feature candidate map .

[0043] Step 3: The methane feature candidate map enters the Query Refiner (QR), and through the cross-attention mechanism, the learnable query Refinement is carried out to provide a narrow search space for the query. The QR module follows an architecture similar to that of the Transformer decoder, and the output value is :

[0044] 。

[0045] Step 4: The output value of the encoder and the output value of QR are fed into the decoder to generate the output embedding. The hyperspectral decoder follows a standard architecture, with a slight difference that there is no self-attention layer, just a stack of multi-head cross-attention layers. The output embedding value is :

[0046] 。

[0047] Step 5: are respectively fed into two feed-forward networks, and the output items with high confidence scores are used as the output bbox and class; 、 and are fed into the mask prediction module, which follows the standard segmentation head of DETR, calculates the multi-head attention scores of each embedding on , generates a low-resolution heatmap for each embedding, and the heatmap with a high confidence score is used as the final output:

[0048] 。

[0049] The specific steps of the said Step 1 are as follows:

[0050] Step 1.1: The complete hyperspectral image is represented as , where and are the height and length respectively, and is the number of channels. The image is orthorectified before processing.

[0051] Step 1.2: The RGB filter produces a 3-channel output corresponding to the normal red, green, and blue wavelengths. The SWIR produces channels at approximately 5nm intervals. The filtered outputs are and . Two traditional CNN backbones, ResNet-50, use and to generate two feature maps of size respectively. Among them, , , 。

[0052] Step 1.3: These feature maps are concatenated along the channel dimension and passed through The convolution is projected to keep the channel dimension as n. The resulting output is . Use convolution to reduce the channel dimension of , , and supplement the position information by adding fixed - position embeddings . The encoder consists of a multi - head self - attention module and a feed - forward network. The encoded feature map is :

[0053] .

[0054] The specific steps of step 2 are as follows:

[0055] Step 2.1: The Spectral Linear Filter (SLF) accepts all channels of the hyperspectral image to obtain methane candidates;

[0056] Step 2.2: Feed the methane candidate map into the feature extractor (ResNet - 50) to generate a methane candidate feature map :

[0057]

[0058] The specific steps of step 3 are as follows:

[0059] Step 3.1: The methane candidate feature map is fed into the Query Refiner QR module together with a set of 100 learnable queries . The randomly initialized queries first focus on themselves through the self - attention layer;

[0060] Step 3.2: These queries focus on the methane candidate feature map from the spectral feature generator module through the cross - attention layer , which act as key - value pairs in the attention structure.

[0061] In this method, a novel Spectral Linear Filter (SLF) is used to extract pixels from the input hyperspectral image and project them onto a methane spectral absorption feature vector of the same size, reducing ground terrain interference in the data and enhancing the representation of methane gas. The design of the SLF specifically considers the spectral absorption characteristics of methane gas and the distribution of the ground terrain, because traditional linear filtering methods are not very effective in dealing with weak methane. The traditional ground terrain noise whitening method is based on a single covariance representation ; where, is the spectrum in the presence of methane, is the methane gas absorption pattern, mainly calculating the change in background radiation caused by the absorption of methane with the mixing ratio added. and represent the covariance and mean of ground terrain pixels and sensor noise respectively, is the estimated methane content in the pixel, calculating the outer product of the pixel brightness minus the average. The covariance matrix is estimated from the data, and this method assumes that all elements have similar absorption patterns. However, due to the frequent changes in the terrain along the flight path during the acquisition process, including water bodies, bare soil, vegetation, buildings, etc., this method cannot give an accurate methane gas density. The present invention adopts a whitening method based on local context, calculates the covariance of the land cover classes through land cover classification and segmentation, thus effectively eliminating the influence of confounding factors and improving the methane detection effect in the case of low methane concentration.

[0062] This method re - establishes the weighted normal distribution representation of the spectrum for the RGB and near - infrared regions of the ground terrain, classifies and segments the ground terrain by a simple and efficient method, and quantifies the vegetation and water body characteristics by calculating the Normalized Difference Vegetation Index (NDVI) and the Normalized Difference Water Index (NDWI). NDVI quantifies vegetation by measuring the difference between near - infrared and red light, while NDWI is used to highlight the open water features in satellite images.

[0063] The calculation formulas of the Normalized Difference Vegetation Index (NDVI) and the Normalized Difference Water Index (NDWI) are respectively:

[0064] and ,

[0065] where represents the normalized near - infrared band near the 880nm wavelength, represents the normalized mid - infrared band near the 1240nm wavelength, represents the normalized red band near the 660nm wavelength. The characteristics in specific wavelength ranges are used to effectively represent different types of vegetation, water bodies, bare soil, rocks, mountains, cities, roads, etc.

[0066] There are 20 types of vegetation, water bodies, bare soil, rocks, mountains, cities, roads objects processed by this method, and each class has its own segmentation mask. When the number of pixels in a class is less than 10000, two or more adjacent classes are merged into one class. When the number of pixels in each class is large enough to ensure that the judgment of the presence of methane gas is not affected when calculating the covariance matrix. Adjacent object classes are merged into a mixed class. Since the regions are all small and the objects in the region have similar brightness and reflectivity, use the formula: , calculate the covariance matrix of class is the number of pixels in the th class, is the mean of the k-th class, T is the matrix transpose operator; i and j are the horizontal and vertical coordinates of the k-th class object respectively; x i represents reading multi-column image data; k represents the k-th class object on the ground (e.g., buildings, woods, fields, etc.);

[0067] The spectral linear filter SLF is defined as:

[0068] , , represents the pixel of the input hyperspectral image at the k-th class.

[0069] Different sensors have different physical characteristics and different noise characteristics. To suppress the possible influence of sensor noise on the acquisition of hyperspectral data. Using the depth-first search algorithm, tracking the flight route to obtain the spectral data of each sensor, allocating the boundary pixels to a single sensor, using the normalized data of 10 - 15 adjacent sensors, and calculating the covariance matrix in combination with the segmentation mask. The method is simple and direct, facilitating the accurate detection of methane gas.

[0070] In this method, the training steps of the model are divided into two stages: First, train the detection of the bounding box (box) corresponding to each methane plume; Second, freeze the bounding box detection network and only train the mask prediction module. Use a two-stage loss strategy similar to that in DETR to train the network model proposed in the present invention. The first stage is the bipartite matching between the predicted value and the ground truth in the box and mask predictions, and the Hungarian algorithm is used to find the best match between the prediction and the ground truth; The second stage is the loss calculation of the matching pairs, calculating the L1 and GIoU losses of the box and mask predictions, and the cross-entropy loss of the class prediction.

[0071] In summary, the methane leakage detection method based on DETR provided by the present invention can effectively detect the methane leakage area.

[0072] The above shows and describes the basic principles, main features and advantages of the present invention. Those of ordinary skill in the art should understand that the above embodiments do not limit the protection scope of the present invention in any form. Any technical solutions obtained by means of equivalent replacement and the like all fall within the protection scope of the present invention.

[0073] Parts not involved in the present invention are the same as the prior art or can be implemented by using the prior art.

Claims

1. A multi-spectrum methane gas leak detection method based on DETR, characterized in that: The steps include: Step 1: Obtain hyperspectral image data, perform orthogonal correction on the hyperspectral image, and then extract the RGB channel subset and SWIR channel subset respectively through the RGB bandpass filter and the SWIR bandpass filter; Step 2: Input the RGB channel subset and the SWIR channel subset into the CNN backbone network to extract spatial features, and combine the features of the two parts along the channel dimension to form a fusion feature; compress the fusion feature through the feedforward network to obtain a compressed feature, and input the compressed feature into the Transformer encoder to generate a coded feature map; Step 3: Use spectral linear filters to process all channels of the hyperspectral image, extract candidate regions that match the methane spectral absorption characteristics, and generate a methane candidate feature map ; Step 4: Process the methane candidate feature map through a query refiner module, which uses a cross-attention mechanism to optimize the learnable query and generate refined query features; Step 5: Input the encoded feature map and the refined query features into the Transformer decoder, and generate output embedding through the multi-head cross attention layer; the output embedding is respectively input into the bounding box prediction module and the mask prediction module, and the bounding box coordinates, category confidence and heat map mask of the methane leakage area are output; Step 6: A two-stage training strategy is adopted. In the first stage, the predicted bounding box is matched with the real annotation box through the Hungarian algorithm, and the L1 loss, GIoU loss and cross entropy loss are calculated. In the second stage, the bounding box detection network parameters are frozen, the mask prediction module is trained, and the mask accuracy is optimized.

2. A multi-spectrum methane gas leakage detection method based on DETR according to claim 1, characterized in that: In the step 1, the hyperspectral image data is acquired by the airborne visible near-infrared imaging spectrometer AVIRIS-NG, and the hyperspectral image contains 432 channels and covers a wavelength range of 200-2500nm.

3. The multi-spectrum methane gas leakage detection method based on DETR according to claim 1 is characterized in that: In step 1, the RGB bandpass filter produces a 3-channel output corresponding to normal red, green, and blue wavelengths; the SWIR bandpass filter produces a channel with a spacing of 5 nm.

4. The multi-spectrum methane gas leakage detection method based on DETR according to claim 1 is characterized in that: In step 3, the design of the spectral linear filter includes the following steps: (1) Classify and segment the ground terrain; (2) Calculate the covariance matrix of each land cover class. The specific formula is: ;in, It is The number of pixels in the class, It is class, T is the matrix transpose operator; i and j are the horizontal and vertical coordinates of the object of class k respectively; x i Indicates reading multiple columns of image data; k indicates the kth type of object on the ground; (3) Linearly project the hyperspectral image according to the methane spectral absorption feature vector to suppress the interference of ground terrain. The formula is: ;in, Represents the input hyperspectral image indexed in the kth category The pixel at , i and j represent the horizontal and vertical coordinates of the image, class represents the category, and k represents the kth type of object on the ground.

5. A multi-spectrum methane gas leakage detection method based on DETR according to claim 4, characterized in that: The ground terrain classification and segmentation are achieved by the following steps: (1) Calculate the normalized vegetation index and normalized water index using the following formulas: ;in, represents the normalized near-infrared band, represents the normalized mid-infrared band, represents the normalized red band; (2) Based on and The vegetation coverage area and open water area are divided by the threshold, and the spectral reflectance characteristics of the hyperspectral image are combined to complete the classification and mask generation of the ground terrain.

6. The multi-spectrum methane gas leakage detection method based on DETR according to claim 1 is characterized in that: In step 4, the input of the query refiner module includes a randomly initialized learnable query and a methane candidate feature map, and the implementation of the query refiner module includes: (1) Several randomly initialized learnable queries interact with the methane candidate feature map through a self-attention layer; (2) Learnable queries are transformed into Candidate feature map of methane Perform association to generate refined query features .

7. The multi-spectrum methane gas leakage detection method based on DETR according to claim 1 is characterized in that: In step 5, the mask prediction module generates a heat map mask by the following steps: (1) Perform multi-head attention calculation on the output embedding and the encoded feature map to generate a low-resolution heat map; (2) The low-resolution heat map is upsampled and thresholded, and the areas with confidence levels higher than the preset value are retained as the final mask output of methane leakage.

Images