Combustible gas leakage detection method based on foreground region extraction and semantic segmentation

Abstract

The application provides a combustible gas leakage detection method based on foreground region extraction and semantic segmentation, and the method comprises the following steps: background modeling is performed on each frame of infrared original image to obtain a background image; difference processing is performed on each frame of infrared original image and the corresponding background image to obtain a foreground subgraph; a variance subgraph is calculated, and a foreground estimation image is obtained based on the difference result of the foreground subgraph and the variance subgraph; adjacent K frames of foreground estimation images are subjected to cumulative average processing to obtain a foreground correction image I; threshold segmentation processing is performed on the foreground correction image I to extract a foreground position image; the extracted foreground position image is input into a non-gas foreground region prediction network to obtain a non-gas foreground position image; a mask operation is performed on the foreground correction image I based on the non-gas foreground position image to obtain a foreground correction image II; and threshold segmentation processing is performed on the foreground correction image II to obtain a gas foreground region image.

Description

Technical Field

[0001] This invention relates to the field of gas detection technology, and more specifically, to a method for detecting combustible gas leaks based on foreground region extraction and semantic segmentation. Background Technology

[0002] Whether in daily life or industrial production, the leakage of flammable gases can cause major safety accidents, threatening the lives and property of the public. Existing gas leak detection methods include sensor methods, optical interferometry, negative pressure methods, gas chromatography, and acoustic detection methods. However, these methods can only detect gas leaks at short distances, making it difficult to achieve real-time visual detection over large areas. Furthermore, flammable gases are invisible under visible light and their characteristics remain weak under specific frequencies of infrared light, making this task extremely challenging. With the continuous development of superlattice infrared detector technology, infrared imaging detection technology for gas leaks has attracted attention in recent years. Its significant advantages, such as high efficiency, long distance, large coverage, and dynamic visualization, have made it an important direction for technological development, and its market share in gas detection is increasingly expanding.

[0003] Existing infrared imaging detection technologies for gas leaks mainly fall into two categories when processing infrared images: unsupervised algorithms and supervised algorithms. Unsupervised gas leak detection algorithms typically combine image enhancement algorithms with unsupervised foreground detection algorithms such as Gaussian mixtures and Vibe algorithms to process infrared images. After initially obtaining the foreground, different methods are used to remove non-gas targets from the foreground. Supervised algorithms, on the other hand, primarily rely on deep learning to process infrared images and achieve gas leak detection.

[0004] It should be noted that the current method has the following problems:

[0005] (1) Classic foreground detection algorithms and their improved algorithms model the background and foreground in the temporal domain or the neighborhood of the pixel, ignoring the spatial correlation of each pixel in the gas region in the infrared image. This method results in a weak detection capability for areas with slight changes in the gas leak scene.

[0006] (2) Since combustible gases do not have obvious features in mid-infrared images, and non-deep learning gas leak detection algorithms are easily interfered with by many non-target foregrounds (other moving targets, etc.), the gas leak detection effect is poor and the detection error rate is high.

[0007] (3) Since the pixel grayscale value changes caused by gas leakage in the image are very weak, the current deep learning-based gas leakage detection algorithm can only detect whether there is a gas leak or locate the gas leak, but cannot accurately segment and visualize the weakly changing areas of the gas leakage image.

[0008] In order to solve the above problems, people have been seeking an ideal technological solution. Summary of the Invention

[0009] Therefore, it is necessary to provide a combustible gas leak detection method based on foreground region extraction and semantic segmentation to address the above-mentioned technical problems. This method obtains the background image, foreground sub-image, and foreground estimation image corresponding to each frame of the original infrared image. Based on the foreground estimation images of K adjacent frames, a foreground correction image I is obtained. After threshold segmentation processing of the foreground correction image I, a non-gas foreground location image is obtained using a non-gas foreground region prediction network. Based on the non-gas foreground location image and the foreground correction image I, a foreground correction image II is obtained. Based on the threshold segmentation result of the foreground correction image II, the gas foreground region is obtained.

[0010] To achieve the above objectives, a first aspect of the present invention provides a method for detecting combustible gas leaks based on foreground region extraction and semantic segmentation, comprising:

[0011] Acquire a set of raw infrared images as the infrared images to be analyzed;

[0012] Background modeling is performed on each frame of the original infrared image to obtain the background image corresponding to each frame of the original infrared image; the difference between each frame of the original infrared image and the corresponding background image is calculated to obtain the foreground sub-image corresponding to each frame of the original infrared image; the variance sub-image corresponding to each frame of the original infrared image is calculated, and based on the difference between the foreground sub-image and the variance sub-image, the foreground estimated image corresponding to each frame of the original infrared image is obtained; the foreground estimated images of K adjacent frames are accumulated and averaged to obtain the foreground correction image I.

[0013] Threshold segmentation is performed using the grayscale histogram of the foreground correction image I to extract the foreground position image in the foreground correction image I; the extracted foreground position image is input into a pre-configured non-gas foreground region prediction network to obtain the non-gas foreground position image corresponding to the foreground position image; a masking operation is performed on the foreground correction image I based on the non-gas foreground position image to obtain the foreground correction image II; threshold segmentation is performed using the grayscale histogram of the foreground correction image II to obtain the gas foreground region in the foreground correction image II.

[0014] To achieve the above objectives, a second aspect of the present invention provides a combustible gas leak detection device based on foreground region extraction and semantic segmentation, comprising a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; the memory is used to store computer programs; and the processor is used to execute the program stored in the memory to implement the combustible gas leak detection method based on foreground region extraction and semantic segmentation as described above.

[0015] To achieve the above objectives, a third aspect of the present invention provides a readable storage medium having instructions stored thereon that, when executed by one or more processors, cause the processors to perform the combustible gas leak detection method based on foreground region extraction and semantic segmentation as described above.

[0016] To achieve the above objectives, a fourth aspect of the present invention provides a computer program product, including a computer program that, when executed by a processor, implements the above-described combustible gas leak detection method based on foreground region extraction and semantic segmentation.

[0017] The beneficial effects of this invention are as follows:

[0018] This invention proposes a combustible gas leak detection method based on foreground region extraction and semantic segmentation. The method employs cumulative averaging and variance subgraphs to denoise the foreground, obtaining a corrected foreground image I. Adaptive threshold segmentation is then performed on the corrected foreground image I using a grayscale histogram to effectively extract the foreground region containing weak gas signals. Finally, a multimodal fusion semantic segmentation network is used, along with a decision-level fusion algorithm combining mid-infrared and visible light images, to effectively filter out moving target interference, achieving robust extraction of the gas foreground region and enabling rapid and accurate combustible gas leak detection. Attached Figure Description

[0019] Figure 1 This is a flowchart illustrating the combustible gas leak detection method based on foreground region extraction and semantic segmentation of the present invention.

[0020] Figure 2 This is a schematic block diagram of the combustible gas leak detection method based on foreground region extraction and semantic segmentation of the present invention;

[0021] Figure 3 This is a schematic diagram of the method for generating the foreground correction image I of the present invention;

[0022] Figure 4 This is a schematic diagram of the method for generating a non-gas foreground position image according to the present invention;

[0023] Figure 5 This is a schematic diagram of the gas foreground region extraction method in the foreground correction image II of the present invention;

[0024] Figure 6 This is a schematic diagram comparing the performance of the present invention with some existing foreground detection algorithms;

[0025] Figure 7 This is a schematic diagram of the combustible gas leak detection device based on foreground region extraction and semantic segmentation of the present invention. Detailed Implementation

[0026] The technical solution of the present invention will be further described in detail below through specific embodiments.

[0027] To facilitate understanding, the interactive parties and / or terms and / or custom terms involved in this invention will first be explained in conjunction with the technical solution of this invention:

[0028] Infrared images to be analyzed: This refers to a set of raw infrared images, such as mid-infrared images acquired using a fixed camera with a Type II superlattice infrared detector. A set of raw infrared images consists of K frames acquired consecutively over time, where K can be 8, 10, or other values; for example, a set of 8 frames. When a new raw infrared image is acquired, the first frame is deleted and the new raw infrared image is used as the 8th frame, thus updating the infrared images to be analyzed.

[0029] Background image: Each frame of the original infrared image in the infrared image to be analyzed corresponds to a background modeling image. This invention can use existing background modeling algorithms to model the background of each frame of the original infrared image, or it can use a recursive method to model the background of each frame of the original infrared image to obtain an average background, which is used as the background image corresponding to the original infrared image.

[0030] Foreground subimage: This refers to a foreground image that may be a foreground region. This foreground image contains various noises. Each frame of the original infrared image to be analyzed corresponds to one foreground subimage, as shown in the attached image. Figure 3 As shown, subtracting the corresponding background image from a frame of original infrared image yields a foreground image, which is the foreground sub-image.

[0031] Variance subplots: Each frame of the original infrared image to be analyzed corresponds to a variance subplot, as shown in the attached figure. Figure 3 As shown; for example, the variance is calculated using the gray values ​​of 49 pixels in the square neighborhood of pixel I(i,j) to obtain a variance submap; since the area covered by the variance submap is not complete, a further motion blur operation is performed on the variance submap. By using the motion blur operator to convolve the variance submap, some noise caused by gray value changes at pixel I(i,j) can be removed, thereby effectively reducing the area not processed by the variance submap.

[0032] Foreground estimation image: Each frame of the original infrared image in the infrared image to be analyzed corresponds to one foreground estimation image, as shown in the attached image. Figure 3 As shown, subtract the variance submap corresponding to each frame of the original infrared image from the foreground submap corresponding to each frame of the original infrared image to obtain a new foreground image, which is the foreground estimation image.

[0033] Foreground Correction Image I: In real-world applications, the acquisition of each frame of an image may be affected by salt-and-pepper noise, Gaussian noise, etc. Averaging the foreground estimation images acquired within a short period in the temporal domain can effectively remove this noise and enhance the foreground region. For example, averaging eight adjacent foreground estimation images yields the denoised image, which serves as Foreground Correction Image I, as shown below:

[0034]

[0035] Among them, F i Let I represent the i-th foreground-corrected image, T. j Let (x, y) represent the j-th foreground estimation image, and (x, y) represent the coordinates of any pixel in the image.

[0036] Foreground location image: refers to the foreground region in the foreground correction image I, including all moving foreground regions, including areas with slight gas changes; this invention uses a segmentation threshold adaptively generated from a grayscale histogram to perform threshold segmentation to extract the foreground region, ensuring effective detection of areas with slight changes, including gas edges.

[0037] Non-gas foreground location image: This refers to the predicted result (non-gas foreground) obtained after processing the foreground location image using a pre-configured non-gas foreground region prediction network, as shown in the attached image. Figure 2 As shown, this invention does not use a prediction network to directly predict the gas foreground region, but instead chooses to predict the non-target foreground region first and then remove the non-gas foreground.

[0038] Non-gas foreground region prediction network: This refers to the semantic segmentation network GasLeakNet. GasLeakNet re-split and recombines the input images from different modalities from a frequency perspective, enhancing the learning of high-frequency components in the image and reducing interference between different modalities. GasLeakNet also uses a two-stream UNet as its backbone network to predict the location of non-gas leakage foregrounds and optimizes the input and output designs. The overall structure of the GasLeakNet network is as follows: Figure 4 As shown.

[0039] Foreground Correction Image II: refers to the foreground image after performing a masking operation on the foreground correction image I. After removing the non-target foreground regions in the foreground correction image I using a non-gas foreground position image, a foreground correction image II containing both gas foreground and background regions is obtained.

[0040] Combustible gas leak detection area: refers to the area captured by a Class II superlattice infrared detector using a fixed camera.

[0041] Example 1

[0042] This invention provides a specific implementation of a combustible gas leak detection method based on foreground region extraction and semantic segmentation;

[0043] As attached Figure 1 To be continued Figure 3 As shown, the combustible gas leak detection method based on foreground region extraction and semantic segmentation includes:

[0044] Acquire a set of raw infrared images as the infrared images to be analyzed;

[0045] Background modeling is performed on each frame of the original infrared image to obtain the background image corresponding to each frame of the original infrared image; the difference between each frame of the original infrared image and the corresponding background image is calculated to obtain the foreground sub-image corresponding to each frame of the original infrared image.

[0046] Calculate the variance submap corresponding to each frame of the original infrared image, and obtain the foreground estimation image corresponding to each frame of the original infrared image based on the difference between the foreground submap and the variance submap.

[0047] The foreground estimation images of adjacent K frames are cumulatively averaged to obtain the foreground correction image I;

[0048] Threshold segmentation is performed using the grayscale histogram of the foreground-corrected image I to extract the foreground position image in the foreground-corrected image I;

[0049] The extracted foreground location image is input into a pre-configured non-gas foreground region prediction network to obtain the non-gas foreground location image corresponding to the foreground location image;

[0050] Based on the non-gas foreground position image, a masking operation is performed on the foreground correction image I to obtain the foreground correction image II;

[0051] Threshold segmentation is performed using the grayscale histogram of the foreground image II to obtain the gas foreground region in the foreground image II.

[0052] It should be noted that this invention employs methods such as cumulative averaging, variance subgraph, and mean filtering to denoise the foreground, reducing the impact of random noise on the detection results, and obtaining a foreground corrected image I containing gas and other moving targets. Adaptive threshold segmentation is then performed on the foreground corrected image I using a grayscale histogram to identify all moving foregrounds, including regions with slight gas variations. Next, a semantic segmentation network (non-gas foreground region prediction network) based on multimodal fusion is used to eliminate non-gas foreground targets. Finally, the foreground region extraction and semantic segmentation results are fused, and adaptive threshold segmentation is used to extract the gas foreground region, thereby enabling rapid and accurate detection of combustible gas leaks.

[0053] It should also be noted that this invention utilizes the grayscale histogram of the foreground-corrected image I to perform dynamic threshold segmentation on the foreground-corrected image I, thereby extracting all foreground regions by utilizing the spatial correlation of foreground pixels and detecting all moving foreground regions, including regions with weak gas changes; then, the image after threshold segmentation is input into a pre-configured non-gas foreground region prediction network, aiming to perform binary classification on whether the foreground region belongs to gas through multimodal data, detect the non-gas foreground positions, and use them as the prediction results of the semantic segmentation network;

[0054] The present invention also performs a masking operation on the foreground correction image I based on the prediction results of the semantic segmentation network, thereby fusing the results of the foreground extraction algorithm and the semantic segmentation network to remove the non-gas target region in the foreground correction image I, and obtain a foreground correction image II containing only the gas foreground region and the background region; then, using the grayscale histogram of the foreground correction image II, dynamic threshold segmentation is performed on the foreground correction image II to obtain the gas foreground region.

[0055] In some embodiments, when performing background modeling on each frame of raw infrared image to obtain the background image corresponding to each frame of raw infrared image, the following is executed:

[0056] Read each frame of the original infrared image and obtain the difference numerator D corresponding to each frame of the original infrared image using the frame difference method. i (x,y); where D i (x,y)=I i (x,y)-B i-1 (x,y), I i B represents the original infrared image of the i-th frame. i-1 This represents the background image corresponding to the (i-1)th frame of the original infrared image, where (x, y) represents the coordinates of any pixel in the image, such as B. i-1 (x,y) represents the coordinates of any pixel in the background image of the (i-1)th frame;

[0057] The difference numerator D is calculated using the following formula. i The mean μ and standard deviation σ of (x,y)

[0058]

[0059]

[0060] Where μ represents the difference numerator map D of the i-th frame. i The gray mean, σ represents the difference numerator D of the i-th frame. i The grayscale standard deviation; h is the original infrared image of the i-th frame. i The height, w, is the height of the i-th frame of the original infrared image I. i The width of the image is (x,y), where (x,y) represents the coordinates of any pixel in the image.

[0061] Based on the mean μ and the standard deviation σ, the region Q to be updated is obtained using the following formula. i (x,y);

[0062]

[0063] Among them, Q i D represents the region to be updated corresponding to the i-th frame of the original infrared image. i Let μ represent the difference numerator map corresponding to the i-th frame of the original infrared image, and μ represent the difference numerator map D of the i-th frame. i The gray mean, σ represents the difference numerator D of the i-th frame. i The gray standard deviation is given by (x,y), where (x,y) represents the coordinates of any pixel in the image.

[0064] Based on the region Q to be updated i (x,y) uses a recursive method to update the background image of the corresponding original infrared image.

[0065] It should be noted that the present invention uses the algorithm steps in the table below to perform background modeling on each frame of the original infrared image:

[0066]

[0067] It should also be noted that this embodiment uses a recursive method to model the background and, in conjunction with confidence intervals, updates the region with the highest mutual information between the current frame image and the background image; specifically, the background image corresponding to the i-th frame of the original infrared image is B. i (x,y), B i (x,y)=B i-1 (x,y)+α×Q i (x,y), B i-1 Q represents the background image corresponding to the (i-1)th frame of the original infrared image. i This represents the region to be updated corresponding to the i-th frame of the original infrared image; where B0 represents the background image corresponding to the 1-th frame of the original infrared image, which is pre-configured; α represents the update coefficient, which ranges from 0 to 1, and can be 0.05 for background stability.

[0068] It should also be noted that various noises exist in the foreground subimage, thus requiring image denoising. In infrared images, areas with drastic grayscale value changes, such as leaf edges and areas transitioning from bright to dark, exhibit significant noise. Therefore, based on the correlation between noise and grayscale change areas in terms of location and intensity, this embodiment uses the grayscale values ​​of pixel I(x,y) and its square neighborhood (49 pixels in total) to obtain the variance subimage. The variance subimage can be represented as:

[0069]

[0070]

[0071] Among them, S 2 The variance subplot is represented by I, which represents the original infrared image; N represents the mean subplot, which refers to the mean of all pixels in a 7*7 area centered at the pixel coordinates (x,y).

[0072] It should also be noted that, since the area covered by the variance submap is not complete, after obtaining the variance submap, this embodiment also performs a further motion blur operation on the variance submap, and combines motion blur to remove some of the noise caused by grayscale changes at I(x,y);

[0073] In some embodiments, after calculating the variance submap corresponding to each frame of the original infrared image, the following steps are performed: convolving the calculated variance submap using a motion blur operator to obtain a denoised variance submap. Here, the motion blur operator refers to a motion blur algorithm, including Gaussian blur, mean filtering, etc.

[0074] It should be noted that during gas leak detection, the grayscale values ​​of the foreground corrected image I exhibit a spatial gradient from high to low, while the background region shows a clustering of grayscale values. Therefore, this invention proposes a method for adaptively generating a segmentation threshold using the grayscale histogram by analyzing the characteristics of the grayscale histogram of the foreground corrected image I. The adaptively generated segmentation threshold is then used to perform threshold segmentation on the foreground corrected image I, which can effectively reduce the computational load while improving the integrity of gas edge segmentation.

[0075] In some embodiments, when performing threshold segmentation using the grayscale histogram of the foreground-corrected image I to extract the foreground position image in the foreground-corrected image I, the following steps are performed:

[0076] Generate the grayscale percentage histogram corresponding to the foreground correction image I. Figure I Wherein, the grayscale statistical percentage histogram Figure I The horizontal axis represents the various grayscale values ​​in the foreground correction image I, ranging from 0 to 255. The grayscale statistical percentage histogram... Figure I The vertical axis represents the percentage of pixels with the same gray value out of the total number of pixels, and the value ranges from 0 to 1;

[0077] The grayscale statistical percentage histogram Figure I Determine the dynamic grayscale threshold I required for segmentation; wherein the grayscale percentage corresponding to the dynamic grayscale threshold I is less than a preset percentage value;

[0078] Determine whether the dynamic grayscale threshold I exceeds a preset grayscale value. If it does not exceed the preset grayscale value, then perform threshold segmentation on the foreground correction image I based on the dynamic grayscale threshold I to obtain the corresponding foreground position image; wherein, the foreground position image includes gas foreground and non-gas foreground;

[0079] If the threshold is exceeded, it is determined that there is no gas foreground or non-gas foreground in the foreground correction image I, and the threshold segmentation operation of the foreground correction image I is abandoned.

[0080] It should be noted that the grayscale percentage corresponding to the dynamic grayscale threshold I refers to the grayscale percentage histogram. Figure I In this context, the percentage of pixels with the same grayscale value as the dynamic grayscale threshold I out of the total number of pixels is considered. If the dynamic grayscale threshold I exceeds the preset grayscale value, it indicates that there is no combustible gas leak in the combustible gas leak detection area; if the dynamic grayscale threshold I does not exceed the preset grayscale value, there may be a combustible gas leak in the combustible gas leak detection area.

[0081] It should also be noted that, based on the two characteristics of continuous gray-level changes in the gas region and low gray-level values ​​in the background noise region, this invention determines the dynamic gray-level threshold I required for segmentation through a gray-level statistical percentage histogram, removing gray-level portions that simultaneously have both low gray-level values ​​and high percentages. This approach retains the effectiveness of the K-means algorithm while significantly reducing the computational load.

[0082] It should also be noted that unsupervised background modeling algorithms and foreground detection algorithms mainly rely on whether pixel grayscale values ​​are stable over a long period of time to distinguish between background and foreground. Therefore, when the foreground remains stationary for a long time or some parts of the background suddenly move, unsupervised background modeling algorithms cannot correctly handle the background change problem. In addition, unsupervised foreground extraction algorithms often contain non-target foreground interference in the foreground obtained. To solve these problems, this invention proposes a non-gas foreground region prediction network to remove foreground parts that are not of interest.

[0083] In some embodiments, when the extracted foreground location image is input into a pre-configured non-gas foreground region prediction network to obtain the non-gas foreground location image corresponding to the foreground location image, the following is executed:

[0084] A non-gas foreground region prediction network is pre-configured; wherein the non-gas foreground region prediction network uses parallel UNet networks as its backbone and splits the input image at different frequencies;

[0085] Read the visible light image corresponding to the combustible gas leak detection area, and extract the RGB three-channel image corresponding to the visible light image;

[0086] The extracted RGB three-channel images, along with the original infrared image corresponding to the combustible gas leak detection area, the foreground correction image I, and the foreground position image in the foreground correction image I, are input into the non-gas foreground region prediction network to obtain the non-gas foreground position image corresponding to the foreground position image.

[0087] It should be noted that, as shown in the attached document... Figure 4 As shown, the non-gas foreground region prediction network includes a dual-stream UNet, one of which is a high-frequency branch UNet1, and the other is a full-frequency branch UNet2;

[0088] The input of the dual-stream UNet is optimized by superimposing the RGB three-channel images corresponding to the visible light image, the original infrared image, the foreground correction image I, and the foreground position image in the foreground correction image I to obtain a six-channel superimposed image. The six-channel superimposed image is then input into an average pooling layer to obtain the low-frequency components in the data. The low-frequency components are then subtracted from the original data to obtain the high-frequency components in the data. This processing method can effectively separate low-frequency and high-frequency information in the data.

[0089] After convolution, the high-frequency components in the overlay image are input into the high-frequency branch UNet1 to obtain the prediction results of the high-frequency components; after convolution, the overlay images of the six channels are input into the full-frequency branch UNet2 to obtain the prediction results of the full-frequency components; finally, the prediction results of the high-frequency components and the prediction results of the full-frequency components are overlaid and then convolved to obtain the non-gas foreground position image corresponding to the foreground position image.

[0090] It should also be noted that, in order to extract the high-frequency components from the input data, the non-gas foreground region prediction network first obtains the low-frequency components of the data through average pooling, and then subtracts the low-frequency components from the original data to obtain the high-frequency components. The non-gas foreground region prediction network splits the input image of different channels according to low and high frequencies, and then groups the high-frequency part separately to enhance the learning of high-frequency components. Compared with mixed input of data of different frequencies, this can reduce the interference between high and low frequency data.

[0091] It should also be noted that the non-gas foreground region prediction network does not directly predict the gas region, but instead chooses to predict the non-target foreground region first. The purpose of this design is to preserve the gas edge region as much as possible while ensuring the removal of interference, thereby improving the completeness of leak area detection. To achieve better coverage of the non-leaking gas foreground, the non-gas foreground target region is first expanded, and then the expanded target region is predicted by the network.

[0092] It should also be noted that the loss function of the non-gas foreground region prediction network model adopts weighted binary cross-entropy loss (BCE loss); in order to deal with the case where the distant gas leakage target is too small, the weight of position[i]=1 at the gas leakage location is increased by 50 times, as shown below:

[0093]

[0094]

[0095] Where loss(o,t) represents the difference between the predicted value and the true value; t = (x,y), where x ranges from [0,w) and y ranges from [0,h); wight[k] represents the weight coefficient at position k, which is 1 or 50; t[k] represents the true label value at position k, which ranges from 0 or 1; o[k] represents the network prediction value at position k, which is a real number R; position[k] represents the true label value equal to t[k]; and r represents the number of pixels in the image, which ranges from [0.25*h*w,h*w].

[0096] In some embodiments, when threshold segmentation is performed using the grayscale histogram of the foreground image II to obtain the gas foreground region in the foreground image II, the following is executed:

[0097] Generate the grayscale percentage histogram corresponding to the foreground correction image II. Figure II Wherein, the grayscale statistical percentage histogram Figure II The horizontal axis represents the various grayscale values ​​in the foreground correction image II, ranging from 0 to 255. The grayscale statistical percentage histogram... Figure II The vertical axis represents the percentage of pixels with the same gray value out of the total number of pixels, and the value ranges from 0 to 1;

[0098] The grayscale statistical percentage histogram Figure II Determine the dynamic grayscale threshold II required for segmentation; wherein the grayscale percentage corresponding to the dynamic grayscale threshold II is less than a preset percentage value;

[0099] Determine whether the dynamic grayscale threshold II exceeds the preset grayscale value. If it does not exceed the preset grayscale value, then perform threshold segmentation on the foreground correction image II based on the dynamic grayscale threshold II to obtain the corresponding gas foreground region image.

[0100] If the threshold is exceeded, it is determined that there is no gas foreground in the foreground correction image II, and the threshold segmentation operation of the foreground correction image II is abandoned.

[0101] Specifically, the grayscale preset value is a pre-configured parameter with a value range of 0 to 255; the percentage preset value is also a pre-configured parameter with a value range of 0 to 1; for example, if the grayscale preset value is 35, the percentage preset value is 0.05.

[0102] It should be noted that the present invention designs two threshold segmentation operations. The first threshold segmentation operation is to perform threshold segmentation on the foreground correction image I to extract the foreground position image in the foreground correction image I. The second threshold segmentation operation is to perform threshold segmentation on the foreground correction image II to obtain the gas foreground region in the foreground correction image II. Both threshold segmentation operations are performed using dynamic thresholds generated by histograms. The steps of the two threshold segmentation operations are similar and will not be described in detail here.

[0103] It should also be noted that the grayscale percentage corresponding to the dynamic grayscale threshold II refers to the grayscale percentage histogram. Figure II In the above, the percentage of pixels with the same gray value as the dynamic gray threshold II out of the total number of pixels; if the dynamic gray threshold II exceeds the preset gray value, it indicates that there is no combustible gas leak in the combustible gas leak detection area; if the dynamic gray threshold II does not exceed the preset gray value, there is a combustible gas leak in the combustible gas leak detection area.

[0104] Example 2

[0105] To visualize the results of combustible gas leak detection, this embodiment provides another specific implementation method for combustible gas leak detection based on foreground region extraction and semantic segmentation, building upon Example 1.

[0106] The combustible gas leak detection method based on foreground region extraction and semantic segmentation, after obtaining the gas foreground region, performs the following:

[0107] The gas foreground region is processed with pseudo-color and embedded with a visible light image corresponding to the combustible gas leak detection area, as shown in the attached figure. Figure 5 As shown.

[0108] It should be noted that in this embodiment, the gas foreground region is processed using a pre-configured color mapping scheme and the grayscale values ​​of the gas foreground. The pseudo-color processed image of the gas foreground region is then embedded into a visible light image. By combining the visible light image and the pseudo-color image of the gas foreground region for display, users can easily and intuitively see the location and size of the combustible gas leak. The pre-configured color mapping scheme is used to convert the grayscale values ​​in the gas foreground region into color, such as using COLORMAP_JET (a predefined pseudo-color mapping) from the OpenCV library in Python.

[0109] It should also be noted that in the foreground region extraction stage, this invention uses the spatial correlation of pixels in the foreground region to model the foreground region, thereby improving the detection effect of weak gas leakage areas. At the same time, it proposes a multimodal fusion semantic segmentation network GasLeakNet to eliminate interference problems that the foreground extraction algorithm cannot solve, and optimizes the multimodal image data from a frequency perspective. Finally, the results of the two are fused to obtain the segmentation and visualization effects.

[0110] It should also be noted that, in order to verify the effectiveness of the present invention in a gas leak scenario, this embodiment also compares and analyzes the performance of the present invention with that of some existing foreground detection algorithms. Figure 6 The paper qualitatively demonstrates a performance comparison between the foreground region detection algorithm of this invention and some existing foreground detection algorithms in a scenario of weak gas leakage. Figure 6 In the diagram, row 'a' represents the detection performance of the BSUV-Net-v2 algorithm. Figure 6 In the image, row b represents the visualization effect of the BSUV-Net-v2 algorithm based on predicted probabilities. Figure 6 In the diagram, row c represents a magnified view of the visualization effect of row b. Figure 6 The d-th row in the figure represents the gas foreground detection effect of the present invention. Figure 6 In the diagram, line 'e' represents the visualization effect of gas foreground detection in this invention. Figure 6 In the diagram, row f represents a magnified view of the visualization effect of row e. Therefore, this invention can solve the problem that existing foreground detection methods cannot accurately segment and visualize subtle changes in gas leak images, and it can also solve the problems of incomplete detection and high noise levels in subtle gas leak scenes.

[0111] It should also be noted that, as attached Figure 6 As shown, the present invention can also detect areas of slight gas changes relatively completely, and has low noise and excellent visualization effects, achieving excellent performance in combustible gas leak detection tasks.

[0112] Example 3

[0113] Based on the same inventive concept, embodiments of this application provide a computer device for implementing the above-mentioned combustible gas leak detection method based on foreground region extraction and semantic segmentation.

[0114] As attached Figure 7As shown, the combustible gas leak detection device based on foreground region extraction and semantic segmentation includes a processor, a communication interface, a memory, and a communication bus. The processor, communication interface, and memory communicate with each other through the communication bus. The memory is used to store computer programs. When the processor executes the program stored in the memory, it performs the following steps: acquiring a set of raw infrared images as infrared images to be analyzed.

[0115] Background modeling is performed on each frame of the original infrared image to obtain the background image corresponding to each frame of the original infrared image; the difference between each frame of the original infrared image and the corresponding background image is calculated to obtain the foreground sub-image corresponding to each frame of the original infrared image.

[0116] Calculate the variance submap corresponding to each frame of the original infrared image, and obtain the foreground estimation image corresponding to each frame of the original infrared image based on the difference between the foreground submap and the variance submap.

[0117] The foreground estimation images of adjacent K frames are cumulatively averaged to obtain the foreground correction image I;

[0118] Threshold segmentation is performed using the grayscale histogram of the foreground-corrected image I to extract the foreground position image in the foreground-corrected image I;

[0119] The extracted foreground location image is input into a pre-configured non-gas foreground region prediction network to obtain the non-gas foreground location image corresponding to the foreground location image;

[0120] Based on the non-gas foreground position image, a masking operation is performed on the foreground correction image I to obtain the foreground correction image II;

[0121] Threshold segmentation is performed using the grayscale histogram of the foreground image II to obtain the gas foreground region in the foreground image II.

[0122] It should be noted that the solution provided by the combustible gas leak detection device based on foreground region extraction and semantic segmentation is similar to the solution described in the above method. Therefore, the specific limitations of one or more embodiments of the combustible gas leak detection device based on foreground region extraction and semantic segmentation provided below can be found in the limitations of the detection method in the above embodiments, and will not be repeated here.

[0123] Example 4

[0124] Based on the above embodiments, this embodiment provides a specific implementation of a readable storage medium storing instructions that, when executed by one or more processors, implement the steps of the combustible gas leak detection method based on foreground region extraction and semantic segmentation:

[0125] Acquire a set of raw infrared images as the infrared images to be analyzed;

[0126] Background modeling is performed on each frame of the original infrared image to obtain the background image corresponding to each frame of the original infrared image; the difference between each frame of the original infrared image and the corresponding background image is calculated to obtain the foreground sub-image corresponding to each frame of the original infrared image.

[0127] Calculate the variance submap corresponding to each frame of the original infrared image, and obtain the foreground estimation image corresponding to each frame of the original infrared image based on the difference between the foreground submap and the variance submap.

[0128] The foreground estimation images of adjacent K frames are cumulatively averaged to obtain the foreground correction image I;

[0129] Threshold segmentation is performed using the grayscale histogram of the foreground-corrected image I to extract the foreground position image in the foreground-corrected image I;

[0130] The extracted foreground location image is input into a pre-configured non-gas foreground region prediction network to obtain the non-gas foreground location image corresponding to the foreground location image;

[0131] Based on the non-gas foreground position image, a masking operation is performed on the foreground correction image I to obtain the foreground correction image II;

[0132] Threshold segmentation is performed using the grayscale histogram of the foreground image II to obtain the gas foreground region in the foreground image II.

[0133] Based on the above embodiments, this embodiment provides a specific implementation of a computer program product, which includes a computer program that, when executed by a processor, implements the above-described combustible gas leak detection method based on foreground region extraction and semantic segmentation.

[0134] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0135] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0136] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

[0137] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them; although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications can still be made to the specific implementation of the present invention or equivalent substitutions can be made to some technical features without departing from the spirit of the technical solutions of the present invention, and all such modifications and substitutions should be covered within the scope of the technical solutions claimed in the present invention.

Claims

1. A method for detecting combustible gas leaks based on foreground region extraction and semantic segmentation, characterized in that, include: Acquire a set of raw infrared images as the infrared images to be analyzed; Perform background modeling on each frame of the original infrared image to obtain the background image corresponding to each frame of the original infrared image; The difference between each frame of the original infrared image and the corresponding background image is calculated to obtain the foreground sub-image corresponding to each frame of the original infrared image. Calculate the variance submap corresponding to each frame of the original infrared image, and obtain the foreground estimation image corresponding to each frame of the original infrared image based on the difference between the foreground submap and the variance submap. The foreground estimation images of adjacent K frames are cumulatively averaged to obtain the foreground correction image I; Threshold segmentation is performed using the grayscale histogram of the foreground-corrected image I to extract the foreground position image in the foreground-corrected image I; The extracted foreground location image is input into a pre-configured non-gas foreground region prediction network to obtain the non-gas foreground location image corresponding to the foreground location image; Based on the non-gas foreground position image, a masking operation is performed on the foreground correction image I to obtain the foreground correction image II; Threshold segmentation is performed using the grayscale histogram of the foreground image II to obtain the gas foreground region in the foreground image II; The non-gas foreground region prediction network includes a dual-stream UNet, one of which is a high-frequency branch UNet1, and the other is a full-frequency branch UNet2; The input of the dual-stream UNet is optimized by superimposing the RGB three-channel images corresponding to the visible light image, the original infrared image, the foreground correction image I, and the foreground position image in the foreground correction image I to obtain a six-channel superimposed image. The six-channel superimposed image is then input into an average pooling layer to obtain the low-frequency components in the data. Then, the low-frequency components are subtracted from the original data to obtain the high-frequency components in the data. After convolution, the high-frequency components in the overlay image are input into the high-frequency branch UNet1 to obtain the prediction results of the high-frequency components; after convolution, the overlay images of the six channels are input into the full-frequency branch UNet2 to obtain the prediction results of the full-frequency components; finally, the prediction results of the high-frequency components and the prediction results of the full-frequency components are overlaid and then convolved to obtain the non-gas foreground position image corresponding to the foreground position image.

2. The combustible gas leak detection method based on foreground region extraction and semantic segmentation according to claim 1, characterized in that, To obtain the background image corresponding to each frame of the raw infrared image, the following steps are performed: Read each frame of the original infrared image and obtain the difference numerator map corresponding to each frame of the original infrared image using the frame difference method; Calculate the mean m and standard deviation s of each difference subplot, and obtain the region to be updated based on the mean m and standard deviation s; Based on the area to be updated, the background image of the corresponding original infrared image is updated using a recursive method.

3. The combustible gas leak detection method based on foreground region extraction and semantic segmentation according to claim 2, characterized in that, After calculating the variance subplot corresponding to each frame of the original infrared image, the following steps are also performed: The calculated variance submap is convolved using a motion blur operator to obtain a denoised variance submap.

4. The combustible gas leak detection method based on foreground region extraction and semantic segmentation according to claim 1, characterized in that, When using the grayscale histogram of the foreground-corrected image I to perform threshold segmentation and extract the foreground position image from the foreground-corrected image I, the following steps are performed: Generate a grayscale percentage histogram I corresponding to the foreground corrected image I; The dynamic grayscale threshold I required for segmentation is determined by the grayscale statistical percentage histogram I; wherein the grayscale percentage corresponding to the dynamic grayscale threshold I is less than a preset percentage value. Determine whether the dynamic grayscale threshold I exceeds a preset grayscale value. If it does not exceed the preset grayscale value, then perform threshold segmentation on the foreground correction image I based on the dynamic grayscale threshold I to obtain the corresponding foreground position image; wherein, the foreground position image includes gas foreground and non-gas foreground; If the value exceeds the limit, it is determined that there is no gas foreground or non-gas foreground in the foreground correction image I.

5. The combustible gas leak detection method based on foreground region extraction and semantic segmentation according to any one of claims 1 to 4, characterized in that, When the extracted foreground location image is input into a pre-configured non-gas foreground region prediction network to obtain the non-gas foreground location image corresponding to the foreground location image, the following steps are performed: A non-gas foreground region prediction network is pre-configured; wherein the non-gas foreground region prediction network uses parallel UNet networks as its backbone and splits the input image at different frequencies; Read the visible light image corresponding to the combustible gas leak detection area, and extract the RGB three-channel image corresponding to the visible light image; The extracted RGB three-channel images, along with the original infrared image corresponding to the combustible gas leak detection area, the foreground correction image I, and the foreground position image in the foreground correction image I, are input into the non-gas foreground region prediction network to obtain the non-gas foreground position image corresponding to the foreground position image.

6. The combustible gas leak detection method based on foreground region extraction and semantic segmentation according to claim 5, characterized in that, When using the grayscale histogram of the foreground image II for threshold segmentation to obtain the gas foreground region in the foreground image II, the following steps are performed: Generate a grayscale statistical percentage histogram II corresponding to the foreground corrected image II; The dynamic grayscale threshold II required for segmentation is determined by the grayscale statistical percentage histogram II; wherein the grayscale percentage corresponding to the dynamic grayscale threshold II is less than a preset percentage value. Determine whether the dynamic grayscale threshold II exceeds the preset grayscale value. If it does not exceed the preset grayscale value, then perform threshold segmentation on the foreground correction image II based on the dynamic grayscale threshold II to obtain the corresponding gas foreground region image. If the value exceeds the limit, it is determined that there is no gas foreground in the foreground correction image II.

7. The combustible gas leak detection method based on foreground region extraction and semantic segmentation according to claim 1, characterized in that, After obtaining the gas foreground region, perform the following: The gas foreground region is processed with pseudo-color and then embedded with a visible light image corresponding to the combustible gas leak detection area.

8. A combustible gas leak detection device based on foreground region extraction and semantic segmentation, characterized in that: It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; The processor, when executing a program stored in memory, implements the combustible gas leak detection method based on foreground region extraction and semantic segmentation as described in any one of claims 1 to 7.

9. A readable storage medium, characterized in that: It stores instructions that, when executed by one or more processors, cause the processors to perform the combustible gas leak detection method based on foreground region extraction and semantic segmentation as described in any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the combustible gas leak detection method based on foreground region extraction and semantic segmentation as described in any one of claims 1 to 7.

Images