Multimodal fusion gas leak detection system based on super-resolution image reconstruction

Abstract

This invention discloses a multimodal fusion gas leak detection system based on super-resolution image reconstruction, belonging to the field of gas leak detection technology. It addresses the technical problem that traditional feature matching methods (such as SIFT) perform poorly in low-texture regions, easily leading to fusion misalignment. This invention employs a super-resolution pre-processing strategy (super-resolution only for infrared images) and an optical flow learning mechanism. Super-resolution matches the resolution of the infrared image to that of visible light (eliminating resolution difference error sources). RegNet_lite learns the nonlinear mapping relationship between infrared and visible light through end-to-end training. In tests on the FLIR dataset, the registration error in low-texture regions decreased from 3.2px to 0.5px (a reduction of 84%), and the target boundary alignment accuracy improved by 2.8 times.

Description

Technical Field

[0001] This invention relates to the field of gas leak detection technology, specifically a multimodal fusion gas leak detection system based on super-resolution graph reconstruction. Background Technology

[0002] Currently, visible light and infrared image (multimodal) fusion technology mainly relies on manual registration or pre-aligned image input. Traditional methods use manual feature extraction algorithms such as template matching, SIFT, and ORB to achieve image registration, and then use strategies such as weighted overlay to fuse dual-modal information.

[0003] While deep learning-based fusion schemes (such as RCA Fusion and GIFNet) can improve the fusion effect, their training and inference both require input of strictly registered image pairs, and the network itself does not have end-to-end automatic registration capabilities.

[0004] Super-resolution reconstruction and low-light enhancement are treated as independent tasks, completely decoupled from the fusion process. Super-resolution reconstruction typically employs deep learning models such as Bicubic interpolation or SRGAN, without utilizing the structural information of infrared images to aid reconstruction. Low-light enhancement, on the other hand, only applies to visible light images, using traditional algorithms or data-driven models such as histogram equalization (HE) and CLAHE to improve the visibility of dark areas. It is not optimized in conjunction with the fusion target. Therefore, the imaging of gas features in visible light cannot be revealed, making common visible light image detection algorithms unsuitable for gas detection.

[0005] In summary, the following technical defects still exist in the existing technology:

[0006] 1. Insufficient registration accuracy: Traditional feature matching methods (such as SIFT) perform poorly in low-texture areas, leading to fusion misalignment;

[0007] 2. Missing process: Conventional algorithms lack research on low-light + super-resolution + fusion algorithms, and there is no end-to-end collaboration;

[0008] 3. Poor low-light processing: Traditional methods (such as histogram equalization) are prone to noise amplification and color distortion;

[0009] 4. The detection algorithm cannot detect gas images: Gases cannot be imaged in visible light and therefore cannot be detected. Summary of the Invention

[0010] The purpose of this invention is to solve the problems mentioned above by proposing a multimodal fusion gas leak detection system based on super-resolution graph reconstruction.

[0011] The objective of this invention can be achieved through the following technical solution: a multimodal fusion gas leak detection system based on super-resolution graph reconstruction, comprising a leak detection platform, wherein the leak detection platform is communicatively connected to:

[0012] The preprocessing unit is used to perform resolution unification and enhancement processing on infrared and visible light images;

[0013] The feature extraction unit is used to extract the thermal radiation features of infrared images and the texture and color features of visible light images, and to achieve feature alignment through cross-modal feature transformation;

[0014] An automatic registration unit is used to achieve spatial alignment of infrared and visible light images based on optical flow field calculations.

[0015] A multimodal fusion unit is used to perform feature fusion on the registered bimodal image in the YUV color space;

[0016] Intelligent processing unit for low-light enhancement and parallel processing optimization of fused images;

[0017] The gas detection unit is used to identify gas leaks based on a detection model and output location, concentration, and bounding box information.

[0018] In a preferred embodiment of the present invention, the preprocessing unit performs the following steps:

[0019] Infrared images are acquired, high-resolution images and low-resolution images are distinguished according to a set resolution threshold, and low-resolution infrared images are collected as input data.

[0020] Load the pre-trained Real-ESRGAN model, input the low-resolution infrared image into the generator for super-resolution reconstruction, and output a super-resolution infrared image with enhanced details;

[0021] The bicubic interpolation algorithm was used to uniformly scale the super-resolution infrared image and the original visible light image to a resolution of 256×256.

[0022] In a preferred embodiment of the present invention, the process of the feature extraction unit is as follows:

[0023] Convolution and pooling operations are performed on infrared and visible light images respectively to extract thermal radiation features and texture and color features. Cross-modal transformation is performed on the extracted bimodal features so that the infrared features contain texture and color semantic information and the visible light features contain thermal radiation semantic information. The feature alignment effect is verified by similarity calculation.

[0024] In a preferred embodiment of the present invention, the process of the automatic registration unit is as follows:

[0025] The RegNet_lite network is trained end-to-end to receive bimodal features and calculate the optical flow field between infrared and visible light images, outputting pixel-level displacement information. Based on the optical flow field, a pixel-by-pixel warp transformation is performed on the infrared or visible light images to achieve spatial alignment of the bimodal images, and the registration accuracy is evaluated by the structural similarity index or peak signal-to-noise ratio.

[0026] In a preferred embodiment of the present invention, the process of the multimodal fusion unit is as follows: the registered infrared image and visible light image are converted from the RGB color space to the YUV color space; the dual-modal features of the YUV space are weighted and fused using the FusionNet_lite network, wherein the Y component of the infrared image reflects the thermal radiation intensity, and the Y, U, and V components of the visible light image retain texture and color information; the fused YUV image is converted back to the RGB space, and a fused image that retains both the natural color of visible light and the infrared thermal radiation features is output.

[0027] In a preferred embodiment of the present invention, the process of the intelligent processing unit is as follows:

[0028] Low-light region detection involves converting the fused image to YUV space, calculating pixel brightness values, and determining low-light regions using a brightness threshold of 0.25.

[0029] Multi-scale illumination estimation uses scale parameters [15, 80, 250] to estimate illumination components at small, medium, and large scales under Gaussian convolution;

[0030] Image enhancement involves weighted combination of multi-scale reflection components, adaptively improving brightness and contrast in low-light areas through color restoration, gain, and offset adjustments.

[0031] In a preferred embodiment of the present invention, the intelligent processing unit also implements parallel processing:

[0032] Preprocessing, feature extraction and transformation, automatic registration, multimodal fusion, and intelligent post-processing are divided into independent processes; inter-process communication is carried out through shared memory or message queues to enable the parallel execution of each module and reduce the overall processing latency.

[0033] In a preferred embodiment of the present invention, the process of the gas detection unit is as follows:

[0034] The intelligently post-processed YUV fused image is converted into an RGB space image using the following formula: R=Y+1.14V, G=Y-0.395U-0.581V, B=Y+2.032U.

[0035] The trained YOLOv13-gas model was used to detect gas leaks in RGB images, where the model enhanced gas-related feature channels through the SE attention mechanism.

[0036] Output the center pixel coordinates, concentration heatmap, and bounding box of the gas leak area, where the heatmap uses color depth to represent the concentration level.

[0037] In a preferred embodiment of the present invention, the YOLOv13-gas model is trained in the following manner:

[0038] Collect a multimodal fused image dataset containing gas leaks and label the leak locations and concentration ranges;

[0039] By setting the learning rate and batch size parameters and adjusting the model parameters through backpropagation, the model learns the characteristics of thermal radiation anomalies and the shape and location information of the leakage area.

[0040] As a preferred embodiment of the present invention, the training of the RegNet_lite network includes: collecting infrared-visible image pairs after preprocessing and feature transformation, and labeling them with pixel-level displacement information; constructing a loss function based on the difference between the predicted optical flow field and the actual displacement; optimizing the network parameters through backpropagation; and learning the nonlinear mapping relationship between the two modal images.

[0041] Compared with the prior art, the beneficial effects of the present invention are:

[0042] 1. In this invention, a super-resolution pre-processing strategy (super-resolution only for infrared images) and an optical flow learning mechanism are employed:

[0043] Super-resolution enables infrared image resolution to match visible light (eliminating resolution difference error sources). RegNet_lite learns the nonlinear mapping relationship between infrared and visible light through end-to-end training. In the FLIR dataset test, the registration error of low-texture regions was reduced from 3.2px to 0.5px (a reduction of 84%), and the target boundary alignment accuracy was improved by 2.8 times.

[0044] 2. In this invention, a fusion of illumination-sensing decision-making and adaptive Retinex enhancement is introduced:

[0045] The illumination detection module avoids ineffective enhancement of images with normal brightness;

[0046] Multi-scale Retinex extracts local / global illumination features separately, and the results are validated on the LOL dataset: the detail retention rate in dark areas is 92% (compared to only 65% ​​by traditional methods), the overexposure rate in normal areas is reduced from 35% to 3%, and the processing speed is improved by 3.7 times;

[0047] Structural optimization of resource consumption;

[0048] The drawbacks of traditional solutions: Conventional processes involve super-resolution and fusion of both bimodal images, leading to the following issues:

[0049] Memory usage: 4.2GB (1080P input);

[0050] Computational redundancy: Visible light super-resolution (420ms);

[0051] The innovative aspect of this invention, which contributes less than 15% to the fusion: establishing the optimal path of infrared single-sided super-resolution → unified resolution fusion → on-demand enhancement. Attached Figure Description

[0052] To facilitate understanding by those skilled in the art, the present invention will be further described below with reference to the accompanying drawings.

[0053] Figure 1 This is a schematic diagram of the system principle of the present invention;

[0054] Figure 2 This is a system structure diagram of the present invention. Detailed Implementation

[0055] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0056] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0057] Please see Figures 1-2 As shown, the multimodal fusion gas leak detection system based on super-resolution graph reconstruction includes a leak detection platform, wherein the leak detection platform has the following communication connections:

[0058] Preprocessing unit;

[0059] Infrared images are acquired and classified into high-resolution and low-resolution images according to a set resolution threshold. Low-resolution infrared images are collected as input data, and a pre-trained Real-ESRGAN model is loaded. Relevant parameters, such as super-resolution factor, are configured according to actual needs.

[0060] After completion, image super-resolution processing is performed: the low-resolution infrared image is input into the generator of Real-ESRGAN, and the generator performs feature extraction, upsampling and other operations on the image to output a super-resolution infrared image. At this time, the details of the image (such as subtle changes in thermal radiation related to possible gas leaks) are significantly enhanced and the resolution is greatly improved.

[0061] To further explain, the publicly available information about the Real-ESRGAN super-resolution reconstruction model is as follows:

[0062] Core modules and hierarchical structure of the model:

[0063] The generator module adopts a variant of the U-Net architecture and contains 16 residual dense blocks (RRDB). Each RRDB consists of 3 dense convolutional layers and residual connections. The kernel size of the dense convolutional layers is 3×3, and the number of channels gradually increases from 64 to 256. The dense connections enable full reuse of features. The generator's upsampling module uses the PixelShuffle operation to compress the number of feature map channels to 1 / 4 of the original number of channels, while increasing the spatial resolution by 2 times. The output is 4 times super-resolution after two upsampling operations.

[0064] Discriminator module: It adopts the PatchGAN structure, which contains 7 convolutional layers. The kernel size of each convolutional layer is 3×3. The stride values ​​are 2, 2, 2, 2, 1, 1, 1 respectively. The number of channels increases from 64 to 512. Finally, the true and false prediction results at the patch level are output through the Sigmoid activation function.

[0065] Feature extractor: The first 10 layers of a pre-trained VGG19 network are used to extract perceptual features of the image, providing a basis for perceptual loss calculation.

[0066] Key training steps and parameters:

[0067] Data preparation: 100,000 pairs of low-resolution and high-resolution infrared images were collected. The low-resolution samples were generated by bicubic downsampling of the high-resolution infrared images, with the resolution downsampled from 512×512 to 128×128. The samples cover infrared images under different scenarios (industrial plants, laboratories, outdoor environments) and different lighting conditions (strong light, weak light, nighttime).

[0068] Parameter settings: Batch size is set to 16; initial learning rate is 2e-4, using a cosine annealing learning rate scheduling strategy, with the learning rate decaying to 0.9 of the current value every 10 epochs; training iterations are 200 epochs; the optimizer is AdamW, β1=0.9, β2=0.999, and the weight decay coefficient is 1e-4.

[0069] Loss function: A combined loss function of perceptual loss, adversarial loss, and L1 loss is used, with weights of 0.2, 0.1, and 0.7, respectively. The perceptual loss is calculated based on the feature map extracted by VGG19 to calculate the MSE loss; the adversarial loss uses the cross-entropy loss of standard GANs; and the L1 loss calculates the pixel-level absolute error between the generated image and the real high-resolution image.

[0070] Training process: First, the low-resolution infrared image is input into the generator to generate a super-resolution image; the discriminator distinguishes between the generated image and the real high-resolution image; the error is calculated by combining the loss function, and the parameters of the generator and discriminator are updated using the backpropagation algorithm; the model is validated every 20 epochs, and the model with the highest PSNR on the validation set is saved as the pre-trained model.

[0071] Determine the target resolution: Based on the requirements of subsequent registration and fusion, a unified target resolution of 256×256 is determined. Scale the super-resolution infrared image: Use bicubic interpolation algorithm to scale the super-resolution reconstructed infrared image to 256×256 resolution. During the calculation process, the target pixel value is determined by using the weighted average of surrounding pixels based on the positional relationship between the original image pixels and the target image pixels, thereby ensuring image quality.

[0072] Scaling the original visible light image: Using the same bicubic interpolation algorithm, the original visible light image is also scaled to 256×256 resolution to eliminate the resolution difference between the infrared image (after super-resolution) and the visible light image, so that subsequent registration and fusion operations can be performed on the same resolution, improving the accuracy of registration and the effect of fusion.

[0073] Feature extraction unit;

[0074] The preprocessed infrared image (super-resolution, size normalization) is input into the Encoder network. The image first passes through a convolutional layer for preliminary feature extraction, resulting in a series of feature maps containing thermal radiation information. Then, it passes through a pooling layer to downsample the feature maps, obtaining more representative thermal radiation feature vectors or feature maps. These features can reflect the core information such as the distribution and intensity of thermal radiation in the infrared image, providing a foundation for subsequent cross-modal feature conversion.

[0075] The preprocessed visible light image is input into the corresponding Encoder network. Through the convolutional layer, features such as texture edges and color distribution in the visible light image are extracted to generate a texture color feature map. After being processed by the pooling layer, features that can represent the core texture and color information of the visible light image are obtained. These features will be combined with the thermal radiation features of the infrared image for subsequent cross-modal conversion.

[0076] The infrared thermal radiation features and visible light texture and color features extracted by the Encoder network are input into the trained ModelTransfer_lite module; the module converts the features of the two modes according to the learned mapping relationship.

[0077] For example, infrared features are converted into features with more texture and color semantic information, and visible light features are converted into features with more thermal radiation semantic information, making the features of the two modalities more consistent at the semantic level. By calculating the similarity between the two modal features after conversion (such as cosine similarity, Euclidean distance, etc.), the effect of feature alignment can be verified. After conversion, the similarity between the two modal features should be significantly improved, indicating that they have achieved preliminary alignment at the feature level, laying a good foundation for subsequent automatic registration and multimodal fusion.

[0078] Automatic registration unit;

[0079] The RegNet_lite network is a lightweight network architecture suitable for image registration tasks. When applied to optical flow field calculation of infrared and visible light images, its input layer is adjusted so that it can simultaneously receive infrared and visible light image features after feature extraction and transformation. The network contains multiple convolutional layers, residual blocks and other structures to learn complex nonlinear relationships between images.

[0080] To further explain, the publicly available content of the RegNet_lite network is as follows:

[0081] Core modules and hierarchical structure of the model:

[0082] The feature encoding module consists of four convolutional stages, each containing two convolutional layers, one batch normalization layer, and a ReLU activation function. The first stage has a 3×3 kernel, 32 channels, and a stride of 2. Subsequent stages double the number of channels (64, 128, 256), while maintaining the 3×3 kernel size and stride of 2. Multi-scale features are extracted through progressive downsampling.

[0083] Optical flow estimation module: It adopts a pyramid structure and contains optical flow prediction sub-networks at 5 scales. Each sub-network consists of 3 convolutional layers and 1 deconvolutional layer. The deconvolutional layer is used to upsample the low-scale optical flow map to the current scale and fuse it with the current scale features to perform more accurate optical flow prediction.

[0084] Feature fusion module: The channel attention mechanism (CA) is used to perform weighted fusion of the encoded features of infrared and visible light images, which enhances the weight of effective features and suppresses redundant information.

[0085] Key training steps and parameters:

[0086] Data preparation: 50,000 pairs of infrared-visible images were collected after preprocessing and feature transformation. The image resolution was 256×256. Pixel-level displacement information was obtained through manual annotation and high-precision registration algorithms as real optical flow labels. The samples covered various scenarios, including low-texture areas (such as flat walls and smooth floors) and complex texture areas.

[0087] Parameter settings: Batch size is set to 8; initial learning rate is 1e-4, with step learning rate decay, halving the learning rate every 50 epochs; training iterations are 150 epochs; the optimizer is SGD, with momentum parameter of 0.9 and weight decay coefficient of 5e-5.

[0088] Loss function: A combination of multi-scale photometric loss and smoothing loss is adopted. The photometric loss calculates the grayscale difference between the image after the optical flow field warp and the target image. The weights of each scale from coarse to fine are 0.3, 0.4, 0.6, 0.8 and 1.0 respectively. The smoothing loss adopts L1 regularization of the optical flow field with a weight of 0.1 to suppress abnormal abrupt changes in the optical flow field.

[0089] Training process: Input infrared and visible light image features into the feature encoding module to extract multi-scale features; predict the optical flow field at each scale step by step through the pyramid optical flow estimation module; calculate the combined loss function value and backpropagate to update the network parameters; calculate the optical flow prediction error on the test set every 30 epochs and save the optimal model;

[0090] End-to-end training data preparation: Collect a large number of preprocessed infrared-visible image pairs (super-resolution, size uniformity, feature extraction and transformation), and label these image pairs with accurate pixel-level displacement information (which can be generated by manual annotation or by using existing high-precision registration results) to construct a training dataset;

[0091] Image pairs are input into the RegNet_lite network, and the difference between the predicted optical flow field and the actual displacement information is used as the loss function (such as a combination of photometric loss and smoothing loss). The network parameters are continuously adjusted through the backpropagation algorithm, so that the network learns the nonlinear mapping relationship between infrared and visible light images, thereby enabling accurate calculation of the optical flow field between dual-modal images and capturing pixel-level displacement information.

[0092] The infrared and visible light images after feature extraction and transformation are input into the trained RegNet_lite network. The network outputs optical flow field data between the two modal images based on the learned mapping relationship. Each pixel in the optical flow field contains displacement information of that position between the two images. This information provides a basis for subsequent accurate registration.

[0093] The Warp2D module is a tool for performing pixel-level warp transformations on images based on optical flow fields. It can map pixels in the source image to the target position based on the displacement information of each pixel in the optical flow field, thereby realizing spatial transformation of the image and aligning the bimodal images in space. The optical flow field data calculated by the RegNet_lite network is input into the Warp2D module, and the module will determine the transformation method of each pixel based on these data.

[0094] The Warp2D module performs pixel-by-pixel warp transformations on infrared or visible light images (selecting the transformation object according to registration requirements) based on the optical flow field. For example, for a visible light image, it moves the image to the position corresponding to the infrared image based on the displacement of each pixel in the optical flow field.

[0095] After the transformation is completed, the bimodal images are compared and verified. The registration accuracy can be evaluated by calculating the similarity of the overlapping areas of the registered images (such as the structural similarity index SSIM, peak signal-to-noise ratio PSNR, etc.). In low-texture areas, since the RegNet_lite network can accurately capture displacement information, the transformation of the Warp2D module can also be performed accurately, so as to solve the registration problem in low-texture areas and achieve high-precision spatial alignment of bimodal images.

[0096] Multimodal fusion unit;

[0097] The YUV color space separates the luminance (Y) and chrominance (U, V) information of an image. Compared with the RGB color space, it is more suitable for image fusion operations because luminance and chrominance can be processed independently, making it easier to integrate the luminance and chrominance features of different modalities during fusion.

[0098] The precisely registered infrared and visible light images are converted from the RGB color space to the YUV color space; for the visible light image, its original Y (luminance), U (difference between the blue component and luminance), and V (difference between the red component and luminance) information are preserved.

[0099] For infrared images, YUV conversion is also performed. However, the Y component of the infrared image mainly reflects the thermal radiation intensity (i.e., thermal information related to gas distribution). Since the infrared image itself lacks color information, the U and V components can be regarded as auxiliary information or subjected to special processing (such as setting them to zero or mapping them based on the U and V components of the visible light image).

[0100] FusionNet_lite is a lightweight network specifically designed for multimodal image fusion. Its input consists of features of infrared and visible light images converted to YUV space (including infrared thermal radiation features and visible light texture and color features). The network contains a feature extraction layer, a feature fusion layer, and a reconstruction layer. The feature extraction layer further extracts the thermal radiation features of the infrared image (focusing on hot regions reflecting gas distribution) and the texture and color features of the visible light image (focusing on scene structure, texture, and natural colors). The feature fusion layer uses a weighted fusion method to assign appropriate weights to features of different modalities to balance the contributions of thermal radiation information and texture and color information. The reconstruction layer converts the fused features back to image data in YUV space. In the feature fusion layer, the thermal radiation features of the infrared image (mainly the Y component) and the texture and color features of the visible light image (Y, U, V components) are weighted and fused by learning or pre-setting weights.

[0101] Substituting the YUV fused image output by the FusionNet_lite network into the above conversion formula, a fused image in RGB space is calculated. This fused image retains both the natural colors and scene structural details of the visible light image, and also includes thermal radiation characteristics reflecting gas distribution from the infrared image, providing a direct basis for subsequent gas detection. The formula is as follows: ;

[0102] Intelligent processing unit;

[0103] The multimodal fused image is converted into a color space suitable for brightness calculation (such as YUV space, directly taking the Y component, or calculating the brightness value in RGB space).

[0104] Next, iterate through each pixel of the image and calculate its brightness value;

[0105] Then, the brightness threshold is set to 0.25, and the area where the brightness value is less than this threshold is determined to be a low-light area;

[0106] The scale parameters [15, 80, 250] are set to correspond to small scale (local), medium scale (mesoscopic), and large scale (global), respectively. For each pixel in the low-light region, a Gaussian function is used to convolve the image at the three different scales to estimate the illumination components at different scales. ;

[0107] The reflection components at three scales are weighted and combined (usually with equal weights or weights adjusted as needed) to obtain multi-scale reflection components. Then, the color balance of the image is adjusted through color restoration (CR) operation. Finally, gain and offset adjustments are made to adaptively enhance the brightness and contrast of low-light areas while preserving the details of dark areas.

[0108] Preprocessing (including infrared image super-resolution reconstruction, image size unification, etc.), feature extraction and conversion, automatic registration, multimodal fusion, intelligent post-processing, etc. are divided into independent processing modules. An independent process is created for each processing module, and a communication mechanism (such as shared memory, message queue, etc.) is set up between the processes to transfer image data.

[0109] For example, after the preprocessing process finishes processing the image, it passes the processed image data to the feature extraction and transformation process. At the same time, the preprocessing process can start processing the next set of image data, while the feature extraction and transformation process processes the currently received image. This process continues in parallel, achieving parallel execution of each step, greatly improving the overall processing speed and reducing the latency from image input to output.

[0110] Gas detection unit;

[0111] The YUV color space is mainly used in scenarios such as video encoding to separate luminance and chrominance information. However, most detection networks (such as the YOLO series) require RGB color space images as input. The RGB color space represents colors through three components: red (R), green (G), and blue (B), which is more in line with the processing habits of detection networks for image color information. The conversion process requires calculation based on the conversion formula between YUV and RGB.

[0112] The intelligent post-processed YUV fused image is obtained, where Y represents the luminance component and U and V represent the chrominance components. According to the YUV to RGB conversion formula, the formula is: R=Y+1.14V; G=Y-0.395U-0.581V; B=Y+2.032U;

[0113] The Y, U, and V values ​​of each pixel are calculated to obtain the corresponding R, G, and B values, thereby converting the YUV fused image into an RGB space image to adapt it to the input requirements of the detection network.

[0114] Collect multimodal fused images (RGB images after preprocessing) containing gas leaks, and annotate the gas leak areas in the images, including information such as leak location and concentration range, to construct a dataset for training the YOLOv13-gas model;

[0115] The labeled dataset is input into the YOLOv13-gas model. Appropriate training parameters (such as learning rate, batch size, etc.) are set, and the model parameters are continuously adjusted through the backpropagation algorithm so that the model can learn the characteristic patterns of gas leakage, including information such as abnormal thermal radiation features, the shape and location of the leakage area.

[0116] During training, the SE attention mechanism automatically learns and enhances the channel feature weights related to gas leakage. The converted RGB input image is input into the trained YOLOv13-gas model. The backbone network of the model extracts image features, the infrared feature enhancement channel and the SE attention mechanism enhance gas-related features, and then the gas leakage situation in the image is predicted through the detection head part.

[0117] Output the center pixel coordinates of the gas leak area (or the geographic coordinates in the actual scene, if there is a coordinate mapping relationship) to clearly identify the location where the leak occurred.

[0118] To further explain, the publicly available information for the YOLOv13-gas model is as follows:

[0119] Core modules and hierarchical structure of the model:

[0120] Backbone Feature Extraction Network: The C2f module is used as the core building block, containing 8 C2f modules. The number of channels gradually increases from 64 to 1024. The first 4 C2f modules are followed by the SPPF spatial pyramid pooling module, which performs multi-scale pooling (1×1, 2×2, 4×4) on the feature maps and then concatenates them to enhance the global feature extraction capability.

[0121] Neck feature fusion module: adopts PANet structure, including upsampling module (2 times 2x upsampling) and downsampling module (2 times 2x downsampling). Through cross-scale feature fusion, it combines shallow texture features with deep semantic features, with 512, 256 and 128 channels respectively.

[0122] The detection head module contains three detection branches at different scales (for large, medium, and small targets). Each branch contains two convolutional layers and one SE attention module. The SE attention module adaptively enhances the weights of the gas feature channels through a squeeze-excitation operation. The detection head outputs the target class probability, bounding box coordinate offset, and concentration prediction.

[0123] Key training steps and parameters:

[0124] Data preparation: 80,000 multimodal fused RGB images containing gas leaks were collected. The image resolution was 256×256. The annotations included the bounding box of the leak area (x1, y1, x2, y2), the leak concentration level (level 1-5), and the center pixel coordinates. The samples covered different gas types (methane, hydrogen, carbon monoxide), different leak concentrations, and different environmental scenarios.

[0125] Parameter settings: Batch size is set to 32; initial learning rate is 3e-4, cosine annealing learning rate strategy is adopted, period is 100 epochs; number of training iterations is 300 epochs; AdamW optimizer is selected, β1=0.937, β2=0.999, weight decay coefficient is 0.001.

[0126] Loss function: A combination of classification loss (FocalLoss), bounding box regression loss (CIoULoss), and concentration prediction loss (MSELoss) is used, with weights of 0.3, 0.5, and 0.2, respectively. FocalLoss is used to address the class imbalance problem; CIoULoss comprehensively considers the overlap of bounding boxes, the distance between center points, and the aspect ratio; MSELoss calculates the squared error between the predicted concentration and the true concentration.

[0127] Training process: Input the preprocessed RGB image into the backbone network to extract features; perform cross-scale feature fusion through the neck module; output the prediction results from the detection head and calculate the combined loss; update the model parameters through backpropagation; calculate the detection accuracy (AP) and concentration prediction error on the validation set every 50 epochs and save the best performing model.

[0128] Concentration heat map: The heat distribution of different colors visually reflects the level of leakage concentration. The darker the color (such as red), the higher the concentration, and the lighter the color (such as blue), the lower the concentration. This helps staff quickly determine the severity and distribution range of the leak.

[0129] Bounding box: Precisely selects the area of ​​gas leakage and clearly marks the extent of the leakage on the image, providing direct basis for subsequent leakage handling (such as determining the repair area and taking protective measures).

[0130] In use, this invention includes: preprocessing, which unifies and enhances the resolution of infrared and visible light images; feature extraction, which extracts the thermal radiation features of the infrared image and the texture and color features of the visible light image, and achieves feature alignment through cross-modal feature conversion; automatic registration, which achieves spatial alignment of the infrared and visible light images based on optical flow field calculation; multimodal fusion, which performs feature fusion on the registered dual-modal images in the YUV color space; an intelligent processing unit, which performs low-light enhancement and parallel processing optimization on the fused image; and gas detection, which identifies gas leaks based on a detection model and outputs location, concentration, and bounding box information.

[0131] The adaptation logic of the model to the gas leak detection scenario in the above scheme will be further disclosed:

[0132] The multimodal data collaboration mechanism in this application:

[0133] Infrared images focus on capturing thermal radiation anomalies caused by gas leaks. The Real-ESRGAN model addresses the low resolution of infrared images by using super-resolution reconstruction to enhance thermal radiation details, providing clear input for subsequent thermal radiation feature extraction. Visible light images provide scene texture and color information. After unifying the resolution with the super-resolution infrared images, spatial matching of dual-modal data is achieved, solving the problem of insufficient information in a single modality.

[0134] The RegNet_lite network is specifically designed for the cross-modal characteristics of infrared and visible light images. By learning the nonlinear mapping relationship between the two-modal images, it accurately calculates the optical flow field and can achieve high-precision registration even in low-texture areas (such as the surface of a pipe without obvious texture). This lays the foundation for spatial alignment for multimodal fusion and avoids misjudgment of leakage areas caused by registration misalignment.

[0135] The YOLOv13-gas model enhances the thermal radiation anomaly features and texture features related to gas leaks through the SE attention mechanism, adapting the feature representation of the gas leak area in the fused image (such as the texture difference between the area of ​​abnormal thermal radiation intensity and the surrounding environment), and can effectively distinguish gas leaks from background interference (such as shadows and stains).

[0136] Scenario-based input / output design in this application:

[0137] Input data adaptation: The preprocessing unit sets an infrared image resolution threshold based on the characteristics of the image acquisition equipment in the gas detection scenario (e.g., images below 256×256 are considered low resolution), performs super-resolution processing on low-resolution infrared images, and simultaneously scales the super-resolution infrared image and the original visible light image to 256×256 to adapt to the input requirements of subsequent models, ensuring the consistency and validity of the input data.

[0138] Output results adaptation: The YOLOv13-gas model outputs the center pixel coordinates, bounding box, and concentration heatmap of the gas leak area. The center pixel coordinates can be converted into actual geographical location information through the mapping relationship between the image coordinate system and the actual scene coordinate system; the bounding box accurately marks the leak range, making it easy for staff to locate the leak point; the concentration heatmap intuitively reflects the concentration level with color depth, providing a basis for judging the priority of leak handling, which fully matches the actual application needs of gas leak detection scenarios such as industrial production and outdoor inspection.

[0139] Thresholds, preset values, preset ranges, etc. are set for result comparison and analysis to determine whether they are good or bad. The value of these thresholds is determined by a combination of large-scale model analysis of sample data and human experience. They can also be adjusted appropriately based on seasonal or common-sense influences.

[0140] Furthermore, the settings for weighting ratios, influence factors, etc., are based on the magnitude of each parameter's influence on the results. The specific values ​​are allocated to ultimately reflect the impact on the results. The settings for input and storage are also determined by a combination of large-scale model analysis of sample data and human experience. Appropriate adjustments can also be made based on seasonal or rational influence conditions.

[0141] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to any specific implementation. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. A multimodal fusion gas leak detection system based on super-resolution graph reconstruction, comprising a leak detection platform, characterized in that, The leakage detection platform has the following communication connections: The preprocessing unit is used to perform resolution unification and enhancement processing on infrared and visible light images; the process of the preprocessing unit is as follows: Infrared images are acquired, high-resolution images and low-resolution images are distinguished according to a set resolution threshold, and low-resolution infrared images are collected as input data. Load the pre-trained Real-ESRGAN model, input the low-resolution infrared image into the generator for super-resolution reconstruction, and output a super-resolution infrared image with enhanced details; The super-resolution infrared image and the original visible light image are uniformly scaled to 256×256 resolution using a bicubic interpolation algorithm. The feature extraction unit is used to extract the thermal radiation features of infrared images and the texture and color features of visible light images, and to achieve feature alignment through cross-modal feature transformation; the process of the feature extraction unit is as follows: Convolution and pooling operations are performed on infrared and visible light images respectively to extract thermal radiation features and texture and color features. Cross-modal transformation is performed on the extracted bimodal features so that the infrared features contain texture and color semantic information and the visible light features contain thermal radiation semantic information. The feature alignment effect is verified by similarity calculation. An automatic registration unit is used to achieve spatial alignment of infrared and visible light images based on optical flow field calculations. A multimodal fusion unit is used to perform feature fusion on the registered bimodal image in the YUV color space; Intelligent processing unit for low-light enhancement and parallel processing optimization of fused images; The gas detection unit is used to identify gas leaks based on a detection model and output location, concentration, and bounding box information.

2. The multimodal fusion gas leak detection system based on super-resolution graph reconstruction according to claim 1, characterized in that, The process of the automatic registration unit is as follows: The RegNet_lite network is trained end-to-end to receive bimodal features and calculate the optical flow field between infrared and visible light images, outputting pixel-level displacement information. Based on the optical flow field, a pixel-by-pixel warp transformation is performed on the infrared or visible light images to achieve spatial alignment of the bimodal images, and the registration accuracy is evaluated by the structural similarity index or peak signal-to-noise ratio.

3. The multimodal fusion gas leak detection system based on super-resolution graph reconstruction according to claim 1, characterized in that, The process of the multimodal fusion unit is as follows: the registered infrared image and visible light image are converted from the RGB color space to the YUV color space; the dual-modal features of the YUV space are weighted and fused using the FusionNet_lite network, where the Y component of the infrared image reflects the thermal radiation intensity, and the Y, U, and V components of the visible light image retain texture and color information; the fused YUV image is converted back to the RGB space, and a fused image that retains both the natural color of visible light and the infrared thermal radiation features is output.

4. The multimodal fusion gas leak detection system based on super-resolution graph reconstruction according to claim 1, characterized in that, The process of the intelligent processing unit is as follows: Low-light region detection involves converting the fused image to YUV space, calculating pixel brightness values, and determining low-light regions using a brightness threshold of 0.

25. Multi-scale illumination estimation uses scale parameters [15, 80, 250] to estimate illumination components at small, medium, and large scales under Gaussian convolution; Image enhancement involves weighted combination of multi-scale reflection components, adaptively improving brightness and contrast in low-light areas through color restoration, gain, and offset adjustments.

5. The multimodal fusion gas leak detection system based on super-resolution graph reconstruction according to claim 1, characterized in that, The intelligent processing unit also enables parallel processing: Preprocessing, feature extraction and transformation, automatic registration, multimodal fusion, and intelligent post-processing are divided into independent processes; inter-process communication is carried out through shared memory or message queues to enable the parallel execution of each module and reduce the overall processing latency.

6. The multimodal fusion gas leak detection system based on super-resolution graph reconstruction according to claim 1, characterized in that, The process of the gas detection unit is as follows: The intelligently post-processed YUV fused image is converted into an RGB space image using the following formula: R=Y+1.14V, G=Y-0.395U-0.581V, B=Y+2.032U; The trained YOLOv13-gas model was used to detect gas leaks in RGB images, where the model enhanced gas-related feature channels through the SE attention mechanism. Output the center pixel coordinates, concentration heatmap, and bounding box of the gas leak area, where the heatmap uses color depth to represent the concentration level.

7. A multimodal fusion gas leak detection system based on super-resolution graph reconstruction according to claim 6, characterized in that, The YOLOv13-gas model is trained in the following way: Collect a multimodal fused image dataset containing gas leaks and label the leak locations and concentration ranges; By setting the learning rate and batch size parameters and adjusting the model parameters through backpropagation, the model learns the characteristics of thermal radiation anomalies and the shape and location information of the leakage area.

8. The multimodal fusion gas leak detection system based on super-resolution graph reconstruction according to claim 2, characterized in that, The training of the RegNet_lite network includes: collecting preprocessed and feature-transformed infrared-visible image pairs and labeling them with pixel-level displacement information; constructing a loss function based on the difference between the predicted optical flow field and the actual displacement; optimizing the network parameters through backpropagation; and learning the nonlinear mapping relationship between the two modal images.

Images