A multi-modal super-resolution liquefied gas leakage detection method for heavy tank area

Abstract

This disclosure relates to the field of liquefied gas leak monitoring technology, specifically to a multimodal super-resolution liquefied gas leak detection method for heavy tank areas. The method includes: acquiring liquefied gas leak image samples and dividing them into training sets, validation sets, and test sets; using the training sets, validation sets, and test sets to train, validate, and test a pre-built detection model to obtain a target detection model; and applying the target detection model to liquefied gas leak detection in heavy tank areas. The pre-built detection model includes: a multimodal fusion layer, which extracts effective discriminative features from the liquefied gas leak image samples and fuses them to obtain a feature fusion map, outputting the feature fusion map; a backbone network, which performs super-resolution reconstruction on the low-resolution feature fusion map to obtain and output a high-resolution, high-fidelity image; and a Neck layer, which introduces temporal leak features into the high-fidelity image to improve adaptability to complex scenarios, outputting a high-fidelity image with introduced temporal leak features.

Description

Technical Field

[0001] This disclosure relates to the field of liquefied gas leak monitoring technology, and more specifically, to a multimodal super-resolution liquefied gas leak detection method for heavy tank areas. Background Technology

[0002] Liquefied petroleum gas (LPG) leak detection is crucial for the timely detection of leaks of flammable gases such as propane and butane, preventing fires and explosions and protecting lives and property. Especially in typical high-risk scenarios such as LPG storage tank areas, indoor and outdoor pipelines, heavy-duty tank areas, and heavy-duty tank processing areas, LPG leaks can easily trigger large-scale fires and explosions in confined, narrow, or complex environments, accompanied by violent shock waves and the spread of toxic fumes, posing a serious threat to personnel and facility safety. Furthermore, due to the complex pipeline layouts, numerous obstructions, and drastic changes in environmental conditions in these scenarios, leak detection faces challenges such as concealed leak points, unpredictable gas retention and diffusion, and numerous blind spots. Conventional detection methods are insufficient for rapid and accurate location and carry the risk of false alarms and missed detections.

[0003] Image-based detection methods can be divided into two types: visible light image detection methods and infrared image detection methods. Infrared technology can detect heat, thus enabling the detection of changes in thermal radiation even in the absence of visible light. This allows for the accurate detection of liquefied gas leaks even in dim light or smoky conditions. It is independent of lighting conditions, working effectively in daylight or darkness, and in various weather conditions. It provides high accuracy in locating leak sources, helping to quickly pinpoint the leak and take appropriate action.

[0004] Infrared image-based detection methods first acquire infrared thermal images of the liquefied gas leak area, then preprocess the images, perform hot spot detection and feature extraction, and finally use anomaly identification methods to determine whether a leak exists and issue an alarm in a timely manner, thereby achieving real-time monitoring and safety warnings for liquefied gas leaks. However, the infrared detection process is limited by image resolution, background interference, and algorithm accuracy, posing challenges to leak detection. To effectively compensate for the shortcomings of infrared image detection, super-resolution methods can be used to improve detection accuracy. Currently available super-resolution methods include: Traditional super-resolution methods attempt to generate high-resolution images from low-resolution images using simple interpolation or edge enhancement techniques, such as nearest neighbor, bilinear interpolation, or edge detection (but their application is often limited by loss of detail, blurring, and algorithmic simplicity). Deep learning-based super-resolution methods first collect and preprocess a large number of high-resolution images and their corresponding low-resolution versions as training data. Then, a deep convolutional neural network is used to learn the mapping relationship from low-resolution to high-resolution images, improving visual quality and detection accuracy. However, this may result in problems such as high computational cost in image generation. The main methods include: 2.1 Convolutional Neural Network (CNN) method: capable of learning complex image feature maps and generating high-quality, high-resolution images (large amounts of training data and computational resources required; artifacts or oversmoothing may occur when processing particularly complex textures or very small, low-resolution images). 2.2 Generative Adversarial Network (GAN) Method: By competitively training the generator and discriminator, more realistic and detailed high-resolution images can be generated (the quality of the generated images is highly dependent on the quality and quantity of the training data, requiring more computational resources and stability control). 2.3 The diffusion model approach utilizes probability distribution to gradually enhance the image, which helps to preserve image details and structure more naturally. It adopts probabilistic modeling instead of direct pixel mapping, which can often avoid artifacts and over-smoothing when generating high-resolution images, and can effectively solve problems such as loss of detail in infrared images of liquefied gas leaks (but it involves multiple iterations and sampling of large-scale datasets, which can easily cause text and image misalignment).

[0005] Current methods for detecting liquefied petroleum gas (LPG) leaks include gas sensor methods and image detection methods. Gas sensor methods can detect very low concentrations of gas, but each gas sensor is sensitive to specific types of gas, requiring the selection of an appropriate sensor based on specific circumstances. Furthermore, high-sensitivity and high-accuracy sensors are typically expensive, especially in applications requiring long-term stable operation and high performance, where locating leaks can be challenging. In the aforementioned complex scenarios, sensor deployment is difficult, susceptible to contamination, and maintenance costs are high, further limiting their practicality. Image detection methods, on the other hand, offer rapid location, strong adaptability, and the ability to visually display temperature distribution or anomalies on object surfaces, enabling operators to quickly identify problems. However, in high-risk scenarios, LPG leaks exhibit weak texture, low contrast, and irregular dynamic changes, and are often affected by complex background interference, leading to misidentification or missed detection by existing image detection methods. Summary of the Invention

[0006] To address the problems existing in the prior art, this disclosure proposes a multimodal super-resolution liquefied gas leak detection method for heavy tank areas, aiming to solve the problems of poor real-time performance and lack of intelligent discrimination capabilities in existing liquefied gas leak monitoring methods. The technical solution adopted in this disclosure is as follows: This disclosure provides a multimodal, super-resolution liquefied gas leak detection method for heavy tank areas, the method comprising: Multiple liquefied gas leak image samples covering various scenarios were collected, and all liquefied gas leak image samples were divided into training set, validation set and test set; wherein, the liquefied gas leak image samples include visible light images and / or thermal imaging images of liquefied gas leaks; The pre-built detection model is trained, validated, and tested using the training set, validation set, and test set to obtain the target detection model; The target detection model was applied to the detection of liquefied gas leaks in heavy tank areas.

[0007] Preferably, the pre-built detection model may include: The multimodal fusion layer (MF) takes the liquefied gas leak image sample as input, extracts effective discriminative features from the visible light image and thermal imaging image in the liquefied gas leak image sample based on a multi-channel sensing fusion mechanism, and then performs multimodal fusion on the extracted effective discriminative features to obtain a feature fusion map, which is then output. The backbone network takes the feature fusion map as input, performs super-resolution reconstruction on the low-resolution feature fusion map, and outputs a high-resolution, high-fidelity image. The Neck layer (middle-layer feature representation structure) takes the high-fidelity image as input, introduces temporal leakage features into the high-fidelity image to improve its adaptability to complex scenes, and outputs the high-fidelity image with introduced temporal leakage features.

[0008] Preferably, the pre-built detection model is obtained by improving the YOLOv5 model.

[0009] Preferably, the backbone network includes a super-resolution reconstruction module (SR), which performs super-resolution reconstruction on the low-resolution feature fusion map to obtain the high-resolution high-fidelity image, thereby extracting richer and clearer texture details from the liquefied gas leak image. The super-resolution reconstruction module includes a serially connected encoder and decoder.

[0010] Preferably, the encoder may include: The fine-grained feature extraction layer takes the feature fusion map as input, extracts and outputs the fine-grained features of the feature fusion map based on the fine-grained edge pattern (focusing on local texture and edge information); The global feature extraction layer takes the feature fusion map as input, and extracts and outputs the global context features (containing global semantic structure) of the feature fusion map based on the global context mode (containing global semantic structure). The feature fusion layer takes the upsampled fine-grained features and the global context features processed by convolutional residuals as input, fuses them to obtain a low-resolution feature map, restores the missing edges and structural details in the low-resolution feature map using pre-modeled image prior information, and outputs the low-resolution feature map. The first convolutional residual layer takes the low-resolution feature map as input and performs preliminary processing on the shallow liquefied gas leak image features of the low-resolution feature map. After highlighting the texture boundaries and local details of the low-resolution feature map, it outputs the pre-processed low-resolution feature map. The second convolutional residual layer takes the pre-processed low-resolution feature map as input, performs global processing on the deep liquefied gas leak image features in the pre-processed low-resolution feature map, highlights the global feature expression of the low-resolution feature map, and outputs the globally processed low-resolution feature map.

[0011] Preferably, the decoder may include: three cascaded deconvolutional layers, wherein the three deconvolutional layers are used to progressively downsample the input low-resolution feature map to a high-resolution space; wherein each deconvolutional layer doubles the size of the input low-resolution feature map, the first deconvolutional layer takes the globally processed low-resolution feature map as input, and the last deconvolutional layer outputs the high-fidelity image.

[0012] Preferably, applying the target detection model to liquefied gas leak detection in heavy tank areas may include: Collect real-time monitoring data from the heavy tank area; The real-time monitoring data is input into the target detection model; The target detection model outputs the detection results.

[0013] Preferably, the method further includes: executing an early warning strategy based on the output detection results of the target detection model.

[0014] This disclosure offers several advantages: It provides a multimodal super-resolution liquefied gas (LPG) leak detection method for heavy-duty tank areas, constructing a detection model framework including a multimodal fusion layer, a backbone network, and a neck layer. The multimodal fusion layer, based on a multi-channel perception fusion mechanism, extracts effective discriminative features using two spectral perception modes: visible light images and thermal imaging images. These extracted features are then fused using multimodal methods, achieving deep integration of cross-modal visual signals. The multi-channel perception fusion mechanism employs a channel attention mechanism to dynamically filter and enhance key area features highly correlated with LPG leak events, while effectively compressing redundant feature information. This improves detection efficiency and reduces overall computational load, particularly effective in complex, dynamic environments like heavy-duty tank areas, demonstrating quantifiable detection efficiency and accuracy. In high-risk LPG storage locations like heavy-duty tank areas, leak signals are easily interfered with by noise due to equipment obstruction, complex backgrounds, and dynamic changes. In the multimodal fusion layer, visual signals from cross-modal visible light images and thermal imaging images are deeply integrated, enabling more accurate extraction of detailed information from visible light and infrared images, with particular attention to the leak source area, filtering out redundant background information, thereby achieving efficient and high-precision liquefied gas leak detection.

[0015] The backbone network in this disclosure includes a super-resolution reconstruction module. This module extracts fine-grained features and global contextual features, fusing them into a low-resolution feature map. It then performs preliminary processing on shallow liquefied gas leak image features and global processing on deep liquefied gas leak image features. This achieves high-fidelity restoration of blurred regions through cross-layer feature resampling and texture self-attention mechanisms. The resulting high-fidelity image provides a high-definition visual foundation for liquefied gas leak scenarios while ensuring the resource controllability of model deployment, particularly suitable for the computational resource constraints and real-time requirements of heavy tank areas. Heavy tank areas typically possess low-resolution, low-contrast image features, and detection performance may be affected by factors such as lighting variations and complex backgrounds. Therefore, this disclosure designs a dedicated enhancement path for edge detail reconstruction, focusing particularly on restoring texture information and edge details in the leak area, while avoiding excessive redundant features that consume computational resources, ensuring efficient real-time application of the model in heavy tank areas.

[0016] In this disclosure, temporal leakage features are introduced into the high-fidelity image, which realizes the embedding of gas hydrodynamic characteristics into the Neck layer. Combined with temporal structure perception, this disclosure can accurately simulate the subtle changes in the diffusion process of liquefied gas, enhance the ability to capture dynamic changes in liquefied gas leakage, and ensure efficient response and accurate identification in complex backgrounds and dynamic environments (such as wind speed, temperature changes, etc.).

[0017] Compared with existing technologies, this disclosure has the following advantages: (1) This disclosure expresses the image content of visible light images and thermal imaging images in a multi-level linkage and enhances the complementarity between visible light images and thermal imaging images based on the channel attention mechanism. This improves the quality of information interaction, effectively constrains computing resources, and balances detection accuracy and inference efficiency.

[0018] (2) This disclosure constructs an efficient reconstruction branch for low-resolution feature maps through a super-resolution reconstruction module. This can explicitly enhance blurred regions and edge modalities in low-resolution feature maps without significantly increasing model complexity, thereby obtaining high-resolution and clear high-fidelity images. The super-resolution reconstruction module can restore texture degradation areas in liquefied gas leak scenarios caused by imaging equipment, environmental occlusion, etc., ensuring that subsequent detection stages have clear and highly discriminative input prior data.

[0019] (3) By introducing temporal leakage features into the high-fidelity image, this disclosure improves the adaptability to complex scenes and realizes the integration of the flow continuity and motion features presented in the diffusion process of liquefied gas into the mid-level feature expression structure Neck, thereby making it easier to capture the microscale gas motion trajectory and its relative relationship, enhancing the adaptability to complex environmental interference, and improving the generalization detection capability under non-static lighting and background changes. Attached Figure Description

[0020] The accompanying drawings, which form part of this application, are used to provide a further understanding of this disclosure. The illustrative embodiments of this disclosure and their descriptions are used to explain this disclosure and do not constitute an undue limitation of this disclosure.

[0021] Figure 1 The flowchart of the multimodal super-resolution liquefied gas leak detection method for heavy tank areas described in this disclosure is as follows: Figure 1 .

[0022] Figure 2 The flowchart of the multimodal super-resolution liquefied gas leak detection method for heavy tank areas described in this disclosure is as follows: Figure 2 .

[0023] Figure 3 This is an example of an image sample of a liquefied gas leak as described in this disclosure.

[0024] Figure 4 This is an architecture diagram of the pre-built detection model described in this disclosure.

[0025] Figure 5 This is an architecture diagram of the super-resolution reconstruction module described in this disclosure.

[0026] Figure 6The images show a comparison of the effects of the liquefied gas leak image samples (a, b, c, d) described in this disclosure before and after reconstruction by the super-resolution reconstruction module.

[0027] Figure 7 This is a visualization of the target detection results for the liquefied gas leak image sample described in this disclosure. Detailed Implementation

[0028] The present disclosure will now be described in detail with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in the present application can be combined with each other.

[0029] The following detailed descriptions are exemplary and intended to provide further detailed explanation of this disclosure. Unless otherwise specified, all technical terms used in this disclosure have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this disclosure.

[0030] like Figure 1 As shown, a multimodal super-resolution liquefied gas leak detection method for heavy tank areas is described, the method comprising: S100. Collect multiple liquefied gas leak image samples covering various scenarios, and divide all liquefied gas leak image samples into training set, validation set and test set; wherein, the liquefied gas leak image samples include visible light images and / or thermal imaging images of liquefied gas leaks. S200. The pre-built detection model is trained, validated, and tested using the training set, validation set, and test set to obtain the target detection model; S300. Apply the target detection model to the detection of liquefied gas leaks in the heavy tank area.

[0031] Furthermore, such as Figure 4 As shown, the pre-built detection model may include: The multimodal fusion layer (MF) takes the liquefied gas leak image sample as input, extracts effective discriminative features from the visible light image and thermal imaging image in the liquefied gas leak image sample based on a multi-channel sensing fusion mechanism, and then performs multimodal fusion on the extracted effective discriminative features to obtain a feature fusion map, which is then output. The backbone network takes the feature fusion map as input, performs super-resolution reconstruction on the low-resolution feature fusion map, and outputs a high-resolution, high-fidelity image. The Neck layer (middle-layer feature representation structure) takes the high-fidelity image as input, introduces temporal leakage features into the high-fidelity image to improve its adaptability to complex scenes, and outputs the high-fidelity image with introduced temporal leakage features.

[0032] In one feasible implementation, step S100, which involves collecting multiple liquefied gas leak image samples covering various scenarios and dividing all liquefied gas leak image samples into a training set, a validation set, and a test set, may include: S101. Collect multiple liquefied gas leak image samples covering various scenarios; S102. Label the liquefied gas leak locations in the preprocessed liquefied gas leak image samples. The label category can be uniformly defined as "smoke". The labeling tool can be the labelimg image labeling tool or an online labeling tool. S103. Adjust the image size of the labeled liquefied gas leak image samples to a resolution of 512×512 to adapt to the subsequent model training requirements. S104. Construct a dataset based on the adjusted liquefied gas leak image samples; S105. Divide the dataset into the training set, validation set, and test set. The liquefied gas leak image samples are as follows: Figure 3 As shown.

[0033] Furthermore, the pre-built detection model is preferably obtained by improving the YOLOv5 model.

[0034] In one embodiment of this invention, before training, validating, and testing the pre-built detection model using the training set, validation set, and test set in step S200 to obtain the target detection model, training parameters are set for the pre-built detection model.

[0035] Furthermore, setting training parameters for the pre-built detection model may include steps S201 to S203.

[0036] S201. Initialize the model parameters for the pre-built detection model, such as setting a constant learning rate of 1e. 5 Batch size is 32, etc. S202. Select accuracy, recall, and mean precision as evaluation metrics for the pre-built detection model to effectively and comprehensively assess the performance of the detection method.

[0037] Furthermore, the expressions for precision and recall can be as follows:

[0038] Where Precision is the accuracy rate and Recall is the recall rate; TP (True Positives): The number of features that are predicted to be positive and are actually positive. FP (False Positives): The number of features that are predicted to be positive but are actually negative. TN (True Negatives): The number of features that are predicted to be negative and are actually negative. FN(False Negatives): The number of features that are predicted as negative samples but are actually positive samples.

[0039] Furthermore, the expression for Mean Average Precision (mAP) can be as follows:

[0040] Where p represents precision, r represents recall, and N is the number of categories.

[0041] S203: Select Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM) to evaluate the image quality generated by the SR-assisted module.

[0042] Furthermore, the pre-built detection model was trained on an NVIDIA 3070 GPU with 64GB of memory.

[0043] In one feasible implementation, the multi-channel sensing fusion mechanism may specifically include: A100. Dynamically adjust the output channel dimension of the multimodal fusion layer to ensure its connection consistency and tensor alignment in subsequent networks; A200. Effective discriminative features are extracted from the visible light image and the thermal imaging image respectively. The feature fusion map is obtained by fusing the shared information and unique attributes between different effective discriminative features from multiple sources.

[0044] Further, in step A200, effective discriminative features are extracted from the visible light image and the thermal imaging image respectively. The feature fusion map is obtained by multi-source fusion of the shared information and unique attributes between the different effective discriminative features, specifically including steps A201 to A207.

[0045] A201. Normalize the visible light image and thermal imaging image within the numerical range [0, 1]; wherein, the visible light image is denoted as... Irgb The thermal imaging image is denoted as I ir The liquefied gas leak image sample is denoted as R W ×H×C ,and I rgb , I ir ∈ R W×H×C ; W , H and C These represent the width, height, and number of channels of the liquefied gas leak image sample, respectively.

[0046] A202. Downsample the liquefied gas leak image sample to obtain a visible light image sample. X rgb Thermal imaging image samples X ir The expression can be as follows: .

[0047] A203. Extract the visible light image samples based on the channel attention mechanism (Squeeze-and-Excitation, SE). X rgb Thermal imaging image samples X ir Key region features were used to obtain the visible light image feature map. f rgb and thermal imaging image feature map f ir The expression can be as follows: f rgb = SE ( X rgb ), f ir = SE ( X ir ).

[0048] A204, Feature Map of Visible Light Images f rgb and thermal imaging image feature map f ir Attention modeling of the intrinsic dependencies between multimodal features in the spatial domain can be expressed as follows: m rgb = F 1( f rgb ),m ir = F 2( f ir );in, m rgb Representative visual light image feature map f rgb Attention to modeling results m ir Representative thermal imaging image feature map f ir Attention modeling results.

[0049] A205, Feature Map Based on Visible Light Image f rgb and attention modeling results m rgb Constructing visible light image feature information f in1 Based on thermal imaging image feature maps f ir and attention modeling results m ir Constructing thermal imaging image information f in2 They are represented as follows: f in1 = m rgb f rgb , f in2 = m ir f ir .

[0050] A206. Based on the initial overlay mechanism, the feature information of visible light images... f in1 Visible light image samples X rgb and thermal imaging image information f in2 Thermal imaging image samples X ir After fusing the image features separately, a 1×4 convolution operation is applied to obtain the overall features of the visible light image. f a1 Overall features of visible light images f a2 To achieve texture detail reconstruction and detail blending, the expression can be as follows: fa1 = c 3( f in1 + X rgb ), f a2 = c 4( f in2 + X ir ); in, c 3 and c 4 represents a 1×4 convolution; f a1 The overall features of the visible light image represent the image features extracted under the visible light perception mode; f a2 The overall features of the thermal imaging image represent the image features extracted under the thermal imaging sensing mode.

[0051] A207. Overall features of the obtained visible light image f a1 Overall features of visible light images f a2 The feature fusion map is obtained by performing fusion. f It can be as follows: f=SE ( con ( f a1 ，f a2 )).

[0052] In one feasible implementation, the backbone network includes a super-resolution reconstruction module (SR), which performs super-resolution reconstruction on the low-resolution feature fusion map to obtain the high-resolution high-fidelity image, thereby extracting richer and clearer texture details from the liquefied gas leak image.

[0053] Furthermore, such as Figure 5 As shown, the super-resolution reconstruction module includes a serially connected encoder and decoder.

[0054] Furthermore, such as Figure 5 As shown, the encoder may include: The fine-grained feature extraction layer takes the feature fusion map as input, extracts and outputs the fine-grained features of the feature fusion map based on the fine-grained edge pattern (focusing on local texture and edge information); The global feature extraction layer takes the feature fusion map as input, extracts and outputs the global context features of the feature fusion map based on the global context mode (focusing on global semantic structure); The feature fusion layer takes the upsampled fine-grained features and the global context features processed by convolutional residuals as input, fuses them to obtain a low-resolution feature map, restores the missing edges and structural details in the low-resolution feature map using pre-modeled image prior information, and outputs the low-resolution feature map. The first convolutional residual layer takes the low-resolution feature map as input and performs preliminary processing on the shallow liquefied gas leak image features of the low-resolution feature map. After highlighting the texture boundaries and local details of the low-resolution feature map, it outputs the pre-processed low-resolution feature map. The second convolutional residual layer takes the pre-processed low-resolution feature map as input, performs global processing on the deep liquefied gas leak image features in the pre-processed low-resolution feature map, highlights the global feature expression of the low-resolution feature map, and outputs the globally processed low-resolution feature map.

[0055] Furthermore, such as Figure 5 As shown, the decoder may include: three cascaded deconvolutional layers, which are used to progressively downsample the input low-resolution feature map to a high-resolution space; wherein each deconvolutional layer doubles the size of the input low-resolution feature map, the first deconvolutional layer takes the globally processed low-resolution feature map as input, and the last deconvolutional layer outputs the high-fidelity image.

[0056] Because heavy tank areas typically have low-resolution, low-contrast image features, and the detection performance may be affected by factors such as changes in lighting and complex backgrounds, the fine-grained feature extraction layer, global feature extraction layer, feature fusion layer, first convolutional residual layer, and second convolutional residual layer constitute an enhancement pathway capable of reconstructing image edge details. This pathway particularly focuses on restoring texture information and edge details in leaked areas, avoiding excessive redundant features that consume computational resources, and ensuring the efficient and real-time application of the entire model in heavy tank areas.

[0057] In one feasible implementation, introducing temporal leakage features into the high-fidelity image may include the following process: B100. Extract the movement features of the leaked gas in the high-fidelity image through a background filtering mechanism; B200. The extracted leakage gas motion features are fused according to time sequence to obtain the time sequence leakage feature map, thereby enhancing the sensitivity of the model's feature representation to moving targets; B300. After downsampling and scaling the time-series leakage feature map to the same size as the high-fidelity image, the feature fusion of the time-series leakage feature map and the high-fidelity image is completed in the channel dimension through the Concat operation, and finally the high-fidelity image with the time-series leakage feature is obtained.

[0058] Because liquefied gas leaks in heavy tank areas typically involve unstable diffusion processes, they are affected by external factors such as wind speed and temperature. Therefore, the Neck layer, as a mid-level feature representation structure in the deep model, introduces temporal leakage features into the high-fidelity image. This allows for the incorporation of dynamic evolution characteristics during the liquefied gas leak diffusion process into the high-fidelity image, embedding extended physical behavior feature parameters (such as local concentration gradients and flow trends) into the high-fidelity image. Consequently, it possesses adaptive discrimination capabilities against occlusion, lighting changes, and unstructured moving backgrounds in the dynamic environment of heavy tank areas, thereby achieving robust and efficient gas leak detection.

[0059] In one feasible implementation, the overall loss of the pre-built detection model L Including target detection loss L d (Detection Loss) L d ) and super-resolution reconstruction loss L sr (SR Construction Loss, L sr Overall loss L The expression can be as follows: ;in, Loss for target detection L d The weight, Loss for super-resolution reconstruction L sr The weight.

[0060] Furthermore, the super-resolution reconstruction loss L sr The expression can be as follows: ; in, S I represents the high-resolution, high-fidelity image output by the backbone network, and I represents the feature fusion map input to the backbone network.

[0061] Furthermore, the target detection loss L d The expression can be as follows:

[0062] in, L obj Represents target loss; L loc This represents the loss in target positioning; L cls Represents the target classification loss; a 1. b 1 and c 1 represents the error weight values ​​for the corresponding target, target localization, and target classification in the target detection task; a 2. b 2 and c 2 represents the target loss. L obj Target location loss L loc And target classification loss L cls The weight value.

[0063] In one feasible implementation, S300, applying the target detection model to the liquefied gas leak detection in the heavy tank area may include: S301. Collect real-time monitoring data of the heavy tank area; S302. Input the real-time monitoring data into the target detection model; S303, The target detection model outputs the detection results.

[0064] In one feasible implementation, such as Figure 2 As shown, the method may further include: S400, executing an early warning strategy based on the output detection results of the target detection model.

[0065] To more effectively verify the effectiveness of our proposed method and object detection model, we plan to compare it with other algorithms. The selected algorithms are Faster R-CNN, COCO-Net, LMO-YOLO, and YOLO-SM. The comparison results of the detection performance parameters of our method with other algorithms on the same smoke leak dataset are shown in Table 1. In Table 1, bold text represents the best performance, and underlined text represents the second best performance. The detection performance metrics in Table 1 include: average precision (AP, expressed as a percentage), recall, floating-point operations (FLOPs, in gigabytes), number of parameters (parameters, in megabytes), and frames per second (FPS).

[0066] Table 1 - Comparison of detection performance parameters between this method and other algorithms

[0067] To verify the performance of the super-resolution reconstruction module, the effects of reconstructing liquefied gas leak image samples (a, b, c, d) before and after reconstruction by the module were compared. Figure 6 As shown in Table 2, the performance metrics of the original and reconstructed images of the liquefied gas leak image samples (a, b, c, d) are compared. Performance metrics include Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM).

[0068] Table 2 - Comparison of performance indicators of original and reconstructed images of liquefied gas leak images

[0069] Finally, the target detection results of the liquefied gas leak image samples are visualized, such as... Figure 7 As shown.

[0070] Through the above Tables 1 and 2 and Figure 6 , Figure 7 In comparison, it can be concluded that the super-resolution reconstruction module effectively reconstructs the low-resolution feature map to a high resolution. During reconstruction, it explicitly enhances blurred areas and edge modalities in the low-resolution feature map, thereby obtaining a high-resolution and clear high-fidelity image. The super-resolution reconstruction module can restore texture degradation areas in liquefied gas leak scenarios caused by imaging equipment, environmental occlusion, etc., ensuring that subsequent detection stages have clear and highly discriminative input prior data.

[0071] In summary, the multimodal super-resolution liquefied gas leak detection method for heavy tank areas provided in this disclosure expresses the image content of visible light images and thermal imaging images in a multi-level linkage manner. It also enhances the complementarity between visible light images and thermal imaging images based on a channel attention mechanism. This improves the quality of information interaction while effectively constraining computational resources, and balances detection accuracy and inference efficiency. The super-resolution reconstruction module constructs an efficient reconstruction branch for low-resolution feature maps, which can explicitly enhance blurred regions and edge modalities in low-resolution feature maps without significantly increasing model complexity, thereby obtaining high-resolution and clear high-fidelity images. The super-resolution reconstruction module can restore texture degradation areas in liquefied gas leak scenes caused by imaging equipment, environmental occlusion, etc., ensuring that subsequent detection stages have clear and highly discriminative input prior data. By introducing temporal leakage features into the high-fidelity image, the adaptability to complex scenes is improved. The flow continuity and motion features presented during the diffusion of liquefied gas are integrated into the mid-level feature representation structure (Neck), which makes it easier to capture the microscale gas motion trajectory and its relative relationships, enhances the adaptability to complex environmental interference, and improves the generalization detection capability under non-static lighting and background changes.

[0072] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this disclosure and not to limit them. Although this disclosure has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of this disclosure. Any modifications or equivalent substitutions that do not depart from the spirit and scope of this disclosure should be covered within the protection scope of the claims of this disclosure.

Claims

1. A multimodal super-resolution liquefied gas leak detection method for heavy tank areas, characterized in that, The method includes: Multiple liquefied gas leak image samples covering various scenarios were collected, and all liquefied gas leak image samples were divided into training set, validation set and test set; wherein, the liquefied gas leak image samples include visible light images and / or thermal imaging images of liquefied gas leaks; The pre-built detection model is trained, validated, and tested using the training set, validation set, and test set to obtain the target detection model; The target detection model was applied to the detection of liquefied gas leaks in the heavy tank area. The pre-built detection model includes: The multimodal fusion layer takes the liquefied gas leak image sample as input, extracts effective discrimination features from the visible light image and thermal imaging image in the liquefied gas leak image sample based on the multi-channel perception fusion mechanism, and obtains a feature fusion map by multimodal fusion of the extracted effective discrimination features, and outputs the feature fusion map. The backbone network takes the feature fusion map as input, performs super-resolution reconstruction on the low-resolution feature fusion map, and outputs a high-resolution high-fidelity image. The Neck layer takes the high-fidelity image as input, introduces temporal leakage features into the high-fidelity image to improve its adaptability to complex scenes, and outputs the high-fidelity image with introduced temporal leakage features.

2. The multimodal super-resolution liquefied gas leak detection method for heavy tank areas as described in claim 1, characterized in that, The acquisition covers multiple liquefied gas leak image samples from various scenarios. All liquefied gas leak image samples are divided into training set, validation set, and test set, including: Collect multiple liquefied gas leak image samples covering various scenarios; The locations of liquefied gas leaks in the preprocessed liquefied gas leak image samples are marked; The image size of the labeled liquefied gas leak image samples was uniformly adjusted to a resolution of 512×512; A dataset was constructed based on the adjusted liquefied gas leak image samples; The dataset is divided into the training set, validation set, and test set.

3. The multimodal super-resolution liquefied gas leak detection method for heavy tank areas as described in claim 1, characterized in that, The pre-built detection model is obtained by improving the YOLOv5 model.

4. The multimodal super-resolution liquefied gas leak detection method for heavy tank areas as described in claim 1, characterized in that, The multi-channel sensing fusion mechanism specifically includes: The output channel dimension of the multimodal fusion layer is dynamically adjusted to ensure its connection consistency and tensor alignment in subsequent networks; Effective discriminative features are extracted from the visible light image and the thermal imaging image, respectively. The feature fusion map is obtained by fusing the shared information and unique attributes of different effective discriminative features from multiple sources.

5. The multimodal super-resolution liquefied gas leak detection method for heavy tank areas as described in claim 4, characterized in that, The step of extracting effective discriminative features from the visible light image and the thermal imaging image, and obtaining the feature fusion map by fusing shared information and specific attributes among different effective discriminative features from multiple sources, specifically includes: The visible light image and thermal imaging image are normalized within the numerical range [0, 1]; wherein, the visible light image is denoted as... I rgb The thermal imaging image is denoted as I ir The liquefied gas leak image sample is denoted as R W×H×C ,and I rgb , I ir ∈ R W×H×C ; W , H and C These represent the width, height, and number of channels of the liquefied gas leak image sample, respectively. The liquefied gas leak image samples were downsampled to obtain visible light image samples. X rgb Thermal imaging image samples X ir The expression is as follows: ； Based on the channel attention mechanism, the visible light image samples are extracted. X rgb Thermal imaging image samples X ir Key region features were used to obtain the visible light image feature map. f rgb and thermal imaging image feature map f ir The expression is as follows: f rgb = SE ( X rgb )， f ir = SE ( X ir )； Feature map of visible light image f rgb and thermal imaging image feature map f ir Attention modeling is performed on the intrinsic dependencies between multimodal features in the spatial domain, which is expressed as follows: m rgb = F 1( f rgb ), m ir = F 2( f ir );in, m rgb Representative visual light image feature map f rgb Attention to modeling results m ir Representative thermal imaging image feature map f ir Attention to the modeling results; Based on visible light image feature maps f rgb and attention modeling results m rgb Constructing visible light image feature information f in1 Based on thermal imaging image feature maps f ir and attention modeling results m ir Constructing thermal imaging image information f in2 The expressions are as follows: f in1 = m rgb f rgb ， f in2 = m ir f ir ； Based on the initial overlay mechanism, the visible light image feature information f in1 Visible light image samples X rgb and thermal imaging image information f in2 Thermal imaging image samples X ir After fusing the image features separately, a 1×4 convolution operation is applied to obtain the overall features of the visible light image. f a1 Overall features of visible light images f a2 To achieve texture detail reconstruction and detail blending, the expression is as follows: f a1 = c 3( f in1 + X rgb )， f a2 = c 4( f in2 + X ir )； in, c 3 and c 4 represents a 1×4 convolution; f a1 The overall features of the visible light image represent the image features extracted under the visible light perception mode; f a2 The overall features of the thermal imaging image represent the image features extracted under the thermal imaging sensing mode; Overall features of the obtained visible light image f a1 Overall features of visible light images f a2 The feature fusion map is obtained by performing fusion. f as follows: f=SE ( con ( f a1 ，f a2 ))。 6. The multimodal super-resolution liquefied gas leak detection method for heavy tank areas as described in claim 1, characterized in that, The backbone network includes a super-resolution reconstruction module, which performs super-resolution reconstruction on the low-resolution feature fusion map to obtain the high-resolution high-fidelity image, thereby extracting richer and clearer texture details from the liquefied gas leak image.

7. The multimodal super-resolution liquefied gas leak detection method for heavy tank areas as described in claim 6, characterized in that, The super-resolution reconstruction module includes a serially connected encoder and decoder; The encoder includes: The fine-grained feature extraction layer takes the feature fusion map as input, extracts and outputs the fine-grained features of the feature fusion map based on the fine-grained edge pattern; The global feature extraction layer takes the feature fusion map as input, extracts and outputs the global context features of the feature fusion map based on the global context mode; The feature fusion layer takes the upsampled fine-grained features and the global context features processed by convolutional residuals as input, fuses them to obtain a low-resolution feature map, restores the missing edges and structural details in the low-resolution feature map using pre-modeled image prior information, and outputs the low-resolution feature map. The first convolutional residual layer takes the low-resolution feature map as input and performs preliminary processing on the shallow liquefied gas leak image features of the low-resolution feature map. After highlighting the texture boundaries and local details of the low-resolution feature map, it outputs the pre-processed low-resolution feature map. The second convolutional residual layer takes the pre-processed low-resolution feature map as input, performs global processing on the deep liquefied gas leak image features in the pre-processed low-resolution feature map, highlights the global feature expression of the low-resolution feature map, and outputs the globally processed low-resolution feature map. The decoder includes three cascaded deconvolutional layers, which progressively downsample the input low-resolution feature map to a high-resolution space. Each deconvolutional layer doubles the size of the input low-resolution feature map. The first deconvolutional layer takes the globally processed low-resolution feature map as input, and the last deconvolutional layer outputs the high-fidelity image.

8. The multimodal super-resolution liquefied gas leak detection method for heavy tank areas as described in claim 1, characterized in that, The high-fidelity image is then given a temporal leakage feature. Includes the following processes: The motion features of the leaked gas in the high-fidelity image are extracted using a background filtering mechanism. The extracted leakage gas motion features are fused in time sequence to obtain the time-series leakage feature map, thereby enhancing the sensitivity of the model's feature representation to moving targets; After downsampling and scaling the temporal leakage feature map to the same size as the high-fidelity image, the feature fusion of the temporal leakage feature map and the high-fidelity image is completed in the channel dimension through the Concat operation, and finally the high-fidelity image with temporal leakage features is obtained.

9. The multimodal super-resolution liquefied gas leak detection method for heavy tank areas as described in claim 1, characterized in that, The overall loss of the pre-built detection model L Including target detection loss L d and super-resolution reconstruction loss L sr Overall loss L The expression is as follows: ； in, Loss for target detection L d The weight, Loss for super-resolution reconstruction L sr The weight.

10. The multimodal super-resolution liquefied gas leak detection method for heavy tank areas as described in claim 9, characterized in that, The super-resolution reconstruction loss L sr The expression is as follows: ； in, S I represents the high-resolution, high-fidelity image output by the backbone network, and I represents the feature fusion map input to the backbone network. The target detection loss L d The expression is as follows: in, L obj Represents target loss; L loc This represents the loss in target positioning; L cls Represents the target classification loss; a 1. b 1 and c 1 represents the error weight values ​​for the corresponding target, target localization, and target classification in the target detection task; a 2. b 2 and c 2 represents the target loss. L obj Target location loss L loc And target classification loss L cls The weight value.

Images