An infrared methane leakage grading detection method based on mask video modeling

Abstract

This invention discloses an infrared methane leak classification detection method based on masked video modeling, relating to the fields of infrared gas leak detection and computer vision technology. The method includes: preprocessing acquired infrared methane leak videos; outputting a labeled sequence containing spatiotemporal information as a feature representation of the video segments; obtaining aggregated feature representations at the video segment level; inputting the aggregated feature representations at the video segment level into a classifier head to predict the probability distribution of eight levels of leak severity, thus completing the classification detection of infrared methane leak severity; and applying a Tent time-adaptive method to achieve lightweight online adaptation. This invention addresses the problems in existing infrared methane leak classification detection technologies, such as the difficulty in fully modeling the intermittent spatiotemporal distribution of methane and its long-range spatiotemporal dependence, which dilutes effective leak features, and the degradation of model detection performance due to changes in imaging distance, observation angle, and environmental conditions during actual deployment.

Description

Technical Field

[0001] This invention relates to the fields of infrared gas leak detection and computer vision technology, and more specifically to a graded detection method for infrared methane leaks based on masked video modeling. Background Technology

[0002] Industrial gas leaks pose serious safety and environmental risks to energy, chemical, and nuclear power systems. Methane, a major component of natural gas and many industrial production processes, can trigger explosions, fires, and other safety accidents upon leakage, and also exacerbate greenhouse gas emissions, causing long-term impacts on the atmospheric environment. Therefore, timely detection and accurate assessment of the severity of methane leaks are crucial for ensuring the safe operation of industrial facilities, reducing accident risks, and minimizing environmental pollution.

[0003] Currently, gas leak detection methods mainly include hardware sensor detection, negative pressure wave (NPW) detection, and acoustic detection. While these methods have some application value in specific scenarios, they typically rely on sensor deployment location, equipment sensitivity, and environmental stability. Under extreme weather or other industrial conditions, sensors may experience response lag and decreased sensitivity, making it difficult to identify methane leaks in a timely, stable, and accurate manner. Existing vision-based methane leak identification methods mostly employ convolutional neural networks (CNNs) or their improved structures. Although these methods can extract local spatial texture features, they still have certain limitations when processing infrared methane leak videos. First, infrared methane signals are weak, have low contrast, and are intermittently distributed in space and time, making it difficult for existing visual models to fully extract their long-range spatiotemporal dependent features. Second, the background area accounts for a large proportion of infrared videos, and traditional feature aggregation methods easily dilute effective leak features with background information. Third, differences in imaging distance, viewing angle, and environmental conditions between actual deployment scenarios and training scenarios result in insufficient robustness of the model in unlabeled testing scenarios.

[0004] Therefore, proposing an infrared methane leakage classification detection method based on masked video modeling to address the difficulties of existing technologies is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] In view of this, the present invention provides an infrared methane leak classification detection method based on masked video modeling, which solves the problems of existing infrared methane leak classification detection technologies, such as the difficulty in fully modeling the intermittent spatiotemporal distribution of methane and the long-range spatiotemporal dependence, which dilutes the effective leak features, and the degradation of model detection performance caused by changes in imaging distance, observation angle, and environmental conditions during actual deployment. By adopting a VideoMAE Transformer backbone network pre-trained with masked video and a lightweight spatiotemporal attention pooling module STAP, the dilution effect of the background-dominant region on weak leak features is reduced; and by applying a Tent test-time adaptive strategy based on entropy minimization, the model can achieve online adaptation to different deployment conditions.

[0006] To achieve the above objectives, the present invention provides the following technical solution: A graded detection method for infrared methane leaks based on masked video modeling includes the following steps: S1. Acquire infrared methane leakage video, preprocess the acquired infrared methane leakage video to obtain preprocessed video clips; S2. Input the preprocessed video segment into the VideoMAE encoder pre-trained by masked video modeling. Convert the infrared frame sequence into a spatiotemporal labeled sequence through a tubular embedding scheme. The VideoMAE encoder performs multi-layer self-attention calculation on the labeled sequence and outputs a labeled sequence containing spatiotemporal information as the feature representation of the video segment. S3. Based on self-attention pooling and attention multi-instance learning mechanism, the attention score of each label is calculated for the spatiotemporal label sequence output by S2. After softmax normalization, attention weights are generated. The spatiotemporal labels are weighted and aggregated to obtain the aggregated feature representation at the video segment level. S4. Input the aggregated feature representation at the video segment level into the classifier head to predict the probability distribution of the eight levels of leakage severity and complete the graded detection of the severity of infrared methane leakage. S5. During the inference and deployment phase, the Tent test-time adaptive method is applied to unlabeled target domain video segments to achieve lightweight online adaptation.

[0007] Optionally, the specific preprocessing steps for the acquired infrared methane leak video in S1 are as follows: The collected infrared methane leak video was converted into video segments with a fixed number of frames, and the video was divided into segments using a sliding window method. For each video segment, moving average background subtraction, difference-based thresholding, and normalization are applied to enhance the motion characteristics of the leaky plume, resulting in preprocessed video segments.

[0008] Optionally, the VideoMAE encoder in S2 consists of L layers of Transformer blocks, each layer containing a multi-head self-attention mechanism (MSA) and a feedforward neural network (FFN), and employing residual connections and layer normalization; the computation process of the l-th layer Transformer block is as follows:

[0009] Where LN(·) represents the layer normalization operation, and These are the input and output of the l-th layer Transformer block, respectively; Through multi-layer self-attention computation, the VideoMAE encoder can capture the dependencies between any two spatiotemporal locations in the labeled sequence, thereby outputting a labeled sequence containing spatiotemporal information as a feature representation of the video segment.

[0010] Optionally, the specific details of converting the infrared frame sequence into a spatiotemporal marker sequence using a tubular embedding scheme in S2 are as follows: The tubular embedding scheme first divides each frame of image into segments of size 1. The spatial patches are mapped to feature vectors of dimension C through linear projection. Then, in the time dimension, the corresponding spatial patches of consecutive t frames are spliced ​​into a spatiotemporal tubular marker to form the final spatiotemporal marker sequence.

[0011] Optionally, the attention score calculation of the spatiotemporal attention pooling module in S3 adopts a two-layer fully connected network structure: the first layer performs linear transformation and non-linear activation on the input labels, and the second layer projects the activated features into scalar attention scores; softmax normalization ensures that the sum of the attention weights of all labels is 1, and the attention weights after softmax normalization are used to perform a weighted summation on all spatiotemporal labels to obtain the aggregated feature representation at the video segment level as follows:

[0012] in, The aggregated video segment-level feature vectors have the same dimensions as the feature dimensions of a single tag. same, For the first Normalized attention weights for each label, and .

[0013] Optionally, in S4, the aggregated feature representation at the video clip level is input into the classifier head to predict the probability distribution of eight levels of leakage severity. The specific content of completing the graded detection of infrared methane leakage severity is as follows: The classifier head consists of a linear classification layer and a softmax function; The linear classification layer aggregates feature vectors from Dimension mapping to The unnormalized class score z is obtained from the dimensional class space; The unnormalized class scores z are converted into a probability distribution using the softmax function. ; The cross-entropy loss function is used during training; The model's final prediction is the class with the highest probability:

[0014] in, Let be the predicted probability of the model for the k-th severity level of the leak. This represents the total number of categories; The prediction results directly correspond to the severity level of the leak, providing maintenance personnel with a quantitative basis for leak assessment.

[0015] Optionally, the S5-based entropy-minimization Tent-time adaptive method performs online adaptation on unlabeled target domain video segments during the inference and deployment phase, specifically as follows: The adaptive method during Tent testing assumes that, under domain shift conditions, the model's prediction uncertainty for the target domain data increases, i.e., prediction entropy increases. Minimizing prediction entropy encourages the model to produce more deterministic predictions, thereby achieving adaptation from the source domain to the target domain. The prediction entropy loss function is defined as:

[0016] in, The number of categories representing the severity of the leak. This represents the model's predicted probability for the k-th severity level of the leak. When a model is highly confident about a particular class, its prediction entropy approaches zero; when the model makes uniform predictions for all classes, its prediction entropy reaches its maximum value. .

[0017] As can be seen from the above technical solution, compared with the prior art, the present invention provides an infrared methane leakage classification detection method based on masked video modeling, which has the following beneficial effects: (1) This invention introduces a lightweight spatiotemporal attention pooling (STAP) module, which effectively reduces the dilution effect of background dominant markers on sparse plume signals through a content-adaptive attention weighted aggregation mechanism, enabling the model to focus more accurately on spatiotemporal regions related to leakage and improve the accuracy of hierarchical classification. (2) The present invention applies a test-time adaptive (Tent) strategy based on entropy minimization. During the inference phase, only a small number of affine parameters of the normalization layer need to be updated to achieve online adaptation of the model to different deployment conditions. This significantly improves the robustness of the model in actual deployment while maintaining low computational overhead. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0019] Figure 1 A flowchart of an infrared methane leakage classification detection method based on masked video modeling provided by the present invention; Figure 2 This is a schematic diagram of the network structure provided in an embodiment of the present invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] See Figure 1 As shown, this invention discloses a graded detection method for infrared methane leakage based on masked video modeling, comprising the following steps: S1. Acquire infrared methane leakage video, preprocess the acquired infrared methane leakage video to obtain preprocessed video clips; S2. Input the preprocessed video segment into the VideoMAE encoder pre-trained by masked video modeling. Convert the infrared frame sequence into a spatiotemporal labeled sequence through a tubular embedding scheme. The VideoMAE encoder performs multi-layer self-attention calculation on the labeled sequence and outputs a labeled sequence containing spatiotemporal information as the feature representation of the video segment. S3. Based on self-attention pooling and attention multi-instance learning mechanism, the attention score of each label is calculated for the spatiotemporal label sequence output by S2. After softmax normalization, attention weights are generated. The spatiotemporal labels are weighted and aggregated to obtain the aggregated feature representation at the video segment level. S4. Input the aggregated feature representation at the video segment level into the classifier head to predict the probability distribution of the eight levels of leakage severity and complete the graded detection of the severity of infrared methane leakage. S5. During the inference and deployment phase, the Tent test-time adaptive method is applied to unlabeled target domain video segments to achieve lightweight online adaptation.

[0022] Furthermore, the specific preprocessing steps for the acquired infrared methane leak video in S1 are as follows: The collected infrared methane leak video was converted into video segments with a fixed number of frames, and the video was divided into segments using a sliding window method. For each video segment, moving average background subtraction, difference-based thresholding, and normalization are applied to enhance the motion characteristics of the leaky plume, resulting in preprocessed video segments.

[0023] Furthermore, the VideoMAE encoder in S2 consists of L layers of Transformer blocks, each layer containing a multi-head self-attention mechanism (MSA) and a feedforward neural network (FFN), and employs residual connections and layer normalization; the computation process of the l-th layer Transformer block is as follows:

[0024] Where LN(·) represents the layer normalization operation, and These are the input and output of the l-th layer Transformer block, respectively; Through multi-layer self-attention computation, the VideoMAE encoder can capture the dependencies between any two spatiotemporal locations in the labeled sequence, thereby outputting a labeled sequence containing spatiotemporal information as a feature representation of the video segment.

[0025] Furthermore, the specific details of converting the infrared frame sequence into a spatiotemporal marker sequence using the tubular embedding scheme in S2 are as follows: The tubular embedding scheme first divides each frame of image into segments of size 1. The spatial patches are mapped to feature vectors of dimension C through linear projection. Then, in the time dimension, the corresponding spatial patches of consecutive t frames are spliced ​​into a spatiotemporal tubular marker to form the final spatiotemporal marker sequence.

[0026] Furthermore, the attention score calculation in the spatiotemporal attention pooling module of S3 adopts a two-layer fully connected network structure: the first layer performs linear transformation and non-linear activation on the input labels, and the second layer projects the activated features into scalar attention scores; softmax normalization ensures that the sum of the attention weights of all labels is 1, and the attention weights after softmax normalization are used to perform a weighted summation on all spatiotemporal labels to obtain the aggregated feature representation at the video segment level as follows:

[0027] in, The aggregated video segment-level feature vectors have the same dimensions as the feature dimensions of a single tag. same, For the first Normalized attention weights for each label, and .

[0028] Furthermore, in S4, the aggregated feature representation at the video segment level is input into the classifier head to predict the probability distribution of eight levels of leakage severity. The specific content for completing the graded detection of infrared methane leakage severity is as follows: The classifier head consists of a linear classification layer and a softmax function; The linear classification layer aggregates feature vectors from Dimension mapping to The unnormalized class score z is obtained from the dimensional class space; The unnormalized class scores z are converted into a probability distribution using the softmax function. ; The cross-entropy loss function is used during training; The model's final prediction is the class with the highest probability:

[0029] in, Let be the predicted probability of the model for the k-th severity level of the leak. This represents the total number of categories; The prediction results directly correspond to the severity level of the leak, providing maintenance personnel with a quantitative basis for leak assessment.

[0030] Furthermore, the S5's entropy-minimization-based Tent-time adaptive method for online adaptation of unlabeled target domain video segments during the inference and deployment phase includes the following: The adaptive method during Tent testing assumes that, under domain shift conditions, the model's prediction uncertainty for the target domain data increases, i.e., prediction entropy increases. Minimizing prediction entropy encourages the model to produce more deterministic predictions, thereby achieving adaptation from the source domain to the target domain. The prediction entropy loss function is defined as:

[0031] in, The number of categories representing the severity of the leak. This represents the model's predicted probability for the k-th severity level of the leak. When a model is highly confident about a particular class, its prediction entropy approaches zero; when the model makes uniform predictions for all classes, its prediction entropy reaches its maximum value. .

[0032] In a specific embodiment, such as Figure 2 As shown, it includes the following: S1. Acquire infrared video of methane leakage, and preprocess the acquired infrared video of methane leakage to obtain preprocessed video clips; the specific process is as follows: Assume the input infrared methane leak video is a continuous frame sequence. ,in The image is an infrared grayscale image of frame t with a resolution of H×W pixels. In this embodiment, the infrared video is acquired by an infrared optical gas imaging camera with a frame rate of 30fps and an image resolution of 320×240 pixels. A sliding window method is used to convert the video into video segments of fixed length T=16 frames. The specific sampling method is as follows: starting from frame t, 15 frames are selected consecutively, and frame t+14 is copied as the 16th frame to match the input length requirement of the VideoMAE encoder. The step size of the sliding window is set according to the total video length and the required segment density, and frames overlap between adjacent segments to ensure temporal continuity. Let the generated video segment set be... ,in The total number of segments; Calculate pixel-level moving average background model for consecutive frames , will the current frame With background model Pixel-by-pixel subtraction yields the difference image The formula is as follows:

[0033] in, Represents pixel coordinates, It is updated in real time using an exponentially weighted moving average to adapt to slow changes in the background; Differential images Perform threshold-based binarization; set the threshold. When the pixel difference value exceeds If the value is valid, it is retained; otherwise, it is set to zero, thus eliminating background noise and highlighting the leaking plume region. Finally, the processed image pixel values ​​are linearly normalized to the range of [0,1], as shown in the following formula:

[0034] in, The maximum pixel value in the difference image. The frame is a normalized preprocessed frame. After the above preprocessing, the motion characteristics of the leaky plume are significantly enhanced and the background noise is effectively suppressed.

[0035] S2. Input the preprocessed video segment into the VideoMAE encoder pre-trained with masked video modeling. The infrared frame sequence is converted into a spatiotemporal labeled sequence using a tubular embedding scheme. The VideoMAE encoder performs multi-layer self-attention calculations on the labeled sequence, outputting a labeled sequence containing spatiotemporal information as the feature representation of the video segment. The specific process is as follows: The VideoMAE encoder uses a tubular embedding scheme to convert the input 16 frames of single-channel infrared images into a spatiotemporal marker sequence; let the embedded marker sequence be... Where N is the number of spatiotemporal markers and C is the feature dimension of the markers; the tubular embedding scheme first divides each frame of image into segments of size N. The spatial patches are mapped to feature vectors of dimension C through linear projection; then, in the time dimension, the corresponding spatial patches of consecutive t frames (t=2 in this embodiment) are spliced ​​into a spatiotemporal tubular marker to form the final spatiotemporal marker sequence. The VideoMAE encoder consists of L Transformer blocks (L=12 in this embodiment), each layer containing a multi-head self-attention mechanism (MSA) and a feedforward neural network (FFN), and employs residual connections and layer normalization; the computation process of the l-th Transformer block is as follows:

[0036] Where LN(·) represents the layer normalization operation, and These are the input and output of the l-th layer Transformer block, respectively. Through multi-layer self-attention computation, the encoder can capture the dependency between any two spatiotemporal locations in the labeled sequence, effectively modeling the long-range spatiotemporal dynamics in infrared video. In this embodiment, the pre-training weights of the VideoMAE encoder are derived from the masked video self-supervised pre-training model on the Kinetics-400 dataset. During the pre-training stage, an extremely high masking rate (90% in this embodiment) is used to randomly mask 90% of the spatiotemporal markers in the video clip, encoding only the visible 10% of the markers. Then, a lightweight decoder is used to reconstruct the pixel content of the masked markers. The high masking rate forces the encoder to learn the ability to infer global spatiotemporal information from a very small number of visible markers, thereby obtaining a high-quality spatiotemporal representation. In the downstream leakage classification task, the decoder used in the masked video pre-training stage is discarded, and only the encoder part is retained. During fine-tuning, a layered freezing strategy is adopted: only the parameters of the last four Transformer blocks (layers 9 to 12), the parameters of the spatiotemporal attention pooling module, and the parameters of the classifier head are updated, while the parameters of the first eight Transformer blocks are kept frozen. This strategy not only preserves the general ability of the pre-trained model in low-level feature extraction, but also allows high-level features to be adapted to the specific task of infrared leakage detection. The optimizer used is AdamW, with an initial learning rate of 5×10⁻⁶. -4 The weight decay was set to 0.05, the learning rate was scheduled using a cosine annealing strategy, the total number of training rounds was set according to the performance of the validation set, and all models were trained and evaluated under five different random seeds to ensure the statistical reliability of the results.

[0037] S3. Based on self-attention pooling and attention multi-instance learning mechanisms, the attention score of each label in the spatiotemporal label sequence output by S2 is calculated. After softmax normalization, attention weights are generated. The spatiotemporal labels are then weighted and aggregated to obtain the aggregated feature representation at the video segment level. The specific process is as follows: Because infrared methane plumes are spatially localized and temporally intermittent, only a small portion of the N spatiotemporal markers corresponding to the 16-frame video segment contain information related to the leaking plume, while a large number of markers correspond to static background regions. If a traditional uniform aggregation method (such as global average pooling, i.e., taking a simple average of all markers) is used, the weak plume signal will be diluted by a large number of background-dominated markers, resulting in an extremely low signal-to-noise ratio of leakage-related information in the aggregated features. The STAP module (Spatiotemporal Attention Pooling module) assigns content-adaptive attention weights to each spatiotemporal marker through a self-attention mechanism, giving higher weights to markers with high leakage relevance and achieving selective aggregation of markers. Given the label sequence output by S2 The STAP module first calculates the raw attention score for each label using an attention network:

[0038] Where σ(·) represents a nonlinear activation function (the tanh activation function is used in this embodiment). Here, d is the weight matrix of the first layer, and d is the dimension of the hidden layer. This is the weight vector for the second layer. and These are the bias terms; this attention network has a shared parameter structure, using the same weights for all labels, resulting in a computational complexity of only [missing value]. The cost is extremely low; The attention scores are normalized using a softmax operation to ensure that the attention weights of all labeled items are non-negative and sum to 1.

[0039] in, For the first Normalized attention weights for each label, and ; Softmax normalization gives the attention weights a probabilistic interpretation, which facilitates subsequent gradient calculation and optimization; By applying the normalized attention weights to a weighted sum of all spatiotemporal labels, we obtain an aggregated feature representation at the video segment level:

[0040] in, The aggregated video segment-level feature vectors have the same dimensions as the feature dimensions of a single tag. Same; compared to global average pooling Attention-weighted aggregation can adaptively assign weights based on the content of each tag: when the spatial region and temporal frame corresponding to a tag contain obvious leaky plume signals, its attention weights are adjusted accordingly. The weight will be significantly higher than that of the background label, thus making the aggregated features... The proportion of information related to the leak was higher, effectively highlighting the spatiotemporal characteristics of the methane leak plume.

[0041] S4. Input the aggregated feature representation at the video segment level into the classifier head to predict the probability distribution of eight levels of leakage severity, thus completing the graded detection of infrared methane leakage severity; the specific process is as follows: The classifier head consists of a linear classification layer and a softmax function; The linear classification layer aggregates feature vectors from Dimension mapping to The unnormalized class scores are obtained from the dimensional class space. :

[0042] in, This is the classification weight matrix. This is the classification bias vector; The category score z is converted into a probability distribution using the softmax function. :

[0043] in, Let be the predicted probability of the model for the k-th severity level of the leak. The total number of categories; the eight leakage levels, from Level 0 to Level 7, correspond to eight intervals of a logarithmically uniform distribution of leakage rates from 5.3 to 2051.6 gCH4 / h; the cross-entropy loss function is used during training:

[0044] in, The true label for leakage level k; The model's final prediction is the class with the highest probability:

[0045] The prediction results directly correspond to the severity level of the leak, providing maintenance personnel with a quantitative basis for leak assessment.

[0046] S5. During the inference and deployment phase, the Tent test-time adaptive method is applied to unlabeled target domain video segments to achieve lightweight online adaptation; the specific process is as follows: In actual deployments, due to changes in imaging distance (e.g., switching from close-range to long-range observation during training), adjustments in observation angle, and fluctuations in environmental conditions (temperature, humidity, wind speed, etc.), the distribution of test domain data may differ from that of training data, i.e., the domain offset problem. This domain offset can lead to a decrease in the model's classification accuracy in the target domain. Traditional domain adaptation methods require labeled data from the target domain for model fine-tuning, but obtaining a large amount of labeled data is costly in practical applications. The Tent test-adaptive method utilizes the structural information of the unlabeled target domain data itself to achieve model adaptation in a self-supervised manner. The core idea of ​​the Tent-based adaptive method is to minimize the model's prediction entropy on the target domain video segment. Prediction entropy is a measure of the model's prediction uncertainty; the more certain the model's prediction, the smaller the prediction entropy. The Tent-based adaptive method assumes that under domain shift conditions, the model's prediction uncertainty for the target domain data increases (i.e., prediction entropy increases). Minimizing prediction entropy can encourage the model to produce more certain predictions, thereby achieving adaptation from the source domain to the target domain. The prediction entropy loss function is defined as:

[0047] in, The number of categories representing the severity of the leak. This represents the model's predicted probability for the k-th severity level of the leak; the prediction entropy approaches 0 when the model is highly confident about a certain category; the prediction entropy reaches its maximum when the model gives uniform predictions for all categories. ; During the adaptive testing process, a parameter selective update strategy is employed. All parameters of the self-attention layer (including the query, key, value projection matrices, and output projection matrices in multi-head self-attention), feedforward neural network parameters (including the two linear layers and their biases), STAP module parameters, and classifier parameters are completely frozen and do not participate in gradient updates. Only the affine parameters of the Layer Normalization layer in each Transformer block are updated. ,in For scaling parameters, The offset parameter is used; the formula for calculating layer normalization is:

[0048] in, and These are the mean and standard deviation of the labeled features, respectively. and The parameters are learnable affine parameters; by updating only the normalized layer parameters, the number of trainable parameters in the adaptive process is extremely small, accounting for less than 1% of the total number of parameters in the VideoMAE architecture, which ensures the adaptive capability and avoids the risk of performance degradation caused by over-updating. The gradient update uses stochastic gradient descent, and the update formula is:

[0049] in, For adaptive learning rate, this embodiment sets it to... Gradient updates are performed once per minimum batch. The adaptive process follows an online streaming protocol: the normalized layer parameters after adaptation are continuously passed and accumulated between test batches, rather than being adapted and reset independently for each video segment; this streaming protocol enables the model to gradually adapt to the overall distribution characteristics of the target domain, avoiding the parameter oscillation problem that may be introduced by independent adaptation segment by segment; the global test batch size is set to 8, that is, parameter updates are performed after processing 8 video segments each time. It should be noted that this invention enhances the motion characteristics of the leaking plume through video preprocessing in S1, providing high-quality input data for subsequent feature extraction. Building upon this, S2 utilizes a VideoMAE encoder pre-trained with masked video to effectively capture long-range spatiotemporal dependencies in infrared video, overcoming the inherent limitations of traditional CNN methods in sparse plume modeling. The high masking rate pre-training strategy enables the model to achieve excellent spatiotemporal representation capabilities. Simultaneously, S3 employs a content-adaptive attention-weighted aggregation mechanism using a lightweight spatiotemporal attention pooling module, effectively reducing the dilution effect of background-dominant markers on the sparse plume signal, making the aggregated features more focused on the leak-related region. S4 achieves fine classification of eight levels of leak severity through the classifier head, outputting quantified leak level assessment results. S5 utilizes a Tent-time adaptive method, requiring only minor parameter updates (less than 1% of the normalization layer) to achieve online adaptation of the model under different deployment conditions. The streaming protocol ensures the stability of the adaptation process, significantly improving robustness in actual deployment while maintaining low latency for each segment.

[0050] The method of this invention fully meets the actual needs of infrared methane leakage classification detection in fields such as energy, chemical industry and nuclear power system through the collaborative innovative design of three core modules: masked video pre-training backbone network, lightweight attention pooling mechanism and test-time adaptation. Masked video pre-training provides high-quality spatiotemporal feature representation, attention pooling mechanism enhances the ability to extract leakage signals, and test-time adaptation improves the robustness of the model under different deployment conditions.

[0051] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0052] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A graded detection method for infrared methane leakage based on masked video modeling, characterized in that, Includes the following steps: S1. Acquire infrared methane leakage video, preprocess the acquired infrared methane leakage video to obtain preprocessed video clips; S2. Input the preprocessed video segment into the VideoMAE encoder pre-trained by masked video modeling. Convert the infrared frame sequence into a spatiotemporal labeled sequence through a tubular embedding scheme. The VideoMAE encoder performs multi-layer self-attention calculation on the labeled sequence and outputs a labeled sequence containing spatiotemporal information as the feature representation of the video segment. S3. Based on self-attention pooling and attention multi-instance learning mechanism, the attention score of each label is calculated for the spatiotemporal label sequence output by S2. After softmax normalization, attention weights are generated. The spatiotemporal labels are weighted and aggregated to obtain the aggregated feature representation at the video segment level. S4. Input the aggregated feature representation at the video segment level into the classifier head to predict the probability distribution of the eight levels of leakage severity and complete the graded detection of the severity of infrared methane leakage. S5. During the inference and deployment phase, the Tent test-time adaptive method is applied to unlabeled target domain video segments to achieve lightweight online adaptation.

2. The infrared methane leakage classification detection method based on masked video modeling according to claim 1, characterized in that, The specific preprocessing steps for the acquired infrared methane leak video in S1 are as follows: The collected infrared methane leak video was converted into video segments with a fixed number of frames, and the video was divided into segments using a sliding window method. For each video segment, moving average background subtraction, difference-based thresholding, and normalization are applied to enhance the motion characteristics of the leaky plume, resulting in preprocessed video segments.

3. The infrared methane leakage classification detection method based on masked video modeling according to claim 1, characterized in that, The VideoMAE encoder in S2 consists of L Transformer blocks, each containing a multi-head self-attention mechanism (MSA) and a feedforward neural network (FFN), and employs residual connections and layer normalization; the computation process of the l-th Transformer block is as follows: Where LN(·) represents the layer normalization operation, and These are the input and output of the l-th layer Transformer block, respectively; Through multi-layer self-attention computation, the VideoMAE encoder can capture the dependencies between any two spatiotemporal locations in the labeled sequence, thereby outputting a labeled sequence containing spatiotemporal information as a feature representation of the video segment.

4. The infrared methane leakage classification detection method based on masked video modeling according to claim 1, characterized in that, The specific details of converting the infrared frame sequence into a spatiotemporal marker sequence using the tubular embedding scheme in S2 are as follows: The tubular embedding scheme first divides each frame of image into segments of size 1. The spatial patches are mapped to feature vectors of dimension C through linear projection. Then, in the time dimension, the corresponding spatial patches of consecutive t frames are spliced ​​into a spatiotemporal tubular marker to form the final spatiotemporal marker sequence.

5. The infrared methane leakage classification detection method based on masked video modeling according to claim 1, characterized in that, The attention score calculation of the spatiotemporal attention pooling module in S3 adopts a two-layer fully connected network structure: the first layer performs linear transformation and non-linear activation on the input label, and the second layer projects the activated features into a scalar attention score. Softmax normalization ensures that the sum of the attention weights of all labels is 1. The attention weights after softmax normalization are then used to perform a weighted summation of all spatiotemporal labels, resulting in the aggregated feature representation at the video segment level: in, The aggregated video segment-level feature vectors have the same dimensions as the feature dimensions of a single tag. same, For the first Normalized attention weights for each label, and .

6. The infrared methane leakage classification detection method based on masked video modeling according to claim 1, characterized in that, In S4, the aggregated feature representation at the video clip level is input into the classifier head to predict the probability distribution of eight levels of leakage severity. The specific content of completing the graded detection of infrared methane leakage severity is as follows: The classifier head consists of a linear classification layer and a softmax function; The linear classification layer aggregates feature vectors from Dimension mapping to The unnormalized class score z is obtained from the dimensional class space; The unnormalized class scores z are converted into a probability distribution using the softmax function. ; The cross-entropy loss function is used during training; The model's final prediction is the class with the highest probability: in, Let be the predicted probability of the model for the k-th severity level of the leak. This represents the total number of categories; The prediction results directly correspond to the severity level of the leak, providing maintenance personnel with a quantitative basis for leak assessment.

7. The infrared methane leakage classification detection method based on masked video modeling according to claim 1, characterized in that, The S5-based Tent-based adaptive testing method, which minimizes entropy, performs online adaptation on unlabeled target domain video segments during the inference and deployment phase. The specific details are as follows: The adaptive method during Tent testing assumes that, under domain shift conditions, the model's prediction uncertainty for the target domain data increases, i.e., prediction entropy increases. Minimizing prediction entropy encourages the model to produce more deterministic predictions, thereby achieving adaptation from the source domain to the target domain. The prediction entropy loss function is defined as: in, The number of categories representing the severity of the leak. This represents the model's predicted probability for the k-th severity level of the leak. When a model is highly confident about a particular class, its prediction entropy approaches zero; when the model makes uniform predictions for all classes, its prediction entropy reaches its maximum value. .

Images