A method for monitoring a debris flow based on a scene change

Abstract

The present application belongs to the technical field of deep learning target detection, and particularly relates to a debris flow monitoring method based on scene change. The method labels the scenes before and after the occurrence of debris flow, generates a change binary graph, uses a Light-ChangeCNNformer network to detect changes, extracts the changed area in the scene, and then inputs the extracted changes into a MobileNetV3 classification network trained by the change binary graph to determine whether the change is a debris flow. In the Light-ChangeCNNformer, the CNN is creatively embedded into the Transformer encoder, and the CNN efficiently extracts the shallow local features to depict the change boundary. Moreover, the intermediate features extracted by the CNN branch, which focus on spatial details and local patterns, are highly matched with the feature types that the MobileNetV3 lightweight deep separable convolution is good at processing, avoiding the "feature redundancy" and "information overload" problems that may be caused by directly inputting the deep abstract features of the Transformer into the lightweight classifier, reducing the false detection rate of the debris flow detection, and improving the accuracy of the debris flow detection.

Description

Technical Field

[0001] This invention belongs to the field of deep learning target detection technology, and in particular relates to a debris flow monitoring method based on scene changes. Background Technology

[0002] With the continuous development of natural disaster monitoring technology, debris flow monitoring plays a crucial role in geological disaster early warning, emergency response, and environmental monitoring. As a natural disaster characterized by its sudden onset and destructive power, the timeliness and accuracy of debris flow monitoring data are essential for reducing casualties and property damage. Traditional debris flow monitoring mainly relies on field investigations, remote sensing image analysis, and manual visual judgment. These methods are not only time-consuming and labor-intensive, but also struggle to capture the dynamic changes of debris flows in real time and accurately.

[0003] Currently, debris flow detection methods based on computer vision technology are emerging. These methods primarily utilize high-resolution image data such as satellite images and drone aerial images, employing image processing and analysis techniques to identify the occurrence, development, and impact range of debris flows. Based on their detection principles, these methods can be categorized into debris flow detection based on traditional image processing algorithms and debris flow detection based on deep learning. While detection methods based on traditional image processing algorithms have lower computational complexity, their detection accuracy is often limited by image quality, terrain complexity, and the diversity of debris flow features. Deep learning-based methods, although offering significant advantages in detection accuracy, are mostly based on single-scene segmentation or direct detection. This approach is susceptible to background interference in complex environments, leading to decreased detection accuracy, and it fails to utilize information about scene changes before and after a debris flow. Debris flow images exhibit the following characteristics: complex and diverse terrain, with debris flow paths potentially traversing valleys, rivers, and vegetated areas; significant variations in morphology and color across different stages and environments, potentially manifesting as mudflows, gravel flows, and other forms, which can easily be confused with the surrounding environment; furthermore, image noise, variations in illumination, and seasonal vegetation changes can significantly impact detection accuracy. These factors make it difficult for existing debris flow detection methods to simultaneously achieve high accuracy and high efficiency.

[0004] Chinese patent document CN117636071A discloses a deep learning-based debris flow susceptibility assessment method based on multi-source data. The method first uses factor collinearity analysis and importance analysis to output key debris flow influencing factors within a region, then uses a deep learning-based feature extraction network to extract features from both the key debris flow influencing factors and remote sensing images. These features are then fused using a feature-level fusion strategy. Finally, the method assesses the region's debris flow susceptibility and generates a distribution map of the region's susceptibility. However, this patent focuses on macro-level prediction and classification of long-term risk areas. Its method relies on the analysis and fusion of multi-source influencing factors, aiming to create a risk distribution map. It lacks the ability to capture the dynamic evolution of disasters within time-series image sequences, making it difficult to directly meet the real-time detection, segmentation, and confirmation needs of ongoing or recently occurred debris flow events.

[0005] In summary, existing debris flow detection methods still suffer from low detection accuracy and are prone to missed and false detections, making it difficult to meet the needs of real-time and efficient debris flow disaster monitoring. Therefore, this invention proposes a debris flow monitoring method based on ChangeFormer change detection and MobileNetV3 classification to improve the model's detection performance and achieve accurate and rapid debris flow monitoring. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the aforementioned background technology and provide a debris flow monitoring method based on scene changes. This method involves labeling the collected debris flow scenes before and after the event to generate a change binary map. This map is then combined with the original images (before and after the change) as training data. A Changeformer network optimized for debris flow scenes is used for change detection, extracting the changed regions in the scene. The extracted changes are then input into a MobileNetV3 classification network trained from the change binary map to determine whether the change is a debris flow. This invention uses a CNN architecture within the Changeformer encoder and optimizes the network decoder for debris flow scenes, making the network more suitable for debris flow scenarios and significantly reducing network parameters.

[0007] The detailed technical solution of this invention is as follows: A debris flow monitoring method based on scene changes includes the following steps: S1. Extract frames from the video of the debris flow to obtain a comparison image of the scene before and after the debris flow, and use image generation technology to increase the data capacity. S2. Preprocess the scene change comparison image caused by non-debris flow and the scene comparison image before and after debris flow obtained in step S1 to generate a binarized label image, obtain the change detection dataset, and divide it into training set, validation set and test set. Each data sample in the change detection dataset consists of a scene image before the change, a scene image after the change and the corresponding binarized label image, as well as the debris flow and non-debris flow categories corresponding to the binarized label image. S3. Construct a debris flow change detection model Light-ChangeCNNformer based on the Changeformer model. Train the debris flow change detection model with the training set divided in step S2. Input the validation set divided in step S2 into the trained Light-ChangeCNNformer model and output the predicted scene change binary map to select the detection model Light-ChangeCNNformer with the optimal parameters. S4. Build a change detection and classification network MobileNetV3 based on the MobileNet network. Use the binarized labeled images in the training set divided in step S2 and their corresponding debris flow and non-debris flow categories to train it. Input the predicted scene change binary map output in S3 into the trained MobileNetV3 to determine whether the change in the scene is a debris flow and select the MobileNetV3 model with the optimal parameters. S5. Input the test set from step S2 into the detection models Light-ChangeCNNformer and MobileNetV3 with optimal parameters selected in steps S3 and S4 for testing, evaluate the feasibility of the model output results, detect the real debris flow scene captured by the camera, and verify the feasibility of the method.

[0008] Preferably, the debris flow video in step S1 is obtained from real camera footage and online video. The monitoring equipment is installed at an angle in the debris flow-prone area, with the viewing angle of the monitoring equipment being 30-70 degrees. The real debris flow scene is used as the reference image, and image generation technology is used to generate an image that matches the real debris flow scene.

[0009] Preferably, the preprocessing in step S2 includes: labeling the changed areas of the scene change comparison map not caused by debris flow and the scene change comparison map before and after debris flow obtained in step S1, obtaining the changed locations, performing binarization processing on the changed locations to generate binarized label images, and then performing data augmentation to obtain a change detection dataset.

[0010] Preferably, the debris flow change detection model Light-ChangeCNNformer in step S3 includes a Transformer encoder embedded with a CNN and a lightweight decoder. The CNN includes multiple convolutional layers, BN layers and activation functions connected in sequence, and is embedded in the transformer structure through patch-embedding. The Transformer structure includes several stacked Transformer layers. The lightweight decoder optimizes the parameters based on the changeformer decoder to reduce the number of model parameters and make it adaptable to different types of devices.

[0011] More preferably, the construction of the Transformer encoder embedded in the CNN is as follows: The system is configured with dual input channels to receive scene images before and after the changes in time phases T1 and T2, respectively, and preprocessing is performed using shared weights. Constructing a CNN feature extraction branch: Perform CNN feature extraction on the preprocessed image data to obtain single-phase feature maps. And single-phase feature maps Perform secondary CNN feature extraction to output single-temporal feature maps. ; Constructing the Transformer structure: The Transformer structure is divided into two parts. The first part consists of PatchEmbedding and multiple Transformer layers. The second part is based on the first part, with the addition of normalization and convolution. Patch Embedding is used to output single-phase feature maps. Importing this into the Transformer network, the 2D feature map output from the CNN feature extraction branch is converted into a 1D continuous sequence. Simultaneously, the 2D feature map is sliced ​​to preserve local continuity, resulting in the output of the transition layer. The formula is as follows: (1) (2) in, Indicates after processing This is also the first input to the Transformer, d=256. For learnable positional encoding, and These represent the length and width of the image, respectively. This refers to the number of spatial locations; The output of the transition layer is normalized and a multi-head attention mechanism is applied to stabilize the training process and prevent gradient explosion or gradient vanishing. The calculation formula is as follows: (3) (4) in, This represents the output of the previous layer; the first input is... , The result after normalization. () represents the normalization process. MHSA() is a multi-head self-attention mechanism, composed of multiple spatial reduction attention heads (SRA) connected in parallel. This represents the result of the transition layer output after normalization and multi-head attention, and it is also the input to the second part of the Transformer network. The features are obtained after passing through multiple Transformer layers. ; feature After the second part of Transformer structural feature processing, the output features are... .

[0012] More preferably, the multi-head attention consists of multiple spatially compressed attention heads connected in parallel, including query-key-value calculation, attention weight calculation, and output projection.

[0013] More preferably, the lightweight decoder is constructed as follows: Multi-scale feature fusion: combining the two layers of features output from the feature extraction branch of a CNN. , The two-layer features output by the Transformer structure , Perform dual-temporal feature differencing to obtain the feature differences corresponding to different time periods. The calculation formula is as follows: (5) in, This indicates the feature hierarchy; there are four feature layers in the network. , , , These represent the corresponding layer output features for the two inputs. This indicates the difference between the two outputs.

[0014] High-level characteristics Perform 2x upsampling: ,in For subpixel convolution, its mathematical expression is: (6) Where R represents the feature map resolution, and F represents the upsampled features of the decoder. This refers to tensor reorganization, which rearranges the channel dimensions into the spatial dimensions to improve resolution. The upsampling factor; Then, the upsampling results are compared with the mid-layer Transformer features. The data is concatenated to achieve cross-scale feature fusion, and 1×1 convolution is used for dimensionality reduction. The calculation formula is as follows: (7) in, This represents the feature obtained by upsampling the features from the fourth layer and fusing them with the features from the third layer.

[0015] Progressive upsampling decoding: This method uses upsampling to convert low-resolution features into high-resolution features. It consists of three stages, and the calculation formula is as follows: (8) (9) (10) in, This represents the characteristic difference value calculated for the first time in formula (5). This represents the feature map from the first bilinear interpolation upsampling. This represents the feature map after upsampling and fusion in the second stage. This is the feature map output by the decoder.

[0016] in It includes bilinear interpolation upsampling, 3×3 depthwise separable convolution and ReLU activation function, and Proj is a 1×1 convolution responsible for aligning channels; The construction of the network detection head mainly consists of three parts: input processing, change prediction, and loss function; input processing involves converting the feature map output by the decoder... Features are enhanced by compressing channels using 1×1 convolutions and increasing the depth of 3×3 separable convolutions, resulting in an 8-channel feature output. Change prediction is achieved through 1×1 convolution and Processing features Obtain a 2-channel probability map Binarization based on P yields a predicted scene change binary map, used to distinguish between changed and unchanged regions; the loss function used is Dice loss, calculated as follows: (11) in A binary matrix representing the true labels, where a value of 1 indicates a changed region and a value of 0 indicates an unchanged region.

[0017] Preferably, the variation classification network MobileNetV3 described in step S4 introduces depthwise separable convolution and activation functions. The depthwise separable convolution decomposes standard convolution into two independent steps: depthwise convolution and pointwise convolution. The depthwise convolution performs convolution operation on each input channel individually, while the pointwise convolution mixes the output of the depthwise convolution through 1×1 convolution.

[0018] More preferably, the change classification network in step S4 uses the H-Swish activation function, which is defined as: (12) Where f is the feature value of the input activation function.

[0019] Furthermore, MobileNetV3 introduces a Squeeze-and-Excitation (SE) module. The SE module compresses the feature map into a channel-dimensional vector using global average pooling, then learns the dependencies between channels through two fully connected layers, generates channel weights using the sigmoid function, and finally multiplies the weights with the original feature map to recalibrate the features. The calculation process of the SE module can be represented as follows: (13) in It is the input feature map. This indicates element-wise multiplication. It is the sigmoid function. and These are the weights of the fully connected layer, and GAP is the global average pooling operation.

[0020] Preferably, the optimal parameters mentioned in steps S3 and S4 represent the optimal accuracy, recall, and F-value of the model generated in each round of training on the validation set as training progresses.

[0021] Compared with the prior art, the beneficial effects of the present invention are: (1) Existing technologies for debris flow target detection often employ manual observation or image-based segmentation methods, such as real-time manual observation of camera data or detection of continuous frame images from a camera based on a deep learning model, or segmentation of debris flow regions in an image using a segmentation model. This invention uses a method of comparing differences before and after a scene to determine debris flow. It judges the changed areas of the scene based on the differences in terrain or surface color gamut before and after a debris flow, and classifies the changed areas to determine whether a debris flow has occurred, thereby reducing the false detection rate of debris flow detection and improving the accuracy of debris flow detection.

[0022] (2) The method of this invention achieves end-to-end collaborative optimization of feature extraction and classification tasks, solving the core contradiction of difficulty in balancing "accuracy" and "efficiency" in debris flow monitoring. Traditional methods either rely on computationally intensive complex models that are difficult to deploy in real time, or use lightweight models but sacrifice detection accuracy. This invention organically combines the lightweight change detection network Light-ChangeCNNformer with the classification network MobileNetV3: Light-ChangeCNNformer focuses on accurately locating and extracting change areas in the scene, and its output change binary map provides a clean and structured input for subsequent classification, significantly reducing the complexity and computational load of the classification task; MobileNetV3 relies on this optimized input to efficiently identify change attributes, and its classification results can inversely optimize the accuracy of change detection. The two support each other, forming an efficient closed loop from "change detection" to "disaster identification", and the overall performance far exceeds that of a single model or simple combination.

[0023] (3) The network structure of this invention is designed with deep customization for debris flow scenarios, and there are non-obvious technical connections between the components. This invention is not a simple splicing of general models, but an integrated design based on the characteristics of debris flow, which has "large change area and relatively simple outline". In Light-ChangeCNNformer, CNN is creatively embedded into the Transformer encoder, and CNN is used to efficiently extract shallow local features to characterize the change boundary. This not only adapts to the morphological characteristics of debris flow, but also greatly reduces the shallow computational complexity of Transformer. More importantly, the intermediate features extracted by the CNN branch, which focus on spatial details and local patterns, are highly matched with the feature types that MobileNetV3 lightweight deep separable convolution is good at handling, avoiding the "feature redundancy" and "information overload" problems that may occur if the deep abstract features of Transformer are directly input into the lightweight classifier. This "lightweight feature transfer chain" design from feature extraction to classifier is not something that those skilled in the art would easily think of when directly applying existing models.

[0024] (4) The method of this invention breaks the technical inertia in the field of debris flow detection and achieves unexpected technical results. In the prior art, high-performance change detection models (such as the original ChangeFormer) are mostly designed for complex scenes with small targets such as urban buildings in remote sensing images. They have a large number of parameters and high computational cost. On the other hand, lightweight classification networks are easily affected by background interference when directly processing complex remote sensing images. This invention is the first to jointly optimize and adapt the two for debris flow scenarios, filling the technical gap in this sub-field. Experimental results show that the optimized Light-ChangeCNNformer improves the F-value of change detection by 2.4 percentage points (from 95.2% to 97.6%) while reducing the number of parameters by 76% (from 41Mb to 9.7Mb). Finally, the precision (P3) of the entire scheme for debris flow identification reaches 0.9294, successfully achieving high-precision monitoring within the computing power range that edge devices can bear, and providing a practical technical solution for real-time early warning of debris flows.

[0025] (5) The classification method of this invention is based on the lightweight network architecture of MobileNetV3. Through its efficient InvertedResidual structure and Squeeze-Excite module, it achieves efficient extraction and classification of input image features. By utilizing the depthwise separable convolution of MobileNetV3 to reduce the amount of computation, and combining it with activation functions to improve the model's expressive power, the model complexity is significantly reduced and the classification efficiency is improved while ensuring classification accuracy. It is suitable for various image classification scenarios. Attached Figure Description

[0026] Figure 1 This is a schematic flowchart of the debris flow monitoring method based on scene changes according to the present invention. Figure 2 This describes the data processing flow and results in this invention. Figure 3 This is a schematic diagram of the Light-ChangeCNNformer debris flow change detection network in this invention; Figure 4 This invention presents the monitoring results of real debris flow occurrence scenes captured by cameras. Detailed Implementation To make the objectives, technical solutions, and advantages of the present invention clearer, the technical solutions in the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings. It should be emphasized that the following description is merely exemplary and not intended to limit the scope or application of the present invention.

[0027] Example 1 This embodiment proposes a debris flow monitoring method based on scene changes. The method includes the following steps, such as... Figure 1 As shown: S1: Extract frames from the video of the debris flow to obtain a comparison image of the scene before and after the debris flow, and use image generation technology to increase the data capacity. Specifically, the debris flow videos are sourced from real cameras and online videos. The monitoring equipment is installed at an angle in debris flow-prone areas, with a viewing angle of 30-70 degrees. The actual debris flow scene is used as a reference image, and image-generated images that match the actual debris flow scene are generated using image-generated image technology.

[0028] Specifically, the project involved obtaining comparative images of the scene before and after a debris flow, deploying data acquisition equipment in areas prone to debris flows to monitor for debris flows in real time, saving video segments from before to after a debris flow, crawling relevant data from websites such as Baidu and video material websites, using Comfyui technology to generate images that highly match the actual debris flow scene, cleaning the data, and obtaining 1200 pairs of usable data images.

[0029] S2. Preprocess the scene change comparison images caused by non-debris flow and the scene comparison images before and after debris flow obtained in step S1. Process them into images that conform to the input format of the Changeformer network, generate binarized label images, obtain the change detection dataset, and divide it into training set, validation set and test set. Each data sample in the change detection dataset consists of a scene image before the change, a scene image after the change and the corresponding binarized label image, as well as the debris flow and non-debris flow categories corresponding to the binarized label image. Specifically, the acquired images are first resized to 512×512 pixels. The images are then manually annotated using the labelme tool to identify areas of change. These changed areas are then binarized to generate binary label images. Unchanged areas have a pixel value of 0, while changed areas have a pixel value of 255. Data augmentation is then performed to obtain the change detection dataset, which is divided into training, validation, and test sets in a 7:2:1 ratio. Image manipulation techniques such as jitter blurring and illumination change adjustments are applied to the image pairs in the change detection dataset to increase data complexity. Each data sample in the object detection dataset consists of an unchanged image from the same viewpoint, a changed image, and binary label images before and after the change. The locations of changes in the labeled areas are marked in the binary label images. Figure 2 As shown.

[0030] S3: Construct a debris flow change detection model Light-ChangeCNNformer based on the Changeformer model. Train the debris flow change detection model with the training set divided in step S2. Input the validation set divided in step S2 into the trained Light-ChangeCNNformer model and output the predicted scene change binary map to select the detection model Light-ChangeCNNformer with the optimal parameters. S4: Build a change detection and classification network MobileNetV3 based on MobileNet network. Use the binarized labeled images in the training set divided in step S2 and their corresponding debris flow and non-debris flow categories to train it. Input the predicted scene change binary map output in S3 into the trained MobileNetV3 to determine whether the change in the scene is a debris flow and select the MobileNetV3 with the optimal parameters. S5: Input the test set from step S2 into the detection models Light-ChangeCNNformer and MobileNetV3 with optimal parameters selected in steps S3 and S4 for testing, evaluate the feasibility of the model output results, detect the real debris flow scene captured by the camera, and verify the feasibility of the method.

[0031] Specifically, the method is tested using test images to evaluate its feasibility. Real-time video footage of actual debris flow scenes is input into the debris flow change detection framework to identify and classify scene changes, thereby enabling the detection of debris flow occurrences.

[0032] Example 2 In this embodiment, the debris flow change detection network model is the Light-ChangeCNNformer. The Light-ChangeCNNformer network includes a Transformer encoder embedded with a CNN and a lightweight decoder, such as... Figure 3 As shown.

[0033] Specifically, the CNN includes multiple convolutional layers, batch normalization (BN) layers, and activation functions connected in sequence, and is embedded in a transformer structure through patch-embedding. The transformer structure includes several stacked transformer layers. The lightweight decoder optimizes the parameters based on the transformer decoder to reduce the number of model parameters, making it adaptable to different types of devices, thus forming the decoder of this application after parameter tuning for mudslide scenarios.

[0034] Specifically, the construction of the Transformer encoder embedded in the CNN is as follows: Preprocessing dual-phase input: Configure dual input channels to receive scene images before and after the changes in phases T1 and T2 respectively, with a size of 512×512×3. Use shared weights to perform preprocessing, normalize the image pixel values ​​to [-1,1], and use Gaussian filtering to remove noise and eliminate the difference in radiation between time and image. Constructing a CNN feature extraction branch: Perform CNN feature extraction on the preprocessed image data to obtain single-phase feature maps. And single-phase feature maps Perform secondary CNN feature extraction to output single-temporal feature maps. ; Specifically, the encoder CNN structure consists of two parts: the first part is a 7×7 convolution with an output size of 64×64×64; the second part is a 3×3 convolution with an output size of 32×32×128. For the input image data, a 7×7 convolutional kernel with a stride of 4 is used to achieve 4x downsampling at high resolution. Two depthwise separable convolutional blocks with BN layers and ReLU activation functions are used to limit the range of activation values ​​and improve the stability of quantization deployment. The formula is as follows: (1) in This indicates that the ReLU activation function is used to limit the numerical range. BatchNorm is used to standardize the data. Represents a 7×7 convolution. For the input image, and These represent the length and width of the image, respectively. This represents the first layer of features extracted from the input using CNN. R represents the feature map resolution, 64 is the number of channels, H / 4 means the height is 1 / 4 of the original image, and W / 4 means the width is 1 / 4 of the original image. In the medium-resolution stage, a 3×3 convolution kernel with a stride of 2 is used to achieve secondary downsampling, as shown in the following formula: (2) in, Indicates to The second layer of features extracted by CNN; Constructing the Transformer structure: The Transformer structure is divided into two parts. The first part consists of PatchEmbedding and multiple Transformer layers. The second part is based on the first part, with the addition of normalization and convolution. Specifically, the first part consists of Patch Embedding and six Transformer layers, with an output size of 16×16×256; the second part is based on the first part, with the addition of normalization and 3×3 convolution, and an output size of 8×8×512. Patch Embedding is used to output single-phase feature maps. Importing this into the Transformer network converts the 2D feature maps output by the CNN into a continuous 1D sequence. Simultaneously, the 2D feature maps are sliced ​​with a 50% overlap to preserve local continuity, resulting in the output of the transition layer. The calculation formula is as follows: (3) (4) in, Indicates after processing This is also the first input for the six Transformers, d=256. For learnable location encoding; is the initial input to the six Transformer layers, and is the feature when entering the Transformer layer for the first time. This indicates that during computation in the current layer, the input features of the previous layer are used; in other words, the first Transformer layer is a... The subsequent Transformer layers process the output of the previous Transformer layer.

[0035] The output of the transition layer is normalized and a multi-head attention mechanism is applied to stabilize the training process and prevent gradient explosion or gradient vanishing. The calculation formula is as follows: (5) (6) in This represents the output of the previous layer, with the initial input being... , The result after normalization. () represents the normalization process. MHSA() is a multi-head self-attention mechanism, composed of multiple spatial reduction attention heads (SRA) connected in parallel. This represents the result of the transition layer output after normalization and multi-head attention, and it is also the input to the second part of the Transformer network. The features are obtained after passing through 6 Transformer layers. ; feature After the second part of Transformer structural feature processing, the output features are... .

[0036] Specifically, Multi-Head Self-Attention (MHSA) consists of multiple Spatial Reduction Attention (SRA) heads connected in parallel. The core mechanism of multi-head attention SRA includes query-key-value calculation, attention weight calculation, and output projection; Normalized input features The QKV values ​​are calculated using the projection matrix, and the calculation formula is as follows: (7) (8) (9) Where Z represents the characteristics of all input MHSA modules, , , These represent the query projection matrix, key projection matrix, and value projection matrix, respectively, mapping the input to the corresponding spaces. Indicates the input feature sequence. For spatial location number, For feature dimensions; The feature dimension of the value vector. This represents the feature dimension of the key vector.

[0037] Using the calculated Q, K, and V in combination The attention weights are calculated using the following formula: (10) in This is a scaling factor to prevent gradient vanishing. For relative encoding positions, T represents matrix transpose and is responsible for dimension matching; Finally, the obtained attention calculation results are concatenated using multi-head attention, and the calculation formula is as follows: (11) in, This represents h attention heads, which are concatenated and projected. The total attention head after splicing is called multi-head attention (MHSA).

[0038] After the acquired attention heads are concatenated, a multi-head attention projection output matrix is ​​generated. Merging all features represents multi-head attention; Specifically, the lightweight decoder employs multi-scale feature fusion, upsamples and splices the difference features between CNN and Transformer at each level, and outputs a segmentation map using 3×3 convolution to 1×1 convolution. It also reduces the number of model parameters through channel clipping and structure replacement.

[0039] Multi-scale feature fusion: combining two layers of features output by a CNN , The two-layer features output by the Transformer , Perform dual-temporal feature differencing to obtain the feature differences corresponding to different time periods. The calculation formula is as follows: (12) in, This indicates the feature hierarchy; there are four feature layers in the network. , , , These represent the corresponding layer output features for the two inputs. This indicates the difference between the two outputs.

[0040] Upsample high-level features by a factor of 2: ,in For subpixel convolution, its mathematical expression is: (13) Where F represents the upsampling feature of the decoder. This refers to tensor reorganization, which rearranges the channel dimensions into the spatial dimensions to improve resolution. The upsampling factor is used; the upsampling result is then concatenated with the mid-layer Transformer features to achieve cross-scale feature fusion, and dimensionality reduction is performed using 1×1 convolution. The calculation formula is as follows: (14) in, This represents the feature obtained by upsampling the features from the fourth layer and fusing them with the features from the third layer.

[0041] Progressive upsampling decoding: This method uses upsampling to convert low-resolution features into high-resolution features. It consists of three stages, and the calculation formula is as follows: (15) (16) (17) in, This represents the feature difference value calculated for the first time in Formula 3. This represents the feature map upsampled by the first bilinear interpolation, with a resolution higher than... Double it, then... Perform bilinear interpolation upsampling, at which point the sampled dimension is relatively... It has been improved fourfold, with the same resolution. Same as, with By concatenating the results of the proj channel alignment, a second-stage upsampled and fused feature map is obtained. And then Subpixel convolution is performed to obtain the feature map output by the decoder. .

[0042] in It includes bilinear interpolation upsampling, 3×3 depthwise separable convolution and ReLU activation function, and Proj is a 1×1 convolution responsible for aligning channels; The network detection head construction mainly consists of three parts: input processing, change prediction, and loss function; input processing involves converting the feature map output by the decoder... Features are enhanced by compressing channels using 1×1 convolutions and increasing the depth of 3×3 separable convolutions, resulting in an 8-channel feature output. Change prediction is achieved through 1×1 convolution and Processing features Obtain a 2-channel probability map After binarization, a binary map of the predicted scene changes is generated, used to distinguish between changed and unchanged areas; the loss function used is Dice loss, calculated as follows: (18) in A binary matrix representing the true labels, where a value of 1 indicates a changed region and a value of 0 indicates an unchanged region; The loss function evaluates the degree of fit between the model's predictions and the true labels, that is, the accuracy of the model during training. However, the loss function is generally used in network training to provide feedback to the network and optimize the network parameters.

[0043] Model training and inference: The training set from the change detection dataset is input into the constructed Light-ChangeCNNformer network for training, and the model performance is measured using the test set; Example 3 In this embodiment, the debris flow change classification network is MobileNetV3, which is a lightweight deep convolutional neural network architecture. By introducing depthwise separable convolution and efficient activation functions such as H-Swish, it significantly reduces the computational complexity and number of parameters of the model while maintaining high classification accuracy.

[0044] The core of MobileNetV3 is depthwise separable convolution, which decomposes standard convolution into two independent steps: depthwise convolution and pointwise convolution. Depthwise convolution performs a convolution operation on each input channel individually, while pointwise convolution performs channel blending on the output of the depthwise convolution through 1×1 convolutions. This decomposition significantly reduces computational cost. Assume the input feature map has a dimension of... The dimension of the output feature map is The computational cost of standard convolution is ,in It is the size of the convolution kernel, while the computational cost of depthwise separable convolution is... This significantly reduces computational complexity. Furthermore, MobileNetV3 employs the H-Swish activation function, defined as: (1) Where f is the feature value of the input activation function. While maintaining nonlinearity, the function can better handle small input values, thereby improving the model's expressive power. MobileNetV3 also introduces the Squeeze-and-Excitation (SE) module to enhance feature expressiveness. The SE module compresses the feature map into a channel-dimensional vector through Global Average Pooling, then learns the dependencies between channels through two fully connected layers (one for dimensionality reduction and one for dimensionality increase), generates channel weights through the Sigmoid function, and finally multiplies the weights with the original feature map to recalibrate the features. The calculation process of the SE module can be represented as follows: (2) in It is the input feature map. This indicates element-wise multiplication. It is the sigmoid function. and These are the weights of the fully connected layer, and GAP is the global average pooling operation. MobileNetV3 maintains high computational efficiency while exhibiting high accuracy and robustness in debris flow change classification, making it particularly suitable for resource-constrained real-time monitoring scenarios.

[0045] Example 4 In this embodiment, the test set from the collected data is fed into the detection model with optimal parameters in S3 and S4 for testing, and the feasibility of the model output is evaluated. Optimal parameters indicate that, as training progresses, the model generated in each training round achieves the best precision, recall, and F-score on the validation set, and outputs the final detection result, which is then evaluated for feasibility. This invention selects three commonly used evaluation metrics for the Light-ChangeCNNformer, including precision P1, recall R, and F-score; precision P2 is selected for the MobileNetV3 network; precision P3 is selected for defining the final joint output of the two models; and the algorithm's inference speed (FPS) is also considered. The definitions of these metrics are as follows:

[0046]

[0047]

[0048] Wherein, TP represents the number of positive examples identified as positive in the positive class samples, and in this method, it represents the number of samples correctly identified as debris flow in the debris flow samples; FP represents the number of positive examples identified as negative in the negative class samples, and in this method, it represents the number of samples identified as debris flow in the non-debris flow samples; FN represents the number of negative examples identified as positive in the positive class samples, and in this method, it represents the number of samples that were not correctly identified as debris flow in the debris flow samples.

[0049] To verify the effectiveness and feasibility of the improved model, the test set was put into the model. The experimental environment is shown in Table 1, and the experimental results are shown in Table 2.

[0050] Table 1 Experimental Environment Setup

[0051] Table 2 Feasibility Assessment Results

[0052] Example 5 In this embodiment, the debris flow video acquisition device consists of two cameras: a visible light camera and a thermal infrared camera. The visible light camera is used to capture debris flow scene videos when there is sufficient daylight, while the thermal infrared camera is used to capture debris flow scene videos when there is less light at night. In practical applications, the scene videos captured by the cameras half an hour ago are saved in the background, and the current debris flow video image is compared with the video image half an hour ago for change detection. The binary image output by the change detection is then input into a classification network to determine whether a debris flow has occurred. Considering the rapid changes in light intensity in the early morning and evening, and to ensure data continuity, this invention turns on the visible light camera at 5:30 AM but turns off the thermal infrared camera at 7:00 AM, turns on the thermal infrared camera at 6:00 PM, and turns off the visible light camera at 7:30 PM. To further verify the feasibility of this invention for debris flow detection, the invention was integrated into a camera device to perform real-time monitoring of debris flow scenes, such as... Figure 4 As shown.

[0053] Any aspects not covered in this invention are applicable to existing technologies.

Claims

1. A debris flow monitoring method based on scene changes, characterized in that, Includes the following steps: S1. Extract frames from the video of the debris flow to obtain a comparison image of the scene before and after the debris flow, and use image generation technology to increase the data capacity. S2. Preprocess the scene change comparison image caused by non-debris flow and the scene comparison image before and after debris flow obtained in step S1 to generate a binarized label image, obtain the change detection dataset, and divide it into training set, validation set and test set. Each data sample in the change detection dataset consists of a scene image before the change, a scene image after the change and the corresponding binarized label image, as well as the debris flow and non-debris flow categories corresponding to the binarized label image. S3. Construct a debris flow change detection model Light-ChangeCNNformer based on the Changeformer model. Train the debris flow change detection model with the training set divided in step S2. Input the validation set divided in step S2 into the trained Light-ChangeCNNformer model and output the predicted scene change binary map to select the detection model Light-ChangeCNNformer with the optimal parameters. S4. Build a change detection and classification network MobileNetV3 based on the MobileNet network. Use the binarized labeled images in the training set divided in step S2 and their corresponding debris flow and non-debris flow categories to train it. Input the predicted scene change binary map output in S3 into the trained MobileNetV3 to determine whether the change in the scene is a debris flow and select the MobileNetV3 model with the optimal parameters. S5. Input the test set from step S2 into the detection models Light-ChangeCNNformer and MobileNetV3 with optimal parameters selected in steps S3 and S4 for testing, evaluate the feasibility of the model output results, detect the real debris flow scene captured by the camera, and verify the feasibility of the method.

2. The debris flow monitoring method according to claim 1, characterized in that, The preprocessing described in step S2 includes: labeling the change areas of the scene change comparison map not caused by debris flow and the scene change comparison map before and after the debris flow obtained in step S1, obtaining the location of the change, performing binarization processing on the location of the change, generating a binarized label image, and then performing data augmentation to obtain a change detection dataset.

3. The debris flow monitoring method according to claim 1, characterized in that, The debris flow change detection model Light-ChangeCNNformer described in step S3 includes a Transformer encoder embedded with a CNN and a lightweight decoder. The CNN includes multiple convolutional layers, BN layers and activation functions connected in sequence, and is embedded in the transformer structure through patch-embedding. The Transformer structure includes several stacked Transformer layers. The lightweight decoder optimizes the parameters based on the changeformer decoder to reduce the number of model parameters.

4. The debris flow monitoring method according to claim 3, characterized in that, The construction of the Transformer encoder embedded in the CNN is as follows: The system is configured with dual input channels to receive scene images before and after the changes in time phases T1 and T2, respectively, and preprocessing is performed using shared weights. Constructing a CNN feature extraction branch: Perform CNN feature extraction on the preprocessed image data to obtain single-phase feature maps. And single-phase feature maps Perform secondary CNN feature extraction to output single-temporal feature maps. ; Constructing the Transformer structure: The Transformer structure is divided into two parts. The first part consists of PatchEmbedding and multiple Transformer layers. The second part is based on the first part, with the addition of normalization and convolution. Patch Embedding is used to output single-phase feature maps. Importing this into the Transformer network, the 2D feature map output from the CNN feature extraction branch is converted into a 1D continuous sequence. Simultaneously, the 2D feature map is sliced ​​to preserve local continuity, resulting in the output of the transition layer. The formula is as follows: （1） （2） in, Indicates after processing This is also the first input to the Transformer, d=256. For learnable positional encoding, and These represent the length and width of the image, respectively. This refers to the number of spatial locations; The output of the transition layer is normalized and a multi-head attention mechanism is applied to stabilize the training process and prevent gradient explosion or gradient vanishing. The calculation formula is as follows: （3） （4） in, This represents the output of the previous layer, with the initial input being... , The result after normalization () represents the normalization process. MHSA() is a multi-head self-attention mechanism, composed of multiple spatial reduction attention heads (SRA) connected in parallel. This represents the result of the transition layer output after normalization and multi-head attention, and it is also the input to the second part of the Transformer network. The features are obtained after passing through multiple Transformer layers. ; feature After the second part of Transformer structural feature processing, the output features are... .

5. The debris flow monitoring method according to claim 4, characterized in that, The multi-head attention mechanism consists of multiple spatially compressed attention heads connected in parallel, including query-key-value calculation, attention weight calculation, and output projection.

6. The debris flow monitoring method according to claim 3, characterized in that, The lightweight decoder is constructed as follows: Multi-scale feature fusion: combining the two layers of features output from the feature extraction branch of a CNN. , The two-layer features output by the Transformer structure , Perform dual-temporal feature differencing to obtain the feature differences corresponding to different time periods. The calculation formula is as follows: （5） in, This indicates the feature hierarchy; there are four feature layers in the network. , , , These represent the corresponding layer output features for the two inputs. This indicates the difference between the two outputs; High-level characteristics Perform 2x upsampling: ,in For subpixel convolution, its mathematical expression is: （6） Where R represents the feature map resolution, and F represents the upsampled features of the decoder. This refers to tensor reorganization, which rearranges the channel dimensions into the spatial dimensions to improve resolution. The upsampling factor; Then, the upsampling results are compared with the mid-layer Transformer features. The data is concatenated to achieve cross-scale feature fusion, and 1×1 convolution is used for dimensionality reduction. The calculation formula is as follows: （7） in, This represents the feature obtained by fusing the fourth layer features with the third layer features after upsampling. Progressive upsampling decoding: This method uses upsampling to convert low-resolution features into high-resolution features. It consists of three stages, and the calculation formula is as follows: （8） （9） （10） in, This represents the characteristic difference value calculated for the first time in formula (5). This represents the feature map from the first bilinear interpolation upsampling. This represents the feature map after upsampling and fusion in the second stage. The feature map output by the decoder; in It includes bilinear interpolation upsampling, 3×3 depthwise separable convolution and ReLU activation function, and Proj is a 1×1 convolution responsible for aligning channels; The construction of the network detection head mainly consists of three parts: input processing, change prediction, and loss function; input processing involves converting the feature map output by the decoder... Features are enhanced by compressing channels using 1×1 convolutions and increasing the depth of 3×3 separable convolutions, resulting in an 8-channel feature output. Change prediction is achieved through 1×1 convolution and Processing features Obtain a 2-channel probability map Binarization based on P yields a predicted scene change binary map, used to distinguish between changed and unchanged regions; the loss function used is Dice loss, calculated as follows: （11） in A binary matrix representing the true labels, where a value of 1 indicates a changed region and a value of 0 indicates an unchanged region.

7. The debris flow monitoring method according to claim 1, characterized in that, The variation classification network MobileNetV3 described in step S4 introduces depthwise separable convolution and activation functions. The depthwise separable convolution decomposes the standard convolution into two independent steps: depthwise convolution and pointwise convolution. The depthwise convolution performs convolution operation on each input channel individually, while the pointwise convolution mixes the channels of the output of the depthwise convolution through 1×1 convolution.

8. The debris flow monitoring method according to claim 7, characterized in that, The change classification network described in step S4 uses the H-Swish activation function, which is defined as: （12） Where f is the feature value of the input activation function.

9. The debris flow monitoring method according to claim 1, characterized in that, MobileNetV3 also introduces the Squeeze-and-Excitation (SE) module. The SE module compresses the feature map into a channel-dimensional vector using global average pooling, then learns the dependencies between channels through two fully connected layers, generates channel weights using the sigmoid function, and finally multiplies the weights with the original feature map to recalibrate the features. The calculation process of the SE module can be represented as follows: （13） in It is the input feature map. This indicates element-wise multiplication. It is the sigmoid function. and These are the weights of the fully connected layer, and GAP is the global average pooling operation.

10. The debris flow monitoring method according to claim 1, characterized in that, The optimal parameters mentioned in steps S3 and S4 represent the optimal accuracy, recall, and F-value of the model generated in each round of training on the validation set as training progresses.

Images