Industrial smoke video target segmentation method based on motion saliency and deformation perception

Abstract

This invention discloses a method for segmenting industrial dust video targets based on motion saliency and deformation perception. The method utilizes a dual-stream weight-shared U-Net backbone network, taking two adjacent RGB frames as input. A saliency-deformation gating fusion module is integrated at each stage of the encoder to jointly utilize temporal and spatial information for accurate segmentation of industrial dust video. A multi-view motion saliency module calculates inter-frame feature differences, which are then fused using an adaptive weighted Gaussian mapping to generate robust motion saliency priors. A deformation edge perception network enables the network to have positional awareness, accurately modeling the non-rigid deformation boundaries of industrial dust. The saliency-deformation gating fusion module provides discriminative spatiotemporal features for the decoder. This invention solves the problems of insufficient detection capability for semi-transparent and weakly boundary dust and the inability to effectively model non-rigid deformation in traditional methods, significantly improving the accuracy and robustness of industrial dust video segmentation while maintaining real-time inference speed.

Description

Technical Field

[0001] This invention relates to the fields of image processing and computer vision technology, specifically to a method for segmenting industrial smoke and dust video targets based on motion saliency and deformation perception. Background Technology

[0002] Industrial smoke and dust emissions are a major source of air pollution. To achieve automated assessment and monitoring of pollution levels, existing industrial monitoring systems widely employ video segmentation technology based on Ringelmann blackness maps to extract smoke and dust regions from dynamic backgrounds. Existing smoke and dust segmentation methods are mainly divided into traditional methods and deep learning-based methods. Traditional static methods typically utilize thresholding, texture analysis, or region growing techniques to process single-frame images. In video segmentation tasks, optical flow or inter-frame difference methods are mainly used to capture temporal changes. However, these methods based on handcrafted features rely on the specific physical property differences between smoke and dust and the background. In real-world industrial scenarios with drastic changes in lighting, interference from moving clouds, or complex factory structures, they exhibit poor robustness and struggle to accurately distinguish smoke and dust from similarly appearing interfering objects.

[0003] Existing dynamic video segmentation methods still have significant drawbacks in practical industrial applications: First, their motion feature extraction methods are simplistic and sensitive. Most methods rely solely on a single motion descriptor. This single feature extraction approach is extremely sensitive to semi-transparent smoke boundaries and background fluctuations, and lacks sufficient sensitivity to thin smoke that diffuses slowly in the early stages, easily leading to mismatches of features. Second, their spatiotemporal feature fusion strategies are simple and lack adaptability. Many methods fuse temporal and spatial features through simple concatenation or element-wise addition, lacking explicit utilization of multi-view motion statistics and geometric deformation priors, resulting in false detections and fragmented segmentation masks in complex backgrounds. Therefore, to address these issues, this paper proposes an industrial smoke video target segmentation method based on motion saliency and deformation perception. Summary of the Invention

[0004] The purpose of this invention is to address the problems of poor robustness of motion feature extraction, high cost of non-rigid boundary modeling, and insufficient spatiotemporal feature fusion in existing industrial dust video segmentation technologies, and to provide an industrial dust video target segmentation method based on motion saliency and deformation perception.

[0005] To achieve the above-mentioned technical effects, the present invention is implemented through the following technical solution: a method for segmenting industrial smoke and dust video targets based on motion saliency and deformation perception, characterized by comprising the following steps:

[0006] S1. Video Data Acquisition and Preprocessing: Acquire videos of industrial dust emissions from real industrial sites, and preprocess the videos to obtain consecutive video frames and corresponding tags. The current frame is identified by the RGB images of two adjacent frames. And the previous frame As network input, multi-scale features are extracted by a dual-stream weight-sharing U-Net encoder;

[0007] S2. Motion Saliency Calculation: Then, at different encoder levels, the Multi-View Motion Saliency Module (MVSM) calculates feature differences at different scales from three perspectives: Euclidean distance, cosine distance, and statistical moment distance. These differences are then fused using adaptive weighted Gaussian mapping to generate a motion saliency prior map. ;

[0008] S3. Deformation Boundary Modeling: The Deformation Edge Awareness Network DEPNet utilizes geometrically embedded separable convolutions (GESConv) to inject normalized spatial coordinate priors to predict non-rigid deformation intensity maps. Accurately model the boundary of smoke and dust diffusion;

[0009] S4. Saliency-Deformation Gated Fusion: The saliency-deformation gated fusion module SDGF adopts a cascaded spatial-channel gating mechanism to adaptively fuse motion saliency features and deformation geometric features, and outputs discriminative spatiotemporal fusion features for use by the decoder via skip connections;

[0010] S5. Decoding Output and End-to-End Optimization: The decoder fuses the bottleneck layer features of the current frame with the fused features of the SDGF outputs from each encoder stage, and restores spatial resolution by upsampling at each stage through skip connections; combined with segmentation loss. Time consistency loss and boundary loss The multi-task joint loss function is used for end-to-end optimization, and the final smoke and dust segmentation mask is output.

[0011] Furthermore, in S1, video of industrial dust emissions from a real industrial site is acquired and preprocessed to obtain consecutive video frames and corresponding tags, using the RGB images of two adjacent frames as the current frame. With the previous frame As network input, multi-scale features are extracted by a dual-stream weight-sharing U-Net encoder. The specific steps are as follows:

[0012] S1.1 Collect real industrial site smoke and dust emission videos. The video sources cover different emission areas of multiple industrial plants, covering a variety of typical and complex industrial scenarios such as thin smoke, multi-target smoke and dust, cloud and fog interference, low light, strong light and colored smoke and dust, in order to ensure the diversity of the dataset and the comprehensiveness of the scenario coverage.

[0013] S1.2 Extract the acquired raw video into an ordered and continuous sequence of RGB static image frames according to a certain inter-frame interval. All frames are uniformly scaled to standard resolution. To meet network input size requirements;

[0014] S1.3. Manually annotate the smoke and dust regions in each frame of the image at the pixel level to generate the corresponding binary segmentation label mask. The pixel values ​​of the smoke and dust areas are labeled as white (255) and the pixel values ​​of the background areas are labeled as black (0), thus constructing a complete dataset with one-to-one correspondence between image frames and labels. At the same time, the complete data is divided into training set, validation set and test set according to different industrial scenario domains to ensure that the three come from different scenarios, so as to rigorously evaluate the model's generalization ability to unknown scenarios.

[0015] S1.4. The network input pair is formed by two adjacent RGB images on the video timeline, i.e., the current frame. With the previous frame As network input, two frames of images are simultaneously input as a set of sample pairs to the two parallel branches of the dual-stream U-Net encoder, with the corresponding supervision label being the segmentation mask of the current frame. ;

[0016] S1.5 The dual-stream U-Net encoder consists of two completely symmetrical parallel coding branches, which share all convolutional layer weight parameters. Each branch contains four coding stages and four downsampling stages. Each coding stage consists of two SimpleConvBlocks stacked together, progressively extracting features and downsampling spatially from the input frame. The current frame... The encoder outputs four different scale features. Previous frame The same shared weight encoder outputs four feature maps at different scales. The weight sharing mechanism ensures that the feature extraction standards of the two frames are strictly consistent and significantly reduces the number of model parameters, effectively suppressing overfitting. The four feature pairs extracted at different scales are output to the subsequent SDGF modules at each scale for spatiotemporal motion feature fusion.

[0017] Furthermore, in S2, at different encoder levels, the Multi-View Motion Saliency Module (MVSM) calculates feature differences at different scales from three perspectives: Euclidean distance, cosine distance, and statistical moment distance. These differences are then fused using an adaptive weighted Gaussian mapping to generate a motion saliency prior map. The specific steps are as follows:

[0018] S2.1, at the nth (n=1,2,3,4) encoder level, using the features of the current frame... Features from the previous frame As input to the MVSM module, inter-frame feature differences are calculated from three complementary perspectives: intensity difference, texture difference, and statistical distribution difference.

[0019] S2.2. Capture the absolute changes in smoke density and brightness from the perspective of Euclidean distance, and calculate the feature maps of the two frames pixel by pixel across all channels. Distance, resulting in intensity difference The formula for calculating the strength difference is as follows:

[0020]

[0021] In the formula, and These represent the feature values ​​of the current frame and the previous frame at the c-th channel spatial position, respectively, where C is the total number of channels; this perspective has a strong response to local intensity changes during the appearance, accumulation, and dissipation of smoke and dust areas;

[0022] S2.3. Decouple the evolution of smoke and dust texture from illumination interference from the perspective of cosine distance. Calculate the cosine similarity between the feature vectors of two frames pixel by pixel and convert it into cosine distance to obtain the texture difference. The formula for calculating texture differences is as follows:

[0023]

[0024] In the formula, express Norm, It is a numerical stability constant to prevent division by zero; the cosine distance only measures the change in the direction of the feature vector and is not sensitive to the magnitude of the vector. It is naturally invariant to illumination and can effectively capture the structural texture evolution inside the smoke and dust clusters.

[0025] S2.4. Characterizing the non-rigid diffusion pattern of smoke and dust from the perspective of statistical moment distance, centered on the spatial location (h,w). Within a local window, calculate the local first-order moments of the features of the current frame and the previous frame. With second moment The formula is as follows:

[0026]

[0027]

[0028] Based on this, the statistical moment distance is calculated by comparing the temporal changes of local statistics in adjacent frames. The formula is as follows:

[0029]

[0030] In the formula, and These are the inter-frame variations of the local mean and standard deviation of the c-th channel, respectively. The mean term reflects local intensity changes, while the standard deviation term reflects changes in texture and contrast patterns. From a statistical perspective, this comprehensively characterizes the state transition process of the plume while smoothing out local disturbances.

[0031] S2.5, Differences in characteristics from three perspectives , , Perform adaptive weighted Gaussian mapping fusion for each distance metric. After nonlinear transformation using a learnable Gaussian kernel function, weighted summation is performed to generate a motion saliency prior map. The formula is as follows:

[0032]

[0033] In the formula, the weights satisfy Gaussian scaling parameters can be learned Normalized using the softmax function, the initial values ​​were set to [0.5, 1.0, 1.5] via grid search, corresponding to fine-grained, mesoscale, and coarse-grained change detection, respectively; weights Uniform initialization And through backpropagation adaptive learning; this fusion mechanism ensures that the corresponding region is identified as a high-confidence smoke and dust change region only when multiple metrics simultaneously detect significant differences. The output is fed into the subsequent SDGF module as the motion prior input for spatial saliency gating.

[0034] Furthermore, in S3, the deformation of non-rigid smoke and dust is modeled using the geometrically embedded separable convolutional network GESConv through the deformation edge-aware network DEPNet, and the deformation intensity map is output. The specific steps are as follows:

[0035] S3.1 Construct a normalized coordinate grid G. For any position within the spatial dimensions, normalize the coordinates to... The calculation formula is as follows:

[0036]

[0037] In the formula, Indicates spatial dimension, Represents any position in the spatial dimension. This indicates coordinate transpose.

[0038] Normalization ensures that the coordinate magnitude is consistent with the scale of pixel feature changes, preventing gradient instability during training.

[0039] S3.2, Set the current frame features Features of the previous frame The composite input tensor is obtained by concatenating the coordinate grid G ​​along the channel dimension. This allows the network to utilize both temporal evolution information and spatial location priors simultaneously;

[0040] S3.3, Define the GESConv operation: Let the input features be... The intermediate feature map U is extracted through depthwise separable convolution, and then the coordinate prior is injected through a lightweight residual branch to obtain the final output of GESConv. The formula is as follows:

[0041]

[0042] In the formula, for A convolutional kernel of size is used to project the coordinate information G onto the same channel dimension as U. For batch normalization, The ReLU activation function is used; this residual design allows the main path to retain the original semantic information, and the coordinate prior is used as a supplementary enhancement without interfering with the semantic features. Compared with CoordConv, which directly concatenates the coordinate channels, it has a lower computational cost.

[0043] S3.4. Through the above operations, the composite input Z undergoes hierarchical feature learning through two stacked GESConv layers, and then through two... Prediction based on convolution End-to-end output deformation intensity diagram The formula is as follows:

[0044]

[0045] In the formula, It is a feature mapping of the backbone network. It is a prediction head used to output deformation intensity maps. It is the sum of the features of the current frame, the features of the previous frame, and the normalized coordinates. The composite tensor.

[0046] Deformation intensity diagram The non-rigid deformation degree at each spatial location is quantized pixel by pixel, providing accurate dynamic boundary guidance information for the subsequent SDGF module. By explicitly embedding spatial coordinate priors, DEPNet breaks the spatial isotropy of standard convolution, enabling the network to have position awareness and distinguish the semantic features of smoke and dust at different diffusion stages, thereby achieving accurate non-rigid boundary modeling of diffuse dynamic smoke and dust in industrial scenarios.

[0047] Furthermore, in S4, the motion saliency prior map output by the MVSM is adaptively fused through the saliency-deformation gating fusion module SDGF using a cascaded spatial-channel dual-gating mechanism. Deformation intensity map compared with DEPNet output The final fused features are generated through the following steps:

[0048] S4.1, Motion saliency diagram With deformation intensity diagram spliced ​​along the channel dimension, sequentially through Convolutional fusion of local context, ReLU nonlinear activation, Convolution is compressed to a single channel, then normalized using the Sigmoid function to generate a spatial weight mask. ;Will Element-wise operation on the current frame coding features Spatial augmentation features are obtained. The expression is as follows:

[0049] ;

[0050] This operation suppresses static background noise and focuses the characteristic response on the smoke and dust diffusion area where motion and deformation are significant;

[0051] S4.2 Next, spatial enhancement features With deformation characteristics After being compressed into compact channel descriptors through global average pooling, the channels are concatenated, and then the cross-channel nonlinear dependencies are modeled through two layers of fully connected MLP. Finally, the channel weight vector is obtained through the Sigmoid function. Adaptively increase the weights of task-sensitive channels related to smoke texture, density, and boundary sharpness, and suppress redundant channel responses;

[0052] S4.3 Finally, residual connections are used to superimpose the channel recalibration results onto the original current frame features to obtain the final fused output features. The expression is as follows:

[0053]

[0054] This residual design preserves original semantic information while introducing deformation-aware attention enhancement, preventing information degradation in deep networks; the resulting discriminative spatiotemporal fusion features It is passed to the decoder via a skip connection for subsequent upsampling and spatial resolution recovery.

[0055] Furthermore, in S5, the specific steps of the decoder and the multi-task joint loss function are as follows:

[0056] S5.1 The decoder first receives the high semantic and low resolution features output from the encoder bottleneck layer as initial input. In the first upsampling stage, the bottleneck layer features are upsampled and then concatenated with the SDGF fusion output features corresponding to the fourth stage of the encoder along the channel dimension. The fused semantic features are then extracted through a convolutional block. In the second to fourth upsampling stages, the above operation is repeated sequentially, and the fused features are concatenated and convolved with the SDGF fusion output features corresponding to the third, second, and first stages of the encoder, respectively, to gradually supplement the spatial detail information lost by downsampling. This allows the decoder to restore resolution while taking into account both high-level semantics and low-level edge textures. After four upsampling stages, the feature map resolution is restored to the input image size, and finally, the image is processed. Convolution compresses the number of channels to a single channel and outputs a smoke segmentation mask probability map. Since the skip connections at each stage introduce spatiotemporal discriminative features fused by SDGF rather than the original encoded features, the decoder can simultaneously acquire motion saliency and deformation geometric information at each stage, thereby effectively improving the segmentation accuracy of thin smoke, small targets and semi-transparent smoke edges.

[0057] S5.2, Use a combination of binary cross-entropy loss and Dice loss as the segmentation loss. The calculation formula is as follows:

[0058] Binary cross-entropy loss :

[0059]

[0060] Dice loss :

[0061]

[0062] In the formula, Indicates the true label, Represents the predicted value. This is the smoothing constant.

[0063] The binary cross-entropy loss is used to evaluate the confidence of pixel-by-pixel prediction, providing stable gradient supervision for the label-blurred region at the smoke boundary; the Dice loss measures the intersection-union ratio of the prediction and the ground truth from a global perspective, effectively alleviating the class imbalance problem caused by the smoke region being smaller than the background.

[0064] S5.3, Time Consistency Loss Motion saliency map generated by MVSM As an adaptive pixel-wise weight, a regularization constraint is applied to the absolute difference between the predicted probability maps of adjacent frames. The calculation formula is as follows:

[0065]

[0066] In the formula, and These represent the predicted probability maps for the current frame and the previous frame, respectively. Indicates the image height. Indicates the image width.

[0067] In regions with insignificant motion, the loss weight is larger, imposing a strong penalty on inconsistencies in cross-frame predictions; while in regions with significant motion, the loss weight is smaller, allowing the network to adaptively respond to real changes in smoke and dust morphology. This design effectively suppresses cross-frame prediction jitter in static background regions, while avoiding imposing excessive consistency constraints on smoke and dust movement regions and suppressing real temporal changes, thus enhancing the temporal coherence and stability of the segmentation results.

[0068] S5.4 Boundary Loss A specific optimization was performed to address the problem of blurred and difficult-to-accurately label smoke and dust boundaries. The boundary loss calculation formula is as follows:

[0069]

[0070] In the formula, This represents the set of boundary pixels in the real mask. This indicates its base number, i.e., the number of pixels. Indicates the true label, This represents the predicted value.

[0071] By applying a truth mask The erosion operation of morphological structural elements subtracts the original mask from the erosion result to extract a set of boundary pixels N with a width of about one pixel. During training, additional penalties are applied to the prediction error at the boundary pixels belonging to set N, amplifying the learning gradient of the network in the boundary region, enhancing the network's ability to finely model the blurred edges of semi-transparent smoke and dust, and suppressing the generation of overly smooth or jagged artifacts in the prediction of the boundary region.

[0072] S5.5 The three loss functions are weighted and summed according to their weight coefficients to form the overall multi-task joint training objective. The total training loss is as follows:

[0073]

[0074] In the formula, the weighting coefficients satisfy... Grid search verification settings , , The three loss terms work together to supervise the network training from three dimensions: spatial segmentation accuracy, temporal prediction coherence, and boundary refinement. This achieves end-to-end joint optimization of the dual-stream encoder-decoder network and ultimately outputs a temporally consistent and clearly defined industrial dust segmentation mask.

[0075] The beneficial effects of this invention are:

[0076] This invention employs an MVSM module that integrates three complementary metrics—Euclidean distance, cosine distance, and statistical moment distance—to simultaneously capture changes in intensity, structure, and distribution. This significantly improves sensitivity to weak, thin, and semi-transparent smoke and overcomes the sensitivity to environmental interference caused by single metrics. Simultaneously, the DEPNet module utilizes geometric embedding-separated convolution to introduce spatial location priors, endowing the network with location awareness to distinguish smoke features at different diffusion stages, achieving accurate modeling of non-rigid boundaries with low computational cost. Furthermore, the SDGF module uses a cascaded gating mechanism to dynamically adjust spatial and channel weights based on motion saliency and deformation characteristics, effectively suppressing background noise such as cloud formations and significantly improving the model's anti-interference capability and segmentation accuracy in complex scenes. Attached Figure Description

[0077] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0078] Figure 1 This is a flowchart of the industrial smoke and dust video segmentation method of Embodiment 1 of the present invention;

[0079] Figure 2 This is a diagram illustrating the industrial dust pollution level monitoring technology of Embodiment 1 of the present invention;

[0080] Figure 3 This is a system structure diagram of the industrial smoke and dust video segmentation method of Embodiment 1 of the present invention;

[0081] Figure 4 This is a visualization result of industrial dust video segmentation in Embodiment 1 of the present invention. Detailed Implementation

[0082] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0083] Example 1

[0084] A video target segmentation method for industrial smoke and dust based on motion saliency and deformation perception is characterized by the following steps:

[0085] S1. Video Data Acquisition and Preprocessing: Acquire videos of industrial dust emissions from real industrial sites, and preprocess the videos to obtain consecutive video frames and corresponding tags. The current frame is identified by the RGB images of two adjacent frames. And the previous frame As network input, multi-scale features are extracted by a dual-stream weight-sharing U-Net encoder; the specific steps are as follows:

[0086] S1.1 Collect real industrial site smoke and dust emission videos. The video sources cover different emission areas of multiple industrial plants, covering a variety of typical and complex industrial scenarios such as thin smoke, multi-target smoke and dust, cloud and fog interference, low light, strong light and colored smoke and dust, in order to ensure the diversity of the dataset and the comprehensiveness of the scenario coverage.

[0087] S1.2 Extract the acquired raw video into an ordered and continuous sequence of RGB static image frames according to a certain inter-frame interval. All frames are uniformly scaled to standard resolution. To meet network input size requirements;

[0088] S1.3. Manually annotate the smoke and dust regions in each frame of the image at the pixel level to generate the corresponding binary segmentation label mask. The pixel values ​​of the smoke and dust areas are labeled as white (255) and the pixel values ​​of the background areas are labeled as black (0), thus constructing a complete dataset with one-to-one correspondence between image frames and labels. At the same time, the complete data is divided into training set, validation set and test set according to different industrial scenario domains to ensure that the three come from different scenarios, so as to rigorously evaluate the model's generalization ability to unknown scenarios.

[0089] S1.4. The network input pair is formed by two adjacent RGB images on the video timeline, i.e., the current frame. With the previous frame As network input, two frames of images are simultaneously input as a set of sample pairs to the two parallel branches of the dual-stream U-Net encoder, with the corresponding supervision label being the segmentation mask of the current frame. ;

[0090] S1.5 The dual-stream U-Net encoder consists of two completely symmetrical parallel coding branches, which share all convolutional layer weight parameters. Each branch contains four coding stages and four downsampling stages. Each coding stage consists of two SimpleConvBlocks stacked together, progressively extracting features and downsampling spatially from the input frame. The current frame... The encoder outputs four different scale features. Previous frame The same shared weight encoder outputs four feature maps at different scales. The weight-sharing mechanism ensures that the feature extraction standards of the two frames are strictly consistent and significantly reduces the number of model parameters, effectively suppressing overfitting; the four feature pairs extracted at different scales are output to the subsequent SDGF modules at each scale for spatiotemporal motion feature fusion.

[0091] S2. Motion Saliency Calculation: Then, at different encoder levels, the Multi-View Motion Saliency Module (MVSM) calculates feature differences at different scales from three perspectives: Euclidean distance, cosine distance, and statistical moment distance. These differences are then fused using adaptive weighted Gaussian mapping to generate a motion saliency prior map. The specific steps are as follows:

[0092] S2.1, at the nth (n=1,2,3,4) encoder level, using the features of the current frame... Features from the previous frame As input to the MVSM module, inter-frame feature differences are calculated from three complementary perspectives: intensity difference, texture difference, and statistical distribution difference.

[0093] S2.2. Capture the absolute changes in smoke density and brightness from the perspective of Euclidean distance, and calculate the feature maps of the two frames pixel by pixel across all channels. Distance, resulting in intensity difference The formula for calculating the strength difference is as follows:

[0094]

[0095] In the formula, and These represent the feature values ​​of the current frame and the previous frame at the c-th channel spatial position, respectively, where C is the total number of channels; this perspective has a strong response to local intensity changes during the appearance, accumulation, and dissipation of smoke and dust areas;

[0096] S2.3. Decouple the evolution of smoke and dust texture from illumination interference from the perspective of cosine distance. Calculate the cosine similarity between the feature vectors of two frames pixel by pixel and convert it into cosine distance to obtain the texture difference. The formula for calculating texture differences is as follows:

[0097]

[0098] In the formula, express Norm, It is a numerical stability constant to prevent division by zero; the cosine distance only measures the change in the direction of the feature vector and is not sensitive to the magnitude of the vector. It is naturally invariant to illumination and can effectively capture the structural texture evolution inside the smoke and dust clusters.

[0099] S2.4. Characterizing the non-rigid diffusion pattern of smoke and dust from the perspective of statistical moment distance, centered on the spatial location (h,w). Within a local window, calculate the local first-order moments of the features of the current frame and the previous frame. With second moment The formula is as follows:

[0100]

[0101]

[0102] Based on this, the statistical moment distance is calculated by comparing the temporal changes of local statistics in adjacent frames. The formula is as follows:

[0103]

[0104] In the formula, and These are the inter-frame variations of the local mean and standard deviation of the c-th channel, respectively. The mean term reflects local intensity changes, while the standard deviation term reflects changes in texture and contrast patterns. From a statistical perspective, this comprehensively characterizes the state transition process of the plume while smoothing out local disturbances.

[0105] S2.5, Differences in characteristics from three perspectives , , Perform adaptive weighted Gaussian mapping fusion for each distance metric. After nonlinear transformation using a learnable Gaussian kernel function, weighted summation is performed to generate a motion saliency prior map. The formula is as follows:

[0106]

[0107] In the formula, the weights satisfy Gaussian scaling parameters can be learned Normalized using the softmax function, the initial values ​​were set to [0.5, 1.0, 1.5] via grid search, corresponding to fine-grained, mesoscale, and coarse-grained change detection, respectively; weights Uniform initialization And through backpropagation adaptive learning; this fusion mechanism ensures that the corresponding region is identified as a high-confidence smoke and dust change region only when multiple metrics simultaneously detect significant differences. The output is fed into the subsequent SDGF module as the motion prior input for spatial saliency gating;

[0108] S3. Deformation Boundary Modeling: The Deformation Edge Awareness Network DEPNet utilizes geometrically embedded separable convolutions (GESConv) to inject normalized spatial coordinate priors to predict non-rigid deformation intensity maps. Accurately model the smoke and dust diffusion boundary; the specific steps are as follows:

[0109] S3.1 Construct a normalized coordinate grid G. For any position within the spatial dimensions, normalize the coordinates to... The calculation formula is as follows:

[0110]

[0111] In the formula, Indicates spatial dimension, Represents any position in the spatial dimension. This indicates coordinate transpose.

[0112] Normalization ensures that the coordinate magnitude is consistent with the scale of pixel feature changes, preventing gradient instability during training.

[0113] S3.2, Set the current frame features Features of the previous frame The composite input tensor is obtained by concatenating the coordinate grid G ​​along the channel dimension. This allows the network to utilize both temporal evolution information and spatial location priors simultaneously;

[0114] S3.3, Define the GESConv operation: Let the input features be... The intermediate feature map U is extracted through depthwise separable convolution, and then the coordinate prior is injected through a lightweight residual branch to obtain the final output of GESConv. The formula is as follows:

[0115]

[0116] In the formula, for A convolutional kernel of size is used to project the coordinate information G onto the same channel dimension as U. For batch normalization, The ReLU activation function is used; this residual design allows the main path to retain the original semantic information, and the coordinate prior is used as a supplementary enhancement without interfering with the semantic features. Compared with CoordConv, which directly concatenates the coordinate channels, it has a lower computational cost.

[0117] S3.4. Through the above operations, the composite input Z undergoes hierarchical feature learning through two stacked GESConv layers, and then through two... Prediction based on convolution End-to-end output deformation intensity diagram The formula is as follows:

[0118]

[0119] In the formula, It is a feature mapping of the backbone network. It is a prediction head used to output deformation intensity maps. It is a composite tensor of the current frame features, the previous frame features, and the normalized coordinates G.

[0120] Deformation intensity diagram The non-rigid deformation degree at each spatial location is quantized pixel by pixel, providing accurate dynamic boundary guidance information for the subsequent SDGF module. By explicitly embedding spatial coordinate priors, DEPNet breaks the spatial isotropy of standard convolution, enabling the network to have position awareness and distinguish the semantic features of smoke and dust at different diffusion stages, thereby achieving accurate non-rigid boundary modeling of diffuse dynamic smoke and dust in industrial scenarios.

[0121] S4. Saliency-Deformation Gated Fusion: The saliency-deformation gated fusion module SDGF employs a cascaded spatial-channel gating mechanism to adaptively fuse motion saliency features and deformation geometric features, outputting discriminative spatiotemporal fusion features via skip connections for use by the decoder; the specific steps are as follows:

[0122] S4.1, Motion saliency diagram With deformation intensity diagram spliced ​​along the channel dimension, sequentially through Convolutional fusion of local context, ReLU nonlinear activation, Convolution is compressed to a single channel, then normalized using the Sigmoid function to generate a spatial weight mask. ;Will Element-wise operation on the current frame coding features Spatial augmentation features are obtained. The expression is as follows:

[0123] ;

[0124] This operation suppresses static background noise and focuses the characteristic response on the smoke and dust diffusion area where motion and deformation are significant;

[0125] S4.2 Next, spatial enhancement features With deformation characteristics After being compressed into compact channel descriptors through global average pooling, the channels are concatenated, and then the cross-channel nonlinear dependencies are modeled through two layers of fully connected MLP. Finally, the channel weight vector is obtained through the Sigmoid function. Adaptively increase the weights of task-sensitive channels related to smoke texture, density, and boundary sharpness, and suppress redundant channel responses;

[0126] S4.3 Finally, residual connections are used to superimpose the channel recalibration results onto the original current frame features to obtain the final fused output features. The expression is as follows:

[0127]

[0128] This residual design preserves original semantic information while introducing deformation-aware attention enhancement, preventing information degradation in deep networks; the resulting discriminative spatiotemporal fusion features It is passed to the decoder via a skip connection for subsequent upsampling and spatial resolution recovery.

[0129] S5. Decoding Output and End-to-End Optimization: The decoder fuses the bottleneck layer features of the current frame with the fused features of the SDGF outputs from each encoder stage, and restores spatial resolution by upsampling at each stage through skip connections; combined with segmentation loss. Time consistency loss and boundary loss The multi-task joint loss function is used for end-to-end optimization, and the final smoke and dust segmentation mask is output.

[0130] The specific steps of the decoder and the multi-task joint loss function are as follows:

[0131] S5.1 The decoder first receives the high semantic and low resolution features output from the encoder bottleneck layer as initial input. In the first upsampling stage, the bottleneck layer features are upsampled and then concatenated with the SDGF fusion output features corresponding to the fourth stage of the encoder along the channel dimension. The fused semantic features are then extracted through a convolutional block. In the second to fourth upsampling stages, the above operation is repeated sequentially, and the fused features are concatenated and convolved with the SDGF fusion output features corresponding to the third, second, and first stages of the encoder, respectively, to gradually supplement the spatial detail information lost by downsampling. This allows the decoder to restore resolution while taking into account both high-level semantics and low-level edge textures. After four upsampling stages, the feature map resolution is restored to the input image size, and finally, the image is processed. Convolution compresses the number of channels to a single channel and outputs a smoke segmentation mask probability map. Since the skip connections at each stage introduce spatiotemporal discriminative features fused by SDGF rather than the original encoded features, the decoder can simultaneously acquire motion saliency and deformation geometric information at each stage, thereby effectively improving the segmentation accuracy of thin smoke, small targets and semi-transparent smoke edges.

[0132] S5.2, Use a combination of binary cross-entropy loss and Dice loss as the segmentation loss. The calculation formula is as follows:

[0133] Binary cross-entropy loss :

[0134]

[0135] Dice loss :

[0136]

[0137] In the formula, Indicates the true label, Represents the predicted value. This is the smoothing constant.

[0138] The binary cross-entropy loss is used to evaluate the confidence of pixel-by-pixel prediction, providing stable gradient supervision for the label-blurred region at the smoke boundary; the Dice loss measures the intersection-union ratio of the prediction and the ground truth from a global perspective, effectively alleviating the class imbalance problem caused by the smoke region being smaller than the background.

[0139] S5.3, Time Consistency Loss Motion saliency map generated by MVSM As an adaptive pixel-wise weight, a regularization constraint is applied to the absolute difference between the predicted probability maps of adjacent frames. The calculation formula is as follows:

[0140]

[0141] In the formula, and These represent the predicted probability maps for the current frame and the previous frame, respectively. Indicates the image height. Indicates the image width.

[0142] In regions with insignificant motion, the loss weight is larger, imposing a strong penalty on inconsistencies in cross-frame predictions; while in regions with significant motion, the loss weight is smaller, allowing the network to adaptively respond to real changes in smoke and dust morphology. This design effectively suppresses cross-frame prediction jitter in static background regions, while avoiding imposing excessive consistency constraints on smoke and dust movement regions and suppressing real temporal changes, thus enhancing the temporal coherence and stability of the segmentation results.

[0143] S5.4 Boundary Loss A specific optimization was performed to address the problem of blurred and difficult-to-accurately label smoke and dust boundaries. The boundary loss calculation formula is as follows:

[0144]

[0145] In the formula, This represents the set of boundary pixels in the real mask. This indicates its base number, i.e., the number of pixels. Indicates the true label, This represents the predicted value.

[0146] By applying a truth mask The erosion operation of morphological structural elements subtracts the original mask from the erosion result to extract a set of boundary pixels N with a width of about one pixel. During training, additional penalties are applied to the prediction error at the boundary pixels belonging to set N, amplifying the learning gradient of the network in the boundary region, enhancing the network's ability to finely model the blurred edges of semi-transparent smoke and dust, and suppressing the generation of overly smooth or jagged artifacts in the prediction of the boundary region.

[0147] S5.5 The three loss functions are weighted and summed according to their weight coefficients to form the overall multi-task joint training objective. The total training loss is as follows:

[0148]

[0149] In the formula, the weighting coefficients satisfy... Grid search verification settings , , The three loss terms work together to supervise the network training from three dimensions: spatial segmentation accuracy, temporal prediction coherence, and boundary refinement. This achieves end-to-end joint optimization of the dual-stream encoder-decoder network and ultimately outputs a temporally consistent and clearly defined industrial dust segmentation mask.

[0150] Example 2

[0151] This embodiment experimentally verifies and analyzes the performance of an industrial smoke and dust video target segmentation method based on motion saliency and deformation perception, as detailed below:

[0152] To verify the effectiveness of the method in this embodiment, tests were conducted using videos of smoke and dust from different real factories.

[0153] Dataset Details: The dataset contains industrial smoke images from 15 different video regions of multiple steel plants; a spatially disjoint data partitioning strategy was adopted to rigorously evaluate generalization ability; seven regions were excluded from training and designated for the actual validation set.

[0154] Dataset processing: The remaining video areas were selected and segmented into frames. Smoke targets were manually labeled using Labelme. A total of 8915 smoke images were extracted, and each image was adjusted to... Pixel resolution;

[0155] A spatially disjoint data partitioning strategy was adopted for partitioning; the 8915 smoke and dust images were divided into: 7125 frames as the historical training set, 900 frames as the validation set, and the remaining 890 frames as the independent test set, which were used to simulate the online segmentation process.

[0156] Input variables include: consecutive adjacent RGB video frames with a resolution of [resolution value missing]. Pixels. Scenes encompass complex conditions such as thin smoke, dense smoke emissions, cloud interference, and changes in lighting.

[0157] The evaluation metrics used are: Intersection over Union (IOU), F-score, and frame rate (FPS).

[0158] Experimental basis: The experiment was conducted on a workstation equipped with an Intel Core i9-12900K CPU (3.20GHz), 64GB RAM and NVIDIA RTX 3090 GPU (24GB VRAM); the software environment was implemented using Python 3.8 and the PyTorch deep learning framework.

[0159] Experimental Procedure: During the training phase, the AdamW optimizer was used to update the network parameters, with an initial learning rate of 0.0001 and a weight decay coefficient of 0.00001. Cosine annealing was used to adjust the learning rate, and the training period was set to 200 epochs. The batch size during training was set to 16. The loss function adopted was the multi-task joint loss function as described above. The weight coefficients of the segmentation loss, temporal consistency loss, and boundary loss were determined through grid search to satisfy... Grid search verification settings , , After 200 training rounds, the system is tested, and the final segmentation result is output.

[0160] Experimental results: The industrial smoke and dust video target segmentation method (DMPNet) based on motion saliency and deformation perception described in this invention: IoU=0.8282, F-score=0.9046, FPS=86.05.

[0161] Compared to the best static method (MSCA-UNet3+): IoU=0.7678, F-score=0.8557.

[0162] Comparing the optimal values ​​of the dynamic method (NetWarp): IoU=0.7723, F-score=0.8470.

[0163] Traditional benchmark method (U-Net): IoU=0.7293, F-score=0.8325.

[0164] The results show that the industrial smoke and dust video target segmentation method based on motion saliency and deformation perception described in this invention significantly outperforms existing technologies in segmentation accuracy; compared with the optimal dynamic comparison method, the IoU is improved by more than 5.5%; compared with traditional static methods, the IoU is improved by about 10%. Meanwhile, DMPNet maintains a real-time inference speed of 86 FPS at high accuracy, significantly outperforming complex dynamic comparison methods.

[0165] In summary, the industrial smoke and dust video segmentation method proposed in this invention solves the problem of confusion between semi-transparent smoke and clouds by fusing multi-view distance metrics through the MVSM module, solves the problem of non-rigid boundary deformation perception through the geometric embedding mechanism of the DEPNet module, and achieves adaptive integration of spatiotemporal features through the gated fusion mechanism of the SDGF module. This method not only significantly improves the segmentation accuracy and anti-interference ability in industrial scenarios, but also has good real-time performance, and can provide accurate and reliable visual analysis results for industrial environmental monitoring, thus having significant industrial application value.

Claims

1. A video target segmentation method for industrial smoke and dust based on motion saliency and deformation perception, characterized in that, Includes the following steps: S1. Video Data Acquisition and Preprocessing: Acquire videos of industrial dust emissions from real industrial sites, and preprocess the videos to obtain consecutive video frames and corresponding tags. The current frame is identified by the RGB images of two adjacent frames. And the previous frame As network input, multi-scale features are extracted by a dual-stream weight-sharing U-Net encoder; S2. Motion Saliency Calculation: Then, at different encoder levels, the Multi-View Motion Saliency Module (MVSM) calculates feature differences at different scales from three perspectives: Euclidean distance, cosine distance, and statistical moment distance. These differences are then fused using adaptive weighted Gaussian mapping to generate a motion saliency prior map. ; S3. Deformation Boundary Modeling: The Deformation Edge Awareness Network DEPNet utilizes geometrically embedded separable convolutions (GESConv) to inject normalized spatial coordinate priors to predict non-rigid deformation intensity maps. Accurately model the boundary of smoke and dust diffusion; S4. Saliency-Deformation Gated Fusion: The saliency-deformation gated fusion module SDGF adopts a cascaded spatial-channel gating mechanism to adaptively fuse motion saliency features and deformation geometric features, and outputs discriminative spatiotemporal fusion features for use by the decoder via skip connections; S5. Decoding Output and End-to-End Optimization: The decoder fuses the bottleneck layer features of the current frame with the fused features of the SDGF outputs from each encoder stage, and restores spatial resolution by upsampling at each stage through skip connections; combined with segmentation loss. Time consistency loss and boundary loss The multi-task joint loss function is used for end-to-end optimization, and the final smoke and dust segmentation mask is output.

2. The industrial smoke and dust video target segmentation method based on motion saliency and deformation perception according to claim 1, characterized in that, In S1, real-world industrial dust emission videos are captured and preprocessed to obtain consecutive video frames and corresponding tags, using the RGB images of two adjacent frames as the current frame. With the previous frame As network input, multi-scale features are extracted by a dual-stream weight-sharing U-Net encoder. The specific steps are as follows: S1.1 Collect real industrial site smoke and dust emission videos. The video sources cover different emission areas of multiple industrial plants, covering a variety of typical and complex industrial scenarios such as thin smoke, multi-target smoke and dust, cloud and fog interference, low light, strong light and colored smoke and dust, in order to ensure the diversity of the dataset and the comprehensiveness of the scenario coverage. S1.2 Extract the acquired raw video into an ordered and continuous sequence of RGB static image frames according to a certain inter-frame interval. All frames are uniformly scaled to standard resolution. To meet network input size requirements; S1.

3. Manually annotate the smoke and dust regions in each frame of the image at the pixel level to generate the corresponding binary segmentation label mask. The pixel values ​​of the smoke and dust areas are labeled as white (255) and the pixel values ​​of the background areas are labeled as black (0), thus constructing a complete dataset with one-to-one correspondence between image frames and labels. At the same time, the complete data is divided into training set, validation set and test set according to different industrial scenario domains to ensure that the three come from different scenarios, so as to rigorously evaluate the model's generalization ability to unknown scenarios. S1.

4. The network input pair is formed by two adjacent RGB images on the video timeline, i.e., the current frame. With the previous frame As network input, two frames of images are simultaneously input as a set of sample pairs to the two parallel branches of the dual-stream U-Net encoder, with the corresponding supervision label being the segmentation mask of the current frame. ; S1.5 The dual-stream U-Net encoder consists of two completely symmetrical parallel coding branches, which share all convolutional layer weight parameters. Each branch contains four coding stages and four downsampling stages. Each coding stage consists of two SimpleConvBlocks stacked together, progressively extracting features and downsampling spatially from the input frame. The current frame... The encoder outputs four different scale features. Previous frame The same shared weight encoder outputs four feature maps at different scales. The weight sharing mechanism ensures that the feature extraction standards of the two frames are strictly consistent and significantly reduces the number of model parameters, effectively suppressing overfitting. The four feature pairs extracted at different scales are output to the subsequent SDGF modules at each scale for spatiotemporal motion feature fusion.

3. The industrial smoke and dust video target segmentation method based on motion saliency and deformation perception according to claim 1, characterized in that, In S2, at different encoder levels, the Multi-View Motion Saliency Module (MVSM) calculates feature differences at different scales from three perspectives: Euclidean distance, cosine distance, and statistical moment distance. These differences are then fused using an adaptive weighted Gaussian mapping to generate a motion saliency prior map. The specific steps are as follows: S2.1, at the nth (n=1,2,3,4) encoder level, using the features of the current frame... Features from the previous frame As input to the MVSM module, inter-frame feature differences are calculated from three complementary perspectives: intensity difference, texture difference, and statistical distribution difference. S2.

2. Capture the absolute changes in smoke density and brightness from the perspective of Euclidean distance, and calculate the feature maps of the two frames pixel by pixel across all channels. Distance, resulting in intensity difference The formula for calculating the strength difference is as follows: In the formula, and These represent the feature values ​​of the current frame and the previous frame at the c-th channel spatial position, respectively, where C is the total number of channels; this perspective has a strong response to local intensity changes during the appearance, accumulation, and dissipation of smoke and dust areas; S2.

3. Decouple the evolution of smoke and dust texture from illumination interference from the perspective of cosine distance. Calculate the cosine similarity between the feature vectors of two frames pixel by pixel and convert it into cosine distance to obtain the texture difference. The formula for calculating texture differences is as follows: In the formula, express Norm, It is a numerical stability constant to prevent division by zero; the cosine distance only measures the change in the direction of the feature vector and is not sensitive to the magnitude of the vector. It is naturally invariant to illumination and can effectively capture the structural texture evolution inside the smoke and dust clusters. S2.

4. Characterizing the non-rigid diffusion pattern of smoke and dust from the perspective of statistical moment distance, centered on the spatial location (h,w). Within a local window, calculate the local first-order moments of the features of the current frame and the previous frame. With second moment The formula is as follows: Based on this, the statistical moment distance is calculated by comparing the temporal changes of local statistics in adjacent frames. The formula is as follows: In the formula, and These are the inter-frame variations of the local mean and standard deviation of the c-th channel, respectively. The mean term reflects local intensity changes, while the standard deviation term reflects changes in texture and contrast patterns. From a statistical perspective, this comprehensively characterizes the state transition process of the plume while smoothing out local disturbances. S2.5, Differences in characteristics from three perspectives , , Perform adaptive weighted Gaussian mapping fusion for each distance metric. After nonlinear transformation using a learnable Gaussian kernel function, weighted summation is performed to generate a motion saliency prior map. The formula is as follows: In the formula, the weights satisfy Gaussian scaling parameters can be learned Normalized using the softmax function, the initial values ​​were set to [0.5, 1.0, 1.5] via grid search, corresponding to fine-grained, mesoscale, and coarse-grained change detection, respectively; weights Uniform initialization And adaptive learning through backpropagation; This fusion mechanism ensures that a region is identified as a high-confidence dust change region only when multiple metrics simultaneously detect significant differences. The output is fed into the subsequent SDGF module as the motion prior input for spatial saliency gating.

4. The industrial smoke and dust video target segmentation method based on motion saliency and deformation perception according to claim 1, characterized in that, In S3, the deformation of non-rigid smoke and dust is modeled using the geometrically embedded separable convolution (GESConv) network DEPNet through deformation edge perception, and the deformation intensity map is output. The specific steps are as follows: S3.1 Construct a normalized coordinate grid G. For any position within the spatial dimensions, normalize the coordinates to... The calculation formula is as follows: In the formula, Indicates spatial dimension, Represents any position in the spatial dimension. Indicates coordinate transpose; Normalization ensures that the coordinate magnitude is consistent with the scale of pixel feature changes, preventing gradient instability during training. S3.2, Set the current frame features Features of the previous frame The composite input tensor is obtained by concatenating the coordinate grid G ​​along the channel dimension. This allows the network to utilize both temporal evolution information and spatial location priors simultaneously; S3.3, Define the GESConv operation: Let the input features be... The intermediate feature map U is extracted through depthwise separable convolution, and then the coordinate prior is injected through a lightweight residual branch to obtain the final output of GESConv. The formula is as follows: In the formula, for A convolutional kernel of size is used to project the coordinate information G onto the same channel dimension as U. For batch normalization, The ReLU activation function is used; this residual design allows the main path to retain the original semantic information, and the coordinate prior is used as a supplementary enhancement without interfering with the semantic features. Compared with CoordConv, which directly concatenates the coordinate channels, it has a lower computational cost. S3.

4. Through the above operations, the composite input Z undergoes hierarchical feature learning through two stacked GESConv layers, and then through two... Prediction based on convolution End-to-end output deformation intensity diagram The formula is as follows: In the formula, It is a feature mapping of the backbone network. It is a prediction head used to output deformation intensity maps. It is a composite tensor of the current frame features, the previous frame features, and the normalized coordinates G; Deformation intensity diagram The non-rigid deformation degree at each spatial location is quantized pixel by pixel, providing accurate dynamic boundary guidance information for the subsequent SDGF module. By explicitly embedding spatial coordinate priors, DEPNet breaks the spatial isotropy of standard convolution, enabling the network to have position awareness and distinguish the semantic features of smoke and dust at different diffusion stages, thereby achieving accurate non-rigid boundary modeling of diffuse dynamic smoke and dust in industrial scenarios.

5. The industrial smoke and dust video target segmentation method based on motion saliency and deformation perception according to claim 1, characterized in that, In S4, the motion saliency prior map output by the MVSM is adaptively fused through the saliency-deformation gating fusion module SDGF using a cascaded spatial-channel dual-gating mechanism. Deformation intensity map compared with DEPNet output Generate the final fused features, specific steps as follows: S4.1, Motion saliency diagram With deformation intensity diagram spliced ​​along the channel dimension, sequentially through Convolutional fusion of local context, ReLU nonlinear activation, Convolution is compressed to a single channel, then normalized using the Sigmoid function to generate a spatial weight mask. ;Will Element-wise operation on the current frame coding features Spatial augmentation features are obtained. The expression is as follows: ； This operation suppresses static background noise and focuses the characteristic response on the smoke and dust diffusion area where motion and deformation are significant; S4.2 Next, spatial enhancement features With deformation characteristics After being compressed into compact channel descriptors through global average pooling, the channels are concatenated, and then the cross-channel nonlinear dependencies are modeled through two layers of fully connected MLP. Finally, the channel weight vector is obtained through the Sigmoid function. Adaptively increase the weights of task-sensitive channels related to smoke texture, density, and boundary sharpness, and suppress redundant channel responses; S4.3 Finally, residual connections are used to superimpose the channel recalibration results onto the original current frame features to obtain the final fused output features. The expression is as follows: This residual design preserves original semantic information while introducing deformation-aware attention enhancement, preventing information degradation in deep networks; the resulting discriminative spatiotemporal fusion features It is passed to the decoder via a skip connection for subsequent upsampling and spatial resolution recovery.

6. The industrial smoke and dust video target segmentation method based on motion saliency and deformation perception according to claim 1, characterized in that, In S5, the specific steps of the decoder and the multi-task joint loss function are as follows: S5.1 The decoder first receives the high semantic and low resolution features output from the encoder bottleneck layer as initial input. In the first upsampling stage, the bottleneck layer features are upsampled and then concatenated with the SDGF fusion output features corresponding to the fourth stage of the encoder along the channel dimension. The fused semantic features are then extracted through a convolutional block. In the second to fourth upsampling stages, the above operation is repeated sequentially, and the fused features are concatenated and convolved with the SDGF fusion output features corresponding to the third, second, and first stages of the encoder, respectively, to gradually supplement the spatial detail information lost by downsampling. This allows the decoder to restore resolution while taking into account both high-level semantics and low-level edge textures. After four upsampling stages, the feature map resolution is restored to the input image size, and finally, the image is processed. Convolution compresses the number of channels to a single channel and outputs a smoke segmentation mask probability map. Since the skip connections at each stage introduce spatiotemporal discriminative features fused by SDGF rather than the original encoded features, the decoder can simultaneously acquire motion saliency and deformation geometric information at each stage, thereby effectively improving the segmentation accuracy of thin smoke, small targets and semi-transparent smoke edges. S5.2, Use a combination of binary cross-entropy loss and Dice loss as the segmentation loss. The calculation formula is as follows: Binary cross-entropy loss : Dice loss : In the formula, Indicates the true label, Represents the predicted value. It is a smoothing constant; The binary cross-entropy loss is used to evaluate the confidence of pixel-by-pixel prediction, providing stable gradient supervision for the label-blurred region at the smoke boundary; the Dice loss measures the intersection-union ratio of the prediction and the ground truth from a global perspective, effectively alleviating the class imbalance problem caused by the smoke region being smaller than the background. S5.3, Time Consistency Loss Motion saliency map generated by MVSM As an adaptive pixel-wise weight, a regularization constraint is applied to the absolute difference between the predicted probability maps of adjacent frames. The calculation formula is as follows: In the formula, and These represent the predicted probability maps for the current frame and the previous frame, respectively. Indicates the image height. Indicates the image width; In regions with insignificant motion, the loss weight is larger, imposing a strong penalty on inconsistencies in cross-frame predictions; while in regions with significant motion, the loss weight is smaller, allowing the network to adaptively respond to real changes in smoke and dust morphology. This design effectively suppresses cross-frame prediction jitter in static background regions, while avoiding imposing excessive consistency constraints on smoke and dust movement regions and suppressing real temporal changes, thus enhancing the temporal coherence and stability of the segmentation results. S5.4 Boundary Loss A specific optimization was performed to address the problem of blurred and difficult-to-accurately label smoke and dust boundaries. The boundary loss calculation formula is as follows: In the formula, This represents the set of boundary pixels in the real mask. This indicates its base number, i.e., the number of pixels. Indicates the true label, Represents the predicted value; By applying a truth mask The erosion operation of morphological structural elements subtracts the original mask from the erosion result to extract a set of boundary pixels N with a width of about one pixel. During training, additional penalties are applied to the prediction error at the boundary pixels belonging to set N, amplifying the learning gradient of the network in the boundary region, enhancing the network's ability to finely model the blurred edges of semi-transparent smoke and dust, and suppressing the generation of overly smooth or jagged artifacts in the prediction of the boundary region. S5.5 The three loss functions are weighted and summed according to their weight coefficients to form the overall multi-task joint training objective. The total training loss is as follows: In the formula, the weighting coefficients satisfy... Grid search verification settings , , The three loss terms work together to supervise the network training from three dimensions: spatial segmentation accuracy, temporal prediction coherence, and boundary refinement. This achieves end-to-end joint optimization of the dual-stream encoder-decoder network and ultimately outputs a temporally consistent and clearly defined industrial dust segmentation mask.

Images