Video smoke saliency target detection method based on weak supervised learning

Abstract

The invention relates to a video smoke saliency target detection method based on weak supervised learning, and the method comprises the steps: collecting fire smoke video data in different scenes, and obtaining a sample set; preprocessing the sample set to generate a pseudo label and optical flow data of the fire smoke video; a video saliency target detection model containing two sub-networks is designed, and motion features of a target are effectively processed; and designing a loss function considering context information to optimize the training process of the model. The method can be applied to different fire smoke video detection scenes under the condition of zero annotation data, the problem of saliency detection of smoke targets can be better solved, and efficient and accurate detection effects can be kept under different background conditions.

Description

Technical Field

[0001] The invention relates to the fields of fire smoke detection, video salient target detection and weakly supervised learning, and in particular to a video smoke salient target detection method based on weakly supervised learning. Background Art

[0002] Since most fires will show smoldering phenomenon at the beginning, smoke will appear before the flames at this time, and gradually develop from the smoke before large-scale burning to the fire state that is difficult to control, so the video-based smoke detection method has gradually become a key research direction in the field of video fire detection. Chinese patent application CN201010136306.1 (Embedded video smoke detector and smoke recognition method) extracts edge features, motion features, and texture features of the image for smoke pattern recognition, but this method is mainly based on the inter-frame difference method for smoke area monitoring. The problem is that when the smoke movement speed is uncertain, the model's ability to extract targets is limited; the use of traditional features such as edge, motion, and texture to describe targets is also relatively limited.

[0003] Video salient object detection aims to discover the most visually salient objects in a video and identify all pixels covering these salient objects. It can be used for many advanced applications such as video object segmentation, object tracking, and video understanding. Saliency detection can be roughly divided into two categories. The first category is eye gaze prediction, which focuses on predicting the viewer's eye gaze in the video, which helps to understand the intrinsic mechanisms of the human visual and cognitive systems from a biological perspective. The second category is salient object detection, which requires segmenting the most important or visually salient objects from the potentially cluttered background.

[0004] With the development of deep learning technology, existing research has explored a variety of video salient object detection methods, which can be roughly divided into two categories: one is the single-stream method using time-domain convolution or long short-term memory (LSTM), and the other is the dual-stream method using optical flow technology. Since the motion information of smoke is particularly critical, the dual-stream method using optical flow technology is more suitable for smoke inspection research. However, the existing dual-stream methods have the following problems: they are highly dependent on high-quality optical flow data, such as using Flownet2.0 and liteFlowNet2.0 to obtain optical flow data. The direct prediction effect of the model for smoke image pairs is poor. If the smoke data set is trained, the optical flow label is required; most methods only use optical flow features to enhance appearance features, which is not sufficient for the use of optical flow features. Some methods blindly merge optical flow features with appearance features, which not only increases the complexity of the model but also has poor results in video smoke tasks.

[0005] Therefore, there is an urgent need for an innovative method to effectively utilize optical flow features, reduce the dependence on high-quality label images and optical flow data, improve the accuracy and robustness of smoke detection, and be more practical. Summary of the invention

[0006] In order to solve the above technical problems, the present invention proposes a video smoke salient target detection method based on weak supervised learning, uses the Dual TV-L1 optical flow algorithm to obtain optical flow data, and adds optical flow and speed thresholds for filtering most of the noise in the fire smoke video to obtain optical flow data. The designed new model uses two sub-networks to complete two sub-tasks, one sub-network is used for salient target detection of still images, and the other sub-network is used for motion salient detection of optical flow images. In multiple stages, the optical flow features and appearance features are directly spliced ​​in a simple channel dimension, and input into a gated attention module that controls the importance of appearance features and motion features. The weighted appearance features and optical flow features of the gated attention module are input into an attention module with residual learning characteristics, and the appearance features are focused on and enhanced by optical flow features. In addition, considering that most of the labels obtained by the foreground detection algorithm only obtain the edge contours and motion salient parts of the moving smoke target and the target is small, a loss function for the missing part of the smoke contour inside the label is designed, and a loss function that weights and pays attention to the overall difference is also designed. The present invention has a significant effect on the detection of salient targets in fire smoke videos.

[0007] In order to achieve the above object, the present invention adopts the following technical scheme: A video smoke salient target detection method based on weakly supervised learning includes the following steps: Step S1: Collect fire smoke video data in different scenes to obtain a sample set; Step S2: Preprocess the sample set, generate pseudo labels and optical flow data of the fire smoke video based on the existing foreground detection and optical flow estimation technology, wherein the pseudo labels are produced using the improved mixed Gaussian model algorithm; the dense optical flow extraction Dual TV-L1 optical flow algorithm is used to obtain optical flow data, and the regularization technology and L1 norm are used to estimate the optical flow between each frame, and the motion information in the image sequence is accurately captured by minimizing the change of image brightness and the smoothness of the flow field; Step S3: Design a video salient target detection network model containing two sub-networks, input the preprocessed sample set into the video smoke salient target detection network model, the two sub-networks in the network model are used for spatial salient target detection and optical flow motion salient target detection respectively, and the optical flow features are used to enhance the spatial image features through the gated attention module at different stages of feature extraction of the two sub-networks to achieve feature interaction, so as to effectively process the motion features of the target; Step S4: Design a loss function that takes contextual information into account, assign higher weights to parts with larger differences in the neighborhood, and take into account global differences to optimize the training process of the network model.

[0008] Compared with the prior art, the present invention has the following beneficial effects: (1) The present invention proposes a salient target detection model that uses optical flow data to enhance the appearance features of smoke. By making full use of the motion characteristics of smoke, the effect of the present invention is better than other existing methods.

[0009] (2) The present invention takes into account both context-based and global loss function designs to make up for the omission of pseudo labels for the target.

[0010] (3) The present invention proposes a weak supervision method that uses pseudo labels, video frames, and optical flow data for training, and manually annotated real labels, video frames, and optical flow data for prediction. This method can accurately detect untrained video smoke data without the need for expensive manual annotation and high-quality optical flow data, which is extremely robust and has practical significance. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 This is a flow chart of a method for detecting salient targets in video smoke based on weakly supervised learning in an embodiment of the present invention; Figure 2 Schematic diagram of the pseudo-labeling process of the smoke area in an embodiment of the present invention, wherein (a) represents the original image of the video frame, (b) represents the pseudo-labeling initially generated by using the improved mixed Gaussian model algorithm, (c) represents the pseudo-labeling image after filtering using the brightness threshold, and (d) represents the final pseudo-labeling image after filtering using the area threshold; Figure 3 Schematic diagram of the optical flow data acquisition process of a smoke target in a video according to an embodiment of the present invention, wherein (a) and (b) represent two adjacent frames of a video, (c) represents an optical flow map initially generated using Dual TV-L1, and (d) represents a final optical flow map generated after filtering using brightness and speed thresholds; Figure 4 Schematic diagram of a video salient object detection network model in an embodiment of the present invention; Figure 5 Schematic diagram of a gated attention module in an embodiment of the present invention; Figure 6 : is a loss function explanation diagram in an embodiment of the present invention; wherein (a) represents the original image, and (b) represents the enlarged original image and pseudo label; Figure 7Graphs showing prediction results of training and testing using pseudo labels in an embodiment of the present invention, wherein (a) represents a video frame, (b) represents a pseudo label, and (c)-(e) represent graphs showing experimental results of an MGA model, a DCFNet model, a FSNet model, and the present invention, respectively; Figure 8 This is a prediction result diagram of an embodiment of the present invention that is trained using pseudo labels but tested using untrained real labels, where (a) represents a video frame, (b) represents a real label, and (c)-(e) represent the experimental result diagrams of the MGA model, the DCFNet model, the FSNet model, and the present invention, respectively. DETAILED DESCRIPTION

[0012] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0013] The present invention provides a video smoke salient target detection method based on weakly supervised learning. In order to make the purpose, technical solution and advantages of the present invention clearer, the present invention is further described in detail below through specific implementation and in combination with the accompanying drawings.

[0014] like Figure 1 As shown, the embodiment of the present invention provides a video smoke salient target detection method based on weakly supervised learning, comprising the following steps: Step S1: Collect fire smoke video data in different scenarios to obtain a sample set. The data used comes from public data sets such as MIVIA, RISE, KMU, Wildfire, D-Fire, etc., which contain smoke data sets. There are 64 fire smoke videos in total. The results of some of the videos are shown through the examples. Manually mark the start and end time points of the fire smoke event, and capture 20,504 video frames according to the start and end time points. Step S2: Preprocess the sample set, generate pseudo labels and optical flow data of the fire smoke video based on the existing foreground detection and optical flow estimation technology, wherein the improved mixed Gaussian model algorithm is used to generate pseudo labels. The process is as follows: Figure 2 As shown in the figure, the improved Gaussian mixture model algorithm is an advanced foreground detection method. It uses an adaptive Gaussian mixture model to dynamically learn and update the background, and filters out most of the noise by designing the area threshold and the connected domain area threshold. The dense optical flow extraction Dual TV-L1 optical flow algorithm is used to obtain optical flow data. The process is as follows Figure 3As shown in the figure, the core of the Dual TV-L1 optical flow algorithm is to use regularization technology and L1 norm to estimate the optical flow between each frame. By minimizing the change of image brightness and the smoothness of the flow field, it can accurately capture the motion information in the image sequence. Finally, most of the noise is removed by designing brightness and speed threshold filtering. Figure 2 (a) represents the original image of the video frame, Figure 2 (b) indicates that the improved Gaussian mixture model algorithm is used to initially generate pseudo labels. Figure 2 (c) represents the pseudo label map after filtering using the brightness threshold. Figure 2 (d) represents the final pseudo-label after filtering using the area threshold. Figure 3 (a) and Figure 3 (b) represents two adjacent frames of the video. Figure 3 (c) represents the optical flow map initially generated using Dual TV-L1. Figure 3 (d) shows the final optical flow map generated after filtering using brightness and speed thresholds.

[0015] Step S3: Design a video salient object detection network model containing two sub-networks, such as Figure 4 As shown in FIG, the preprocessed sample set is input into the video smoke salient target detection network model. The two sub-networks in the network model are used for spatial salient target detection and optical flow motion salient target detection respectively. At different stages of feature extraction of the two sub-networks, the optical flow features are used to enhance the features of the spatial image through the gated attention module to achieve feature interaction, so as to effectively process the motion features of the target. The schematic diagram of the gated attention module is shown in FIG. Figure 5 As shown; Step S4: design a loss function that considers contextual information, assigning higher weights to parts with larger differences in the neighborhood, while taking into account global differences, to optimize the training process of the network model; In an embodiment, step S2 comprises: Pseudo-labeling is performed using an improved Gaussian mixture model algorithm, which is an advanced foreground detection method that dynamically learns and updates the background by using an adaptive Gaussian mixture model. The algorithm is well suited for handling dynamic scenes and lighting changes because it can adapt to changes in the environment and extract foreground objects from complex backgrounds. For each pixel in a single video, the algorithm maintains multiple Gaussian distributions for each pixel, which represent the range of color changes that may occur at that point in the background. This allows the algorithm to flexibly adapt to dynamic changes in the environment, such as light flickering, swaying leaves, etc. The core of the algorithm is to update the background model of each pixel in real time, while identifying pixels that do not match the background model and classifying them as foreground. When dealing with special scenes, since smoke usually has unique color or brightness features, these features can be effectively identified by brightness thresholds. In addition, combined with the application of connectivity threshold area thresholds, the algorithm can further eliminate false detections caused by environmental noise or small interference, thereby minimizing false alarms while ensuring detection sensitivity.

[0016] The optical flow data is obtained by using the Dual TV-L1 optical flow algorithm, which can accurately capture the motion information in the image sequence by minimizing the changes in image brightness and the smoothness of the flow field. The significant advantages of the algorithm are its robustness to noise and its ability to maintain the clarity of moving edges, including: First, the video is broken down into a series of individual frames. By analyzing each frame, the minute changes on the timeline can be tracked in detail. Starting from the second frame, the pixel motion of each frame relative to the previous frame is calculated. The resulting optical flow vector reflects the movement of the pixels between the two frames. This continuous frame analysis method is conducive to capturing moving targets. The calculated optical flow vector contains two components: the horizontal component and the vertical component The movement of smoke is then normalized to convert these optical flow components into a visually recognizable format, namely a pseudo-color image, so that the intensity and direction of the movement can be intuitively represented. Finally, the larger value of each pixel of the two images is retained, and the pseudo-color images in two directions are merged into the final optical flow RGB image. The significant characteristics of smoke are used to further optimize the calculation of optical flow, which is usually slow and has a certain degree of irregular diffusion pattern. By setting a speed threshold To filter out fast-moving elements, which may be caused by camera shake, fast-moving objects such as vehicles, etc., which do not match the motion characteristics of the smoke target.

[0017] According to the above characteristics, a preset optical flow velocity threshold is used to filter out unnecessary dynamic parts, as shown in formula (1), where: and Represent the horizontal component and vertical component of the optical flow vector respectively. The setting of the speed threshold is based on the calculation of the size of the two components of the optical flow vector, that is, by measuring the horizontal and vertical displacement distances of each pixel between consecutive frames. When the movement speed of a pixel exceeds this threshold, it can be considered not to be part of the smoke and can be excluded. This process helps to reduce false detection and noise. At the same time, setting the brightness threshold is also a key step in analyzing smoke in the video. Since smoke usually has a high brightness, by setting a brightness threshold, those areas with lower brightness are excluded before calculating the optical flow, thereby ensuring that only those bright areas that may be smoke are further analyzed. The final optical flow map is shown below. Figure 3 As shown, the final optical flow data significantly reduces noise interference, so that the smoke target part in the final optical flow image can be clearly displayed.

[0018] (1) In this embodiment, step S3 includes: like Figure 4As shown in the figure, the network model contains appearance branch, motion branch and multiple gated attention modules, which serve as bridges connecting appearance and motion branches. The structure of appearance subnetwork is similar to that of motion subnetwork, but there are some differences. Since optical flow images do not contain rich high-level semantics and delicate boundary information like RGB images, the pre-trained Resnet101 framework is selected for appearance subnetwork and the Resnet34 framework is selected for motion subnetwork. Both appearance branch and motion branch consist of three parts: encoder, spatial pyramid pooling module and decoder. The encoder works by extracting visual features from low-level to high-level and reducing the resolution of feature maps. The encoder contains five layers: one head convolution layer and four residual layers, respectively denoted as residual layer-i (i ∈ {1, 2, 3, 4}). The number of output channels of the head convolution layer is 64, the kernel size is 7×7, the stride is 2, followed by a batch normalization and ReLU function. For the appearance branch, the four residual layers contain 3, 4, 23 and 3 "bottleneck" structures based on residual learning. This structure optimizes the feature transfer and conversion process through a series of convolutional layers and batch normalization layers. It starts with a 1x1 convolutional layer to reduce the dimension of the feature map, followed by a 3x3 convolutional layer. Finally, another 1x1 convolutional layer is used to expand the depth of the feature map to four times the original output channels of 256, 512, 1024 and 2048 respectively. The introduction of the residual learning concept enables the network to effectively avoid the gradient vanishing or exploding problems while increasing the depth, improving the learning efficiency and the expression ability of the model. For the motion branch, its residual layer uses 3, 4, 6 and 3 basic residual learning blocks, and the output channels are 64, 128, 256 and 512 respectively. The stride of these four residual layers is set to 2, 2, 1 and 1 in both sub-networks.

[0019] By applying dilated convolutions, the spatial pyramid pooling module is able to capture long-range dependencies from feature maps and fuse this information with local and global representations, which implicitly facilitates the capture of long-range contrast in saliency modeling. Figure 4 As shown in Figure 2, the spatial pyramid pooling module processes the input features through five parallel layers, including a 1×1 point convolution layer, three 3×3 hole convolution layers with different hole rates (12, 24, and 36), and a global average pooling layer. The outputs of these five parallel layers are concatenated in the depth dimension to generate the final feature map.

[0020] The decoder restores the spatial size of the feature map by fusing high-level and low-level features to accurately predict a high-resolution saliency map with clear object boundaries. The high-level output features of the spatial pyramid pooling module are reduced to 256-channel features by a 1×1 convolution 'Convolution-1' in the decoder, while the low-level features from the residual layer-1 are reduced to 48-channel features by another 1×1 convolution 'Convolution-2'. After concatenating the low-level features with the high-level features, two 3×3 convolutions with an output channel of 256 are performed, which are recorded as 'Convolution-3' and 'Convolution-4' respectively. Subsequently, a 1×1 convolution 'Convolution-5' is used, followed by a sigmoid function, to predict the final single-channel saliency map. The decoder of the motion branch adopts a three-layer structure similar to Convolution-{3-5}.

[0021] In the video salient object detection network model designed by the present invention, an important step is to integrate the appearance branch and the motion branch with the gated attention module (Gated-i). Figure 4 As shown in Figure 1, there are 6 gated attention modules (gate-i, i ∈ {0, 1, 2, 3, 4, 5}) in the model. Gate-0 receives the output of the two head convolutional layers in the appearance sub-network and the motion sub-network as input. Gate-i receives the output of residual layer-i in both branches as input. It is worth noting that in the appearance sub-network, the direct connection within the encoder is removed, and the output of gate-0 replaces the output of the head convolution and is passed to the residual layer-1 of the appearance branch. Similarly, the residual layer-i in the appearance branch receives the output generated by gate-(i-1) as input instead of residual layer-(i-1). Gate-4 replaces residual layer-4 and is connected to the spatial pyramid pooling module in the appearance sub-network. Unlike the appearance branch, the encoder of the motion branch retains the internal connection and provides the side output as the input of gate-{0-4}. Gate-{0-4} is located on the encoder side, while gate-5 is located on the decoder side. Since the motion input of gate-5 is the single-channel motion saliency map output by the decoder, it is denoted as , so its structure is different from the Gated-{0-4} module structure. It only uses the spatial attention operation similar to that in Gated-{0-4}, directly multiplies each channel slice of the motion saliency map and the fused features in the appearance branch decoder by elements, and then adds them to the fused features, in order to highlight the significant motion area. Finally, the output of Gated-5 is passed to the 'Convolution-3' of the appearance sub-network. Its formula is as follows: (2) in, For appearance characteristics, is the single-channel motion saliency map output by the decoder, represents element-wise multiplication, is the output feature of gate-5; The gated attention module in step S3 is as follows: In the dual-stream structure, one is the appearance branch for video frame input, and the other is the motion branch for optical flow data input. Therefore, the core task is to study how to effectively use motion information to significantly mark important positions or elements in the representation of appearance features. The gated attention module is as follows: Figure 5 "Appearance characteristics ” and “Sports Features ” is defined as the feature tensor generated by some hidden layers (such as ReLU activation function) in the appearance branch and the motion branch. The appearance features and motion features of different levels and scales are simply concatenated in the channel dimension, and the concatenated fusion features are used Then the fusion features In the input gated attention module, the gated attention module is used to control the importance of appearance features and motion features, which means that the network model can independently adjust the degree of dependence on image features and optical flow features. As shown in formula (3), represents the global average pooling operation, express Activation function, represents the convolution operation, Indicates that according to the fusion features after splicing The generated gating signals are used as learned weights to adjust the importance of different features in subsequent processing stages. Represents the output features of the gated attention module.

[0022] (3) Specifically, the concatenated features are passed to a convolutional layer to generate the gating signal. The convolutional layer in formula (3) sets the input channels of 128, 320, 640, 1280 and 2560 and 2 output channels in the five gated attention modules respectively. After the sigmoid calculation, a global average pooling operation is performed. Finally, a separate appearance gating signal is separated from the gated signal tensor. and optical flow gating signal Used to weight the original and , the original and They are respectively multiplied element by element with their gated signals to achieve feature weighting. The gated attention module can dynamically adjust the appearance and optical flow features, so that the network model can process information more flexibly according to the current input. The weighted features are used and express.

[0023] Weighted optical flow features Apply 1x1 convolution of the spatial dimension to compress the input channel into a single channel output and process it through the sigmoid activation function to obtain a spatial attention map. Then, multiply this spatial attention map with the weighted appearance features to achieve spatial attention weighting. The formula is as follows, where represents the appearance features after spatial attention weighting, represents the convolution operation, Represents the activation function: (4) The appearance features after spatial attention enhancement are average pooled to generate a feature vector, which is then processed by a 1x1 convolution in the channel dimension (the output channel is consistent with the input channel), and the Softmax activation function is applied in the channel dimension. Then, it is multiplied by the number of channels C to ensure that the sum of the weights remains unchanged and generate channel attention weights. These weights reflect the relative importance of different channels. Finally, these weights are applied to the appearance features after spatial attention enhancement to achieve channel attention weighting. The appearance features after channel attention enhancement are combined with the appearance features after gated weighting. Add element by element to generate the final enhanced features. The aim is to retain the original information while introducing the attention mechanism to improve the richness of feature representation. Although the attention mechanism is designed to highlight the motion-guided part, it may sometimes mistakenly weaken or ignore other important features. Therefore, through the setting of the residual term, the model can retain the information of those smoke areas where the motion is not obvious but equally significant, thereby avoiding the loss of important features. The formula is as follows, where is the number of channels, represents the appearance features after weighting by spatial attention and channel attention, represents the appearance features after spatial attention weighting, represents the global average pooling operation, represents the convolution operation, Represents the activation function: (5) In an embodiment, step S4 comprises: The weighted global loss function (WGLoss) proposed in this invention is effective for the task of detecting salient targets in fire smoke videos. Since the pseudo-labeling algorithm sets the part with significant motion as the foreground, and the smoke in the distance in an open outdoor scene such as a forest moves slowly, the algorithm can identify the edge of the smoke contour (the part with significant motion), but the smoke part inside the contour may not move significantly and be regarded as the background, such as Figure 6 as shown in (b). Figure 6 (a) is the original image.

[0024] How to make the model pay more attention to the foreground edge part and the internal part of the smoke outline in the label image during training is the starting point of the loss design. Figure 6 (b) The gray box, assuming the pixel is 30 30, let the label value of each pixel in the box minus the average label value in the box, and use the size of this difference to assign weights to the green pixels in the red box. In the case of this schematic diagram, the foreground part in the box is larger than the background part, so the average label value in the box is greater than 0.5. For the target pixel label value of 0, the missing pixel inside the smoke will be assigned a higher weight, and the same is true for the edge part. Therefore, the first part of the loss function is set as shown in formula (6), where represents a pseudo label, Represents the prediction result, represents the average pooling operation, represents the binary cross entropy loss, and Indicates the height and width of the image, Represents the total number of samples in the batch during training, and Represents the pixel index in the height and width directions of the image, where n is the index value.

[0025] (6) Use F.avg_pool2d to perform an average pooling operation on the pseudo-label target pixel neighborhood. The size of the pooling window is set to 31x31, the step size is 1, and the edge padding is 15 to obtain the average pseudo-label value of each pixel neighborhood. Then, subtract this average from the original pseudo-label value and take the absolute value, which represents the degree of difference between each pixel and the average of its neighborhood. The greater the difference, the higher the weight. Then calculate the binary cross entropy loss between the predicted value and the pseudo-label, and do not sum or average the losses, but retain the loss value of each pixel. Multiply the calculated weights with the binary cross entropy loss so that the loss of each pixel is weighted according to its difference from the neighborhood. Then, sum the weighted losses over the height and width dimensions of each sample and divide them by the sum of the weights over the same dimensions to obtain the average weighted loss for each sample. Since the weighted loss function (WLoss) is applied in each field and needs to take into account the global differences, a global loss function (Gloss) is introduced to evaluate the overall similarity of two samples, as shown in formula (7). The numerator of the fraction in the formula is the true class and the prediction result The pixels and pseudo labels of the image The denominator represents the predicted result ( and pseudo-label ( Total number of Zhongzheng categories, represents the smoothing term, The larger the fractional value of formula (7), the more similar it is. Subtracting the fractional value of formula (7) from 1 represents the global difference. (7) Finally, WLoss and GLoss are weighted. Set it to 0.8, the formula is as follows: (8) By comparing the MGA, DCFNet and FSNet models, we trained and tested them on all the data, where most of the labels are automatically generated pseudo labels. Figure 7 As shown in Figure 2, despite the problem of missing targets in pseudo labels, the proposed model can still learn effective features and outperform other models in terms of indicators. In particular, all pseudo labels generated are used for training and a small number of real labels with pure manual annotations are used for testing. The results are shown in Figure 2. Figure 8 As shown. Among them, Figure 7 (a) represents a video frame, Figure 7 (b) represents the pseudo label, Figure 7 (c)- Figure 7 (e) respectively represent the experimental results of MGA model, DCFNet model, FSNet model and the present invention; Figure 8 (a) represents a video frame, Figure 8 (b) represents the true label, Figure 8 (c)- Figure 8 (e) respectively represent the experimental results of the MGA model, DCFNet model, FSNet model and the present invention. Compared with other models, even in the face of very small smoke targets and strong noise challenges, the method proposed in the present invention can generate higher quality saliency detection maps.

Claims

1. A video smoke salient target detection method based on weakly supervised learning, characterized in that: The steps include: Step S1: Collect fire smoke video data in different scenes to obtain a sample set; Step S2: Preprocess the sample set, generate pseudo labels and optical flow data of the fire smoke video based on the existing foreground detection and optical flow estimation technology, wherein the pseudo labels are produced using the improved mixed Gaussian model algorithm; the dense optical flow extraction Dual TV-L1 optical flow algorithm is used to obtain optical flow data, and the regularization technology and L1 norm are used to estimate the optical flow between each frame, and the motion information in the image sequence is accurately captured by minimizing the change of image brightness and the smoothness of the flow field; Step S3: Design a video salient target detection network model containing two sub-networks, input the preprocessed sample set into the video smoke salient target detection network model, the two sub-networks in the network model are used for spatial salient target detection and optical flow motion salient target detection respectively, and the optical flow features are used to enhance the spatial image features through the gated attention module at different stages of feature extraction of the two sub-networks to achieve feature interaction, so as to effectively process the motion features of the target; Step S4: Design a loss function that takes contextual information into account, assign higher weights to parts with larger differences in the neighborhood, and take into account global differences to optimize the training process of the network model.

2. The video smoke salient target detection method based on weakly supervised learning according to claim 1 is characterized in that: The step S2 includes: when using the improved mixed Gaussian model algorithm to produce pseudo labels, for each pixel in a single video, the improved mixed Gaussian model algorithm maintains multiple Gaussian distributions of each pixel to reflect the possible range of color changes; the background model of each pixel is updated in real time, and those pixels that do not match the background model are identified and classified as foreground; smoke is identified through brightness thresholds; and false detections caused by environmental noise or small interference are further eliminated in combination with connectivity thresholds and area thresholds.

3. The video smoke salient target detection method based on weakly supervised learning according to claim 1 is characterized in that: The step S2 uses the dense optical flow extraction Dual TV-L1 optical flow algorithm to obtain optical flow data, including: first, decomposing the video into a series of individual frames, and tracking the small changes on the time axis in detail by frame-by-frame analysis; starting from the second frame, calculating the pixel movement of each frame relative to the previous frame, and the generated optical flow vector reflects the movement of the pixel between the two frames; the calculated optical flow vector contains two components: the horizontal component and the vertical component The movement of the light flow is then converted into a visually recognizable format, namely a pseudo-color image, through normalization, indicating the intensity and direction of the movement. Finally, the larger value of each pixel of the two images is retained, and the pseudo-color images in the two directions are merged into the final optical flow RGB image. According to the preset speed threshold Filter out the unnecessary dynamic part, as shown in formula (1), where and Represent the horizontal and vertical components of the optical flow vector respectively: （1）。 4. The video smoke salient target detection method based on weakly supervised learning according to claim 1 is characterized in that: The step S3 comprises: The network model includes appearance branch, motion branch and multiple gated attention modules. The appearance sub-network selects the pre-trained Resnet101 framework, and the motion sub-network selects the Resnet34 framework. Both the appearance branch and the motion branch are composed of encoders, spatial pyramid pooling modules and decoders. The encoder contains five layers: a head convolution layer and four residual layers. The head convolution layer has 64 output channels, a kernel size of 7×7, and a step size of 2, followed by a batch normalization and ReLU function. For the appearance branch, the four residual layers include 3, 4, 23 and 3 "bottleneck" structures based on residual learning. The "bottleneck structure" optimizes the transmission and conversion of features through a series of convolution layers and batch normalization layers. The conversion process starts with a 1x1 convolution layer to reduce the dimension of the feature map, followed by a 3x3 convolution layer, and finally, another 1x1 convolution layer is used to expand the depth of the feature map to four times the original output channels, 256, 512, 1024, and 2048 respectively. By introducing the concept of residual learning, the network can effectively avoid the gradient vanishing or exploding problems while increasing the depth, thereby improving the learning efficiency and the expressiveness of the model. For the motion branch, its residual layer uses 3, 4, 6, and 3 basic residual learning blocks, and the output channels are 64, 128, 256, and 512 respectively. The step sizes of these four residual layers are set to 2, 2, 1, and 1 in the two sub-networks.

5. The video smoke salient target detection method based on weakly supervised learning according to claim 4 is characterized in that: The spatial pyramid pooling module processes input features through five parallel layers, including a 1×1 point convolution layer, three 3×3 dilated convolution layers with different dilation rates, and a global average pooling layer; the outputs of these five parallel layers are concatenated in the depth dimension to generate the final feature map; the three different dilation rates are 12, 24, and 36 respectively.

6. The video smoke salient target detection method based on weakly supervised learning according to claim 4 is characterized in that: The decoder restores the spatial size of the feature map by fusing high-level and low-level features to accurately predict a high-resolution saliency map with clear object boundaries; the high-level output features of the spatial pyramid pooling module are reduced to 256-channel features through a 1×1 convolution in the decoder, and the low-level features from the residual layer-1 are reduced to 48-channel features through another 1×1 convolution; after concatenating the low-level features with the high-level features, two 3×3 convolutions with an output channel of 256 are performed immediately; then, a 1×1 convolution is performed and a Sigmoid function is connected to predict the final single-channel saliency map. The decoder of the motion branch adopts the same three-layer structure as convolution-{3-5}.

7. The video smoke salient target detection method based on weakly supervised learning according to any one of claims 4 to 6, characterized in that: In step S3, the appearance branch and the motion branch are integrated with the gated attention module; the six gated attention modules are represented as gate-i, i ∈ {0, 1, 2, 3, 4, 5}; gate-0 receives the output of the two head convolutional layers in the appearance subnetwork and the motion subnetwork as input; gate-{1-4} receives the output of the residual layer-i in the two branches as input; the motion input of gate-5 is the single-channel motion saliency map output by the decoder , its structure is different from the gated-{0-4} structure, specifically: directly convert the motion saliency map and appearance features in the appearance branch decoder Each channel slice is element-wise multiplied and then combined with the appearance feature Added together, the output of the final gate-5 is passed into one of the two 3×3 convolutions with an output channel of 256 in the appearance sub-network, and the formula is as follows: （2） in, is the appearance feature of input gate-5, is the single-channel motion saliency map output by the decoder, represents element-wise multiplication, is the output feature of gate-5.

8. The video smoke salient target detection method based on weakly supervised learning according to claim 7 is characterized in that: In the gated attention module, the appearance features and motion features of different levels and scales are simply spliced ​​in the channel dimension, and the spliced ​​fusion features are used indicates; then In the input gated attention module, the gated attention module is used to control the importance of appearance features and motion features, indicating that the network model independently adjusts the degree of dependence on image features and optical flow features, as shown in formula (3), where represents the global average pooling operation, express Activation function, represents the convolution operation, Indicates that according to the fusion features after splicing The generated gating signals are used as learned weights to adjust the importance of different features in subsequent processing stages. Represents the output features of the gated attention module; （3） The concatenated features are passed to a convolutional layer to generate the gating signal. The convolutional layer in formula (3) sets 128, 320, 640, 1280 and 2560 input channels and 2 output channels in the five gated attention modules respectively. After calculation, a global average pooling operation is performed, and finally a separate appearance gating signal is separated from the gating signal tensor. and optical flow gating signal Used to weight the original appearance features and motion characteristics , the original appearance features and motion characteristics They are respectively multiplied element by element with their gated signals to achieve feature weighting. The gated attention module can dynamically adjust the appearance and optical flow features, so that the model can process information more flexibly according to the current input. The weighted features are used and express; Weighted optical flow features Apply 1x1 convolution of the spatial dimension to compress the input channel into a single-channel output and process it through the sigmoid function to obtain a spatial attention map. Then, multiply this spatial attention map with the weighted appearance features to achieve spatial attention weighting. The formula is as follows, where represents the appearance features after spatial attention weighting, represents the convolution operation, Represents the activation function: （4） The appearance features after spatial attention enhancement are average pooled to generate a feature vector, which is then processed by a 1x1 convolution in the channel dimension, and the Softmax activation function is applied in the channel dimension. Then, it is multiplied by the number of channels C to ensure that the sum of the weights remains unchanged to generate channel attention weights. Finally, these weights are applied to the appearance features after spatial attention enhancement to achieve channel attention weighting. The appearance features after channel attention enhancement are combined with the appearance features after gated weighting. Add element by element to generate the final enhanced features. The formula is as follows, where is the number of channels, represents the appearance features after weighting by spatial attention and channel attention, represents the appearance features after spatial attention weighting, represents the global average pooling operation, represents the convolution operation, Represents the activation function: （5）。 9. According to the video smoke salient target detection method based on weakly supervised learning according to claim 1, the first part of the loss function of step S4 is as shown in formula (6), wherein represents a pseudo label, Represents the prediction result, represents the average pooling operation, represents the binary cross entropy loss, and Indicates the height and width of the image, Represents the total number of samples in the batch during training, and Represents the pixel index in the height and width directions of the image, and n represents the index value; （6） GLoss is introduced to evaluate the overall similarity of two samples. The formula is as follows: （7） The numerator of the fraction in formula (7) is the true class and the predicted result. The pixels and pseudo labels of the image The denominator represents the prediction result. and pseudo labels Total number of Zhongzheng categories, represents the smoothing term. The larger the value of the fractional part of formula (7), the more similar it is. Subtracting the fractional part of formula (7) from 1 represents the degree of global difference. Weight WLoss and GLoss to generate the final WGLoss. The formula is as follows, where: The weight for setting is set to 0.8 according to the empirical value: （8）。

Images