Pump room smoke and fire detection method and system based on multi-task deep learning

Abstract

The invention discloses a pump room smoke and fire detection method and system based on multi-task deep learning, and effectively solves the smoke and fire detection problem under the condition of small samples, the method adopts a U-Net network architecture, uses a ResNet network as an encoder backbone network, and a decoding end is connected with three independent decoder branch networks; the method is used for semantic segmentation of a complete firework area, semantic segmentation of a core firework area and binary classification tasks of existence of fireworks, and the detection effect of the fireworks is improved through multi-task learning. A test result on a pump room smoke and fire data set shows that compared with a traditional single-task detection method, the multi-task learning method provided by the invention has a better smoke and fire detection effect. In addition, the overall performance is further improved through weight configuration obtained through parameter searching, and all performance indexes are improved by 1%-3% compared with a single-task method.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and deep learning technology, and in particular to a method and system for detecting smoke and fire in pump rooms based on multi-task deep learning. Background Technology

[0002] As a critical infrastructure, the safety monitoring of pump stations is of paramount importance. With the development of industrial automation and intelligence, smoke and fire detection technology in pump stations plays an increasingly important role in ensuring safe production and environmental protection. Traditional smoke and fire detection methods mainly rely on hardware devices such as smoke sensors and temperature sensors. These methods suffer from problems such as slow response speed, high false alarm rate, and high maintenance costs, making it difficult to meet the requirements of pump station safety monitoring.

[0003] In recent years, with the development of computer vision and deep learning technologies, image-based smoke detection methods have gradually become a research hotspot. This method utilizes image recognition technology to accurately detect smoke by capturing and analyzing image information within the pumping station. Compared to traditional methods, image-based smoke detection has higher sensitivity and accuracy, and relatively lower maintenance costs.

[0004] However, traditional image-based smoke detection methods mainly employ a single-task learning framework, meaning the model is trained and optimized for a specific task. This single-task learning approach often suffers from overfitting and poor generalization ability when dealing with small sample datasets.

[0005] In recent years, multi-task deep learning has gained increasing attention. This method, through information sharing, offers advantages such as improved model performance, enhanced generalization capabilities, and reduced computational complexity, leading to its successful applications in industrial and medical image processing fields. The core idea of ​​multi-task learning is an inferential transfer learning method. The main task uses domain-related information from the training signals of related tasks as inference biases to improve its generalization performance. Multi-task learning involves multiple related tasks learning in parallel simultaneously, with gradients propagating backwards. These tasks mutually aid each other through low-level sharing, enhancing generalization. In short, multi-task learning combines multiple related tasks for learning. During the learning process, these tasks share and complement each other's learned domain-related information through a shallow sharing layer, mutually promoting learning and improving generalization.

[0006] Applications based on multi-task deep learning require a thorough understanding of the problem and targeted design of task networks to achieve good performance. How to design a multi-task network tailored to the morphological characteristics of smoke and fire in pumping stations to improve smoke and fire detection performance is a problem that needs to be solved by those skilled in the art. Summary of the Invention

[0007] The purpose of this invention is to provide a method and system for detecting smoke and fire in pump rooms based on multi-task deep learning. By utilizing the advantages of multi-task deep learning, it aims to improve the accuracy of smoke and fire detection, while effectively avoiding the overfitting problem that traditional single-task learning methods are prone to when dealing with small sample datasets.

[0008] To achieve the above objectives, the present invention adopts the following technical solution: The first aspect of this application provides a method for detecting smoke and fire in pump rooms based on multi-task deep learning, including the following steps: A multi-task deep network model based on the U-Net architecture is constructed. This model includes a shared encoder network and three independent decoder branch networks: the first decoder branch network, the second decoder branch network, and the third decoder branch network. The encoder network includes multiple encoding modules for extracting multi-level feature representations of the input image, which are shared by all decoder branch networks; The first decoder branch network receives the feature representation output from the encoder network, performs semantic segmentation of the complete fireworks region, and outputs the segmentation result representing the complete fireworks region. The second decoder branch network receives the feature representation output from the encoder network, performs semantic segmentation of the core fireworks region, and outputs the segmentation result representing the core fireworks region. The third decoder branch network receives the feature representation output from the encoder network, performs a binary classification task to determine whether there is smoke or fire, and outputs the result of whether there is smoke or fire. A multi-source training dataset containing smoke and flames is used to train the constructed multi-task deep network model. During training, a multi-task loss function is constructed, which is a weighted sum of the losses of each decoding branch task. The weight coefficients of the multi-task loss function are determined through parameter search. a. The loss function for both the complete fireworks region semantic segmentation task and the core fireworks region semantic segmentation task adopts a combination of cross-entropy loss function and Focal loss function; b. The cross-entropy loss function is used for the binary classification task involving smoke or no smoke. The pre-processed pump room monitoring video or image data is input into the trained multi-task deep network model, which outputs in real time the segmentation results of the complete smoke and fire area, the segmentation results of the core smoke and fire area, and the binary classification results of whether there is smoke and fire.

[0009] In a preferred embodiment, the encoder network adopts a ResNet network architecture as the backbone network and includes multiple encoding modules for extracting features from the input image. The output of the encoder network undergoes feature fusion and upsampling through multiple decoder branch networks. Each decoder branch network includes multiple decoding modules, which contain upsampling layers, convolutional layers, batch normalization layers, and activation function layers. The output of the decoder branch network is concatenated with the feature maps of the corresponding encoding modules in the encoder network through skip connections to achieve feature information fusion. The output of the first decoder branch network is processed by convolutional layers and activation functions to generate the output for the complete fireworks region segmentation task. The output of the second decoder branch network is processed by convolutional layers and activation functions to generate the output for the fireworks core region segmentation task. The output of the third decoder branch network extracts global features through global average pooling, and then generates the output of the binary classification task of whether there are smoke or not through linear layers and activation functions.

[0010] In the above context, the core area of ​​a pyrotechnic fire refers to the area where smoke and flames are most concentrated and visible during a pyrotechnic fire, usually located at the center of the fire; the complete pyrotechnic fire area refers to the entire area covered by smoke and flames during a pyrotechnic fire, including the core area and the surrounding smoke diffusion area.

[0011] In a preferred embodiment, the multi-source training dataset is generated in the following manner: a. Artificially set up smoke and fire in a simulated pump room scenario, collect video data, and extract frames to mark the smoke area; b. Clean and label flame images that match the pump room scene from the open-source dataset.

[0012] In a preferred embodiment, the specific steps of the parameter search include: With a fixed learning rate of 0.0001 and a batch size of 8, the loss weights of each task are adjusted in steps of 0.1 within the range of 0-1.5 using a grid search method to select the weight combination that optimizes the overall quality index Q of the validation set.

[0013] In a preferred embodiment, constructing the multi-task loss function specifically includes the following steps: Loss functions are constructed for semantic segmentation tasks of complete fireworks regions, core fireworks regions, and binary classification tasks based on the presence or absence of fireworks, respectively; among them, The loss function for semantic segmentation of complete fireworks regions is constructed using cross-entropy loss and Focal loss functions. L 1. Loss Function L 1 is represented as: ,in, , , In the formula, L CE Represents the cross-entropy loss function. y i Indicates the tag value. This represents the probability distribution of the predicted values; L Focal This represents the Focal loss function, where α is the weight assigned to the loss of samples from different classes. If there are few positive samples, the weight of the loss of positive samples is increased. γ is the adjustment factor for easy / hard samples, and γ is a hyperparameter that adjusts the weights of easy and hard samples. The role of γ is to adjust the sample prediction value. p t When the sample size is relatively large, that is, the easily separable sample, the loss value will be very small. In this way, the loss of easily separable samples will be significantly reduced, and the model will focus more on optimizing the loss of difficult samples. The loss function for semantic segmentation of the core region of fireworks is constructed using the cross-entropy loss function and the Focal loss function. L 2. Loss Function L 2= L 1; A loss function for a binary classification task involving smoke or no smoke is constructed using the binary cross-entropy loss function. L 3. Loss Function L 3 is represented as: in, y i Indicates sample i The labels are 1 for positive classes and 0 for negative classes. p i Indicates sample i The probability of predicting it as positive; The total loss function for the three tasks is constructed as follows: In the formula, λ1, λ2, and λ3 are the loss weights for the semantic segmentation task of the complete fireworks region, the semantic segmentation task of the core fireworks region, and the binary classification task of whether there are fireworks, respectively.

[0014] In a preferred embodiment, the detection method further includes: evaluating the smoke detection results of the multi-task deep network model using multiple evaluation metrics, including at least the mean intersection-union ratio (mIOU), correctness (COR), integrity (COM), and overall quality (Q); and fine-tuning the multi-task deep network model based on the evaluation results; wherein, The intersection over union (mIOU) ratio is expressed as: in,p ij The actual value is indicated. i It is predicted to be j The number of pixels, k +1 represents the number of categories (including the background category). p ii It's the actual quantity. p ij , p ii These respectively represent false positives and false negatives; Correctness of Reaction (COR) is the ratio of the correct detection value to the detection result in the area of ​​the firework, expressed as: Integrity COM is the ratio of the detected value to the marked value in the area of ​​smoke and fire, expressed as: Quality Q is a comprehensive metric calculated using both integrity (COM) and correctness (COR), and is expressed as: in, p ii , p ij and p ji These represent pixels that indicate true positives, false positives, and false negatives, respectively.

[0015] The second aspect of this application provides a pump room smoke and fire detection system based on multi-task deep learning, which, based on the pump room smoke and fire detection method based on multi-task deep learning described in the first aspect, includes: The dataset building module is used to build a multi-source training dataset containing smoke and flames; The model building module is used to construct a multi-task deep network model based on the U-Net architecture. This multi-task deep network model includes a shared encoder network and three independent decoder branch networks: a first decoder branch network, a second decoder branch network, and a third decoder branch network. The encoder network includes multiple encoding modules for extracting multi-level feature representations of the input image, which are shared by all decoder branch networks; The first decoder branch network receives the feature representation output from the encoder network, performs semantic segmentation of the complete fireworks region, and outputs the segmentation result representing the complete fireworks region. The second decoder branch network receives the feature representation output from the encoder network, performs semantic segmentation of the core fireworks region, and outputs the segmentation result representing the core fireworks region. The third decoder branch network receives the feature representation output from the encoder network, performs a binary classification task to determine whether there is smoke or fire, and outputs the result of whether there is smoke or fire. The training module uses a multi-source training dataset containing smoke and flames to train the constructed multi-task deep network model. During training, a multi-task loss function is constructed, which is a weighted sum of the losses of each decoding branch task. The weight coefficients of the multi-task loss function are determined through parameter search. Specifically, the loss functions for the semantic segmentation tasks of complete smoke and fire regions and the semantic segmentation tasks of core smoke and fire regions both adopt a combination of cross-entropy loss function and Focal loss function. The loss function for the binary classification task with and without smoke and fire adopts cross-entropy loss function. The detection module is used to receive monitoring video or image data from the pump room after the model training is completed, and to perform smoke and fire detection through the trained multi-task deep network model, outputting the complete smoke and fire area, the core smoke and fire area, and the judgment result of whether there is smoke and fire.

[0016] In a preferred embodiment, the multi-source training dataset includes smoke and fire data from a simulated pump room scene, as well as images and labels similar to the pump room scene obtained by cleaning from an open-source dataset.

[0017] In a preferred embodiment, the encoder network uses ResNet34 as the backbone network.

[0018] In a preferred embodiment, the model building module includes a loss function definition unit for defining the loss function for the semantic segmentation task of the complete fireworks region. L 1. Loss function for semantic segmentation of the core region of fireworks L 2. Loss function for binary classification tasks with and without smoke. L 3, and based on L 1. L 2 and L 3. Construct the total loss function for the three tasks; where, The loss function for semantic segmentation of complete fireworks regions is constructed using cross-entropy loss and Focal loss functions. L 1. Loss Function L 1 is represented as: ,in, , , In the formula, L CE Represents the cross-entropy loss function. y i Indicates the tag value. This represents the probability distribution of the predicted values; LFocal This represents the Focal loss function, where α is the weight assigned to the loss of samples from different classes. If there are few positive samples, the weight of the loss of positive samples is increased. γ is the adjustment factor for easy / hard samples, and γ is a hyperparameter that adjusts the weights of easy and hard samples. The role of γ is to adjust the sample prediction value. p t When the sample size is relatively large, that is, the easily separable sample, the loss value will be very small. In this way, the loss of easily separable samples will be significantly reduced, and the model will focus more on optimizing the loss of difficult samples. The loss function for semantic segmentation of the core region of fireworks is constructed using the cross-entropy loss function and the Focal loss function. L 2. Loss Function L 2= L 1; A loss function for a binary classification task involving smoke or no smoke is constructed using the binary cross-entropy loss function. L 3. Loss Function L 3 is represented as: in, y i Indicates sample i The labels are 1 for positive classes and 0 for negative classes. p i Indicates sample i The probability of predicting it as positive; The total loss function for the three tasks is constructed as follows: In the formula, λ1, λ2, and λ3 are the loss weights for the semantic segmentation task of the complete fireworks region, the semantic segmentation task of the core fireworks region, and the binary classification task of whether there are fireworks, respectively.

[0019] In a preferred embodiment, the system further includes: an evaluation module, used to evaluate the performance of the trained multi-task deep network model using a test set after training is completed, wherein the evaluation metrics include mean intersection-union ratio (mIOU), correctness (COR), integrity (COM), and overall quality (Q); wherein... The intersection over union (mIOU) ratio is expressed as: in, p ij The actual value is indicated. i It is predicted to be j The number of pixels, k +1 represents the number of categories (including the background category). p ii It's the actual quantity. p ij , pii These respectively represent false positives and false negatives; Correctness of Reaction (COR) is the ratio of the correct detection value to the detection result in the area of ​​the firework, expressed as: Integrity COM is the ratio of the detected value to the marked value in the area of ​​smoke and fire, expressed as: Quality Q is a comprehensive metric calculated using both integrity (COM) and correctness (COR), and is expressed as: in, p ii , p ij and p ji These represent pixels that indicate true positives, false positives, and false negatives, respectively.

[0020] A third aspect of this application provides a computer device including a processor and a memory, the memory being used to store at least one computer program, the at least one computer program being loaded by the processor and executing the steps of the pump room smoke and fire detection method based on multi-task deep learning described in the first aspect of this application.

[0021] The fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the pump room smoke and fire detection method based on multi-task deep learning described in the first aspect of this application.

[0022] Compared with the prior art, the technical solution of the present invention has the following beneficial effects: To address the need for smoke and fire detection in pumping stations, this application proposes a method and system for smoke and fire detection in pumping stations based on multi-task deep learning. This method effectively solves the problem of smoke and fire detection under small sample conditions. The method adopts a U-Net network architecture and uses ResNet as the encoder backbone network. The decoder is connected to three task-related networks, including a semantic segmentation task of the complete smoke and fire region, a semantic segmentation task of the core smoke and fire region, and a binary classification task of whether there is smoke or fire. Multi-task learning improves the detection effect of smoke and fire.

[0023] Test results on the pump room smoke dataset show that multi-task learning performs better in smoke detection than the single-task model. The weight configuration obtained through parameter search further improves performance, with performance metrics improving by 1%–3% compared to the single-task model.

[0024] In summary, the pump room smoke and fire detection method and system based on multi-task deep learning proposed in this invention have significant beneficial effects in solving the problem of small sample detection and improving detection accuracy and robustness, which is of great significance for ensuring the safety of pump rooms. Attached Figure Description

[0025] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0026] Figure 1 This is a flowchart of a pump room smoke and fire detection method based on multi-task deep learning, as shown in Embodiment 1 of the present invention. Figure 2 These are examples of images from the multi-source training dataset used in this embodiment of the invention, wherein... Figure 2 (a) is an example of artificially simulated data. Figure 2 (b) is an example of data cleaned from an open-source dataset; Figure 3 This is a diagram of the multi-task deep network model structure of the pump room smoke and fire detection method based on multi-task deep learning described in this invention. Figure 4 This is a schematic diagram illustrating the detection effect of the pump room smoke and fire detection method based on multi-task deep learning described in this invention. Detailed Implementation

[0027] To make the above and other features and advantages of the present invention clearer, the invention will be further described below with reference to the accompanying drawings. It should be understood that the specific embodiments given herein are for the purpose of explanation to those skilled in the art and are exemplary only, not restrictive.

[0028] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0029] Example 1: In recent years, multi-task deep learning has received increasing attention. This method, through information sharing, has the advantages of improving performance, enhancing generalization performance, and reducing computational load, and has been well applied in industrial and medical image processing fields.

[0030] Multi-task learning is an inferential transfer learning method where the main task uses domain-related information from the training signals of related tasks as inference biases to improve the generalization performance of the main task. Multi-task learning involves multiple related tasks learning in parallel simultaneously, with gradients backpropagating concurrently. These tasks learn from each other through a low-level shared representation, enhancing generalization. In simple terms: multi-task learning combines multiple related tasks for learning, using a shallow shared representation to share and supplement the domain-related information learned, mutually promoting learning and improving generalization.

[0031] Applications based on multi-task deep learning require a thorough understanding of the problem and targeted design of task networks to achieve good performance. This embodiment, taking into account the morphological characteristics of smoke and fire in pumping stations, employs a multi-task learning framework, combining the U-Net network architecture and the ResNet34 encoder backbone network, to achieve effective detection of smoke and fire in pumping stations.

[0032] Figure 1 This is a flowchart of a pump room smoke and fire detection method based on multi-task deep learning.

[0033] like Figure 1 As shown, a method for detecting smoke and fire in pump rooms based on multi-task deep learning includes the following specific steps: Step 1: Data preparation.

[0034] Since the possibility of fireworks occurring in real-world scenarios is extremely small, the dataset used in this embodiment is derived from artificial simulation and cleaned from open-source datasets.

[0035] Artificial simulation: In a scenario similar to the pump room, smoke and fire are artificially set up, video data is recorded using surveillance cameras, and frames are extracted into a dataset, mainly collecting smoke data.

[0036] Cleaning from open-source datasets: Images and markers similar to pump room scenes were collected and cleaned from the BoWFire dataset (best of both worlds fire detection), with a focus on collecting flame data.

[0037] Figure 2 These are examples of images from the multi-source training dataset used in this embodiment of the invention. Wherein: Figure 2 (a) is an example of data simulated by humans. This data is mainly used to collect smoke data to simulate possible smoke and fire situations in the pump room.

[0038] Figure 2(b) is an example of data cleaned from an open-source dataset. This data was primarily used to collect flame data to supplement and improve the training dataset for pump room smoke detection.

[0039] Based on the requirements for smoke and fire detection in pump rooms, training sets, test sets, and validation sets are prepared. In this embodiment, the dataset partitioning ratio and the number of sets are shown in Table 1: Table 1. Number of training, testing, and validation sets Sample data usage ratio training set test set Validation set 25% 258 113 115 50% 513 113 115 75% 778 113 115 100% 1028 113 115 These datasets contain images of fireworks in pumping stations, along with their corresponding labels, for model training and validation.

[0040] Step 2: Construct a multi-task deep network model.

[0041] U-Net is a fully convolutional network architecture that uses sequential encoder and decoder blocks to perform pixel-level, dense predictions on the input image. Its structure is simple yet highly effective. The encoder is responsible for feature extraction, and the decoder combines feature information through 2x2 upconvolutions. Originally designed to solve semantic segmentation problems in biomedical images, U-Net has also been widely applied in various areas of semantic segmentation, such as satellite image segmentation and industrial defect detection.

[0042] This embodiment refers to the classic U-Net structure and constructs a multi-task network model. For example... Figure 3 As shown, this multi-task deep network model includes a shared encoder network and three independent decoder branches. Specifically, these three decoder branches are a first decoder branch for extracting the complete smoke region, a second decoder branch for extracting the core smoke region, and a third decoder branch for binary classification tasks involving the presence or absence of smoke.

[0043] The encoder network employs the ResNet34 architecture, comprising multiple encoding modules for extracting multi-level feature representations of the input image. These feature representations are shared by all decoder branches to support multi-task learning.

[0044] Specifically, at the input of the encoder network, the input image first passes through a 7x7 convolutional layer (conv7x7) for initial feature extraction. Subsequently, the feature map is normalized by a batch normalization (bn) layer and then non-linear characteristics are introduced via a ReLU activation function layer. Next, the feature map is downsampled by a max-pooling layer to reduce computation and capture more global features. After this, the image features are sequentially processed through multiple encoding modules for in-depth feature extraction. Each encoding module contains a convolutional layer (conv3x3), a batch normalization (bn) layer, a ReLU activation function layer, and residual connections (add). The introduction of residual connections aims to alleviate the vanishing gradient problem during deep network training and promote efficient feature propagation.

[0045] The connection between the decoder and encoder networks is achieved through skip connections. Each decoding module receives feature maps from the encoder module and processes them through upsampling (x2) and convolutional layers (conv 3x3). The feature maps of the decoding module and the corresponding encoder module are concatenated to fuse feature information from different levels.

[0046] In this embodiment, the first decoder branch network receives the feature representation output from the encoder network and processes it through a convolutional layer (conv 3x3) and an activation function (softmax), outputting a segmentation result representing the complete fireworks region, used for the semantic segmentation task of the complete fireworks region; the second decoder branch network receives the feature representation output from the encoder network and processes it through a convolutional layer (conv 3x3) and an activation function (softmax), outputting a segmentation result representing the core fireworks region, used for the semantic segmentation task of the core fireworks region; the third decoder branch network determines whether there is fireworks in the input image, and its output is processed through global average pooling (avgpool), a linear layer (linear), and an activation function (sigmoid), outputting a binary classification result representing the presence or absence of fireworks, used for the binary classification task of the presence or absence of fireworks.

[0047] Through this connection, the decoder network in this embodiment can make full use of the feature information extracted by the encoder network to achieve the tasks of complete region segmentation, core region segmentation, and binary classification of the presence or absence of smoke and fire, thereby improving the accuracy and efficiency of smoke and fire detection.

[0048] Step 3: Train a multi-task deep network model.

[0049] A multi-source training dataset containing smoke and flames was used to train the constructed multi-task deep network model. The dataset was preprocessed to ensure that the image quality met the model training requirements, and was divided into training, testing, and validation sets to facilitate model performance evaluation and tuning during training.

[0050] During training, a multi-task loss function is constructed. This function integrates the weighted losses of the first decoder branch network (performing the semantic segmentation task of the complete fireworks region), the second decoder branch network (performing the semantic segmentation task of the core fireworks region), and the third decoder branch network (performing the binary classification task of whether or not there is fireworks). To balance the contributions of different tasks to model training, adjustable weight coefficients are introduced, dynamically adjusting the weights according to task characteristics and complexity. The weight coefficients are determined through a parameter search method to ensure that the model achieves good performance on all tasks.

[0051] Specifically, the method for constructing the multi-task loss function includes the following steps: For the semantic segmentation task of complete fireworks regions, considering the imbalance between foreground and background in the data samples, this embodiment adopts a composite loss function. L 1. It consists of the cross-entropy loss function and the Focal loss function.

[0052] loss function L 1 is represented as: ,in, , , In the formula, L CE This represents the cross-entropy loss function. Cross-entropy describes the distance between two probability distributions, and therefore can be used to characterize the difference between the predicted and labeled values. y i Indicates the tag value. This represents the probability distribution of the predicted values.

[0053] In the formula, L Focal This represents the Focal loss function, a variant of the cross-entropy loss function proposed by Kaiming He's team to address imbalanced training samples and varying sample difficulty. α assigns weights to the loss of different classes of samples; if there are fewer positive samples, the weight of the positive sample loss is increased. γ is the adjustment factor for easy / hard samples, and γ is a hyperparameter that adjusts the weights of easy and hard samples. The role of γ is to adjust the sample prediction value. p tWhen the sample size is relatively large, i.e., the easily separable sample, the loss value will be very small. This will significantly reduce the loss of easily separable samples, and the model will focus more on optimizing the loss of difficult-to-separate samples.

[0054] The semantic segmentation task of the core region of fireworks and the semantic segmentation task of the complete region of fireworks share similar foreground-background imbalance characteristics, and their loss functions... L 2. It also adopts the same composite loss function as the semantic segmentation task for complete fireworks regions. That is, the loss function... L 2= L 1.

[0055] In the binary classification task without smoke, the loss function L 3 represents the cross-entropy loss function. The model ultimately needs to predict two possible outcomes, with the predicted probability for each class being... p and 1- p。 loss function L 3 is represented as: in, y i Indicates sample i The labels are 1 for positive classes and 0 for negative classes. p i Indicates sample i The probability of predicting it as positive; In summary, the total loss function for the three tasks is as follows: In the formula, λ1, λ2, and λ3 are the loss weights for the semantic segmentation task of the complete fireworks region, the semantic segmentation task of the core fireworks region, and the binary classification task of whether there are fireworks, respectively.

[0056] Step 4: Use evaluation metrics to assess the performance of the multi-task deep network model.

[0057] After the model training is completed, the performance of the multi-task deep network model is evaluated using the following metrics: 1) A commonly used evaluation metric in semantic segmentation is the mean Intersection over Union (mIOU): in, p ij The actual value is indicated. i It is predicted to be j The number of pixels, k +1 represents the number of categories (including the background category). p ii It's the actual quantity. p ij , p iiThese represent false positives and false negatives, respectively.

[0058] 2) Correctness (COR), Completeness (COM), and Overall Quality (Q). COR is the ratio of correctly detected values ​​to the detected values ​​in the firework area. COM is the ratio of the detected values ​​in the firework area to the values ​​in the marked areas. Quality (Q) is a comprehensive metric that can be calculated using COM and COR.

[0059] Correctness (COR) is expressed as: Integrity (COM) is represented as: Quality (Q) is expressed as: in, p ii , p ij and p ji These represent pixels that indicate true positives, false positives, and false negatives, respectively.

[0060] Step S5: Apply the trained multi-task deep network model to detect smoke and fire.

[0061] The preprocessed pump room monitoring video or image data is input into the trained multi-task deep network model. The multi-task deep network model outputs the segmentation results of the complete smoke and fire area, the segmentation results of the core smoke and fire area, and the binary classification results of whether there is smoke and fire based on the input data.

[0062] In one embodiment, such as Figure 4 The diagram shows the detection effect of a pump room smoke and fire detection method based on multi-task deep learning. Figure 4 In the image, the red markers indicate the complete area of ​​the fireworks, while the blue markers indicate the core area of ​​the fireworks.

[0063] Example 2: Based on the same design concept as Embodiment 1, this embodiment also provides a pump room smoke and fire detection system based on multi-task deep learning, specifically including: The dataset building module is used to build a multi-source training dataset containing smoke and flames; The model building module is used to construct a multi-task deep network model based on the U-Net architecture. This multi-task deep network model includes a shared encoder network and three independent decoder branch networks: a first decoder branch network, a second decoder branch network, and a third decoder branch network. The encoder network includes multiple encoding modules for extracting multi-level feature representations of the input image, which are shared by all decoder branch networks; The first decoder branch network receives the feature representation output from the encoder network, performs semantic segmentation of the complete fireworks region, and outputs the segmentation result representing the complete fireworks region. The second decoder branch network receives the feature representation output from the encoder network, performs semantic segmentation of the core fireworks region, and outputs the segmentation result representing the core fireworks region. The third decoder branch network receives the feature representation output from the encoder network, performs a binary classification task to determine whether there is smoke or fire, and outputs the result of whether there is smoke or fire. The training module uses a multi-source training dataset containing smoke and flames to train the constructed multi-task deep network model. During training, a multi-task loss function is constructed, which is a weighted sum of the losses of each decoding branch task. The weight coefficients of the multi-task loss function are determined through parameter search. Specifically, the loss functions for the semantic segmentation tasks of complete smoke and fire regions and the semantic segmentation tasks of core smoke and fire regions both adopt a combination of cross-entropy loss function and Focal loss function. The loss function for the binary classification task with and without smoke and fire adopts cross-entropy loss function. The detection module is used to receive monitoring video or image data from the pump room after the model training is completed, and to perform smoke and fire detection through the trained multi-task deep network model, outputting the complete smoke and fire area, the core smoke and fire area, and the judgment result of whether there is smoke and fire.

[0064] The encoder network uses ResNet34 as its backbone network.

[0065] In a preferred embodiment, the model building module includes a loss function definition unit for defining the loss function for the semantic segmentation task of the complete fireworks region. L 1. Loss function for semantic segmentation of the core region of fireworks L 2. Loss function for binary classification tasks with and without smoke. L 3, and based on L 1. L 2 and L 3. Construct the total loss function for the three tasks; where, The loss function for semantic segmentation of complete fireworks regions is constructed using cross-entropy loss and Focal loss functions. L 1. Loss Function L 1 is represented as: ,in, , , In the formula, L CE Represents the cross-entropy loss function. y i Indicates the tag value. This represents the probability distribution of the predicted values; L Focal This represents the Focal loss function, where α is the weight assigned to the loss of samples from different classes. If there are few positive samples, the weight of the loss of positive samples is increased. γ is the adjustment factor for easy / hard samples, and γ is a hyperparameter that adjusts the weights of easy and hard samples. The role of γ is to adjust the sample prediction value. p t When the sample size is relatively large, that is, the easily separable sample, the loss value will be very small. In this way, the loss of easily separable samples will be significantly reduced, and the model will focus more on optimizing the loss of difficult samples. The loss function for semantic segmentation of the core region of fireworks is constructed using the cross-entropy loss function and the Focal loss function. L 2. Loss Function L 2= L 1; A loss function for a binary classification task involving smoke or no smoke is constructed using the binary cross-entropy loss function. L 3. Loss Function L 3 is represented as: in, y i Indicates sample i The labels are 1 for positive classes and 0 for negative classes. p i Indicates sample i The probability of predicting it as positive; The total loss function for the three tasks is constructed as follows: In the formula, λ1, λ2, and λ3 are the loss weights for the semantic segmentation task of the complete fireworks region, the semantic segmentation task of the core fireworks region, and the binary classification task of whether there are fireworks, respectively.

[0066] In a preferred embodiment, the system further includes: an evaluation module, used to evaluate the performance of the trained multi-task deep network model using a test set after training is completed, wherein the evaluation metrics include mean intersection-union ratio (mIOU), correctness (COR), integrity (COM), and overall quality (Q); wherein... The intersection over union (mIOU) ratio is expressed as: in, p ij The actual value is indicated. i It is predicted to bej The number of pixels, k +1 represents the number of categories (including the background category). p ii It's the actual quantity. p ij , p ii These respectively represent false positives and false negatives; Correctness of Reaction (COR) is the ratio of the correct detection value to the detection result in the area of ​​the firework, expressed as: Integrity COM is the ratio of the detected value to the marked value in the area of ​​smoke and fire, expressed as: Quality Q is a comprehensive metric calculated using both integrity (COM) and correctness (COR), and is expressed as: in, p ii , p ij and p ji These represent pixels that indicate true positives, false positives, and false negatives, respectively.

[0067] On the other hand, this embodiment also provides a computer device, which includes a processor and a memory. The memory is used to store at least one computer program, which is loaded by the processor and executes the steps of the pump room smoke detection method based on multi-task deep learning described in Embodiment 1.

[0068] On the other hand, this embodiment also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the pump room smoke and fire detection method based on multi-task deep learning described in Embodiment 1.

[0069] Experimental verification: 1. Experimental Setup This application designs the following experiments on a pump room fireworks dataset to verify the effectiveness of the proposed method.

[0070] First, a single-task semantic segmentation method was used for smoke detection, and the results were compared with those of multi-task experiments. Then, multi-task experiments were conducted, including extraction of the complete smoke region, extraction of the smoke core region, and ablation experiments for binary classification tasks involving the presence or absence of smoke, testing the ability of different tasks to improve smoke detection accuracy. To avoid certain tasks dominating the gradient descent algorithm and failing to fully optimize other tasks, different loss weights were set for each single task through parameter search. Finally, experiments were conducted with datasets of different proportions to highlight the method's advantage in small samples: 25%, 50%, 75%, and 100%.

[0071] All experimental batch sizes were set to 8, the total number of training iterations was 100, the initial learning rate was set to 0.0001, the learning rate decay strategy was a Poly strategy with a power of 1.5, and the Adam optimizer was used for optimization during training. β 1 = 0.9 β 2=0.999, attenuation coefficient ω=0.00001. During training, the training loss of each task in step 3 of Example 1 is used to track the convergence state. When training is completed, the evaluation index of each task in step 4 of Example 1 is used to evaluate the performance of the trained model.

[0072] Before the multi-task experiment, a parameter search was performed on the task weights to determine the optimal weight combination. The learning rate for the parameter search was set to 0.0001, the number of training generations was 80, the batch size was 8, the step size was 0.10, and the range was from 0 to 1.5.

[0073] Experimental Results In this experiment, a comparison experiment was conducted between single-task and multi-task models using different data ratios. Table 2 shows the specific experimental results.

[0074] Table 2 shows the test results of the single-task model and the multi-task model on the fireworks test set. In the table above, Seg represents the semantic segmentation of the entire fire area, which is performed by the first decoder branch network; Cls represents the binary classification of the presence or absence of fire, which is performed by the third decoder branch network; and Cen represents the semantic segmentation of the core fire area, which is performed by the second decoder branch network.

[0075] From the experimental results data table above, the following conclusions can be drawn: Regardless of the data volume ratio, each task showed varying degrees of accuracy improvement after participating in multi-task learning, compared to single-task training results. This indicates that the multi-task learning framework can effectively integrate relevant information between different tasks, thereby improving the accuracy of smoke detection.

[0076] When the data volume ratio is 25%, the integrity (COM) of fireworks detection is relatively low compared to other data ratios, for the following reasons: 1) The amount of data is too small, which prevents the model from fully learning the complete features of fireworks.

[0077] 2) The fireworks sample is not representative. The limited sample size is insufficient to represent all the variations in fireworks, thus affecting the model's generalization ability.

[0078] However, with the increase in data volume, the multi-task model showed varying degrees of improvement in all performance metrics compared to the single-task model, especially in the completeness of fireworks detection, which was significantly enhanced. This further validates the effectiveness of the multi-task learning framework when there is sufficient data.

[0079] The weight configuration obtained through parameter search improved various performance metrics by 1–3 percentage points. This indicates that reasonable task weight allocation is crucial for the performance of multi-task learning frameworks.

[0080] In summary, this application presents a method and system for detecting smoke and fire in pumping stations based on multi-task deep learning. The method employs a U-Net basic network architecture, with ResNet as the encoder backbone. The decoder connects to three task-related networks: a network for detecting the complete smoke and fire region, a network for detecting the core smoke and fire region, and a binary classification network for determining the presence or absence of smoke and fire. Test results on a pumping station smoke and fire dataset demonstrate that the multi-task learning framework performs better in smoke and fire detection compared to a single-task model, especially when the data volume is sufficient and the task weights are configured appropriately. The weight configuration obtained through parameter search further improves performance, with performance metrics improving by 1%–3% compared to the single-task model.

[0081] The specific embodiments of the present invention have been described in detail above, but they are merely examples, and the present invention is not limited to the specific embodiments described above. For those skilled in the art, any equivalent modifications and substitutions to the present invention are also within the scope of the present invention. Therefore, all equivalent transformations and modifications made without departing from the spirit and scope of the present invention should be covered within the scope of the present invention.

Claims

1. A method for detecting smoke and fire in pump rooms based on multi-task deep learning, characterized in that, Includes the following steps: A multi-task deep network model based on the U-Net architecture is constructed. This model includes a shared encoder network and three independent decoder branch networks: the first decoder branch network, the second decoder branch network, and the third decoder branch network. The encoder network includes multiple encoding modules for extracting multi-level feature representations of the input image, which are shared by all decoder branch networks; The first decoder branch network receives the feature representation output from the encoder network, performs semantic segmentation of the complete fireworks region, and outputs the segmentation result representing the complete fireworks region. The second decoder branch network receives the feature representation output from the encoder network, performs semantic segmentation of the core fireworks region, and outputs the segmentation result representing the core fireworks region. The third decoder branch network receives the feature representation output from the encoder network, performs a binary classification task to determine whether there is smoke or fire, and outputs the result of whether there is smoke or fire. A multi-source training dataset containing smoke and flames is used to train the constructed multi-task deep network model. During training, a multi-task loss function is constructed, which is a weighted sum of the losses of each decoding branch task. The weight coefficients of the multi-task loss function are determined through parameter search. a. The loss function for both the complete fireworks region semantic segmentation task and the core fireworks region semantic segmentation task adopts a combination of cross-entropy loss function and Focal loss function; b. The cross-entropy loss function is used for the binary classification task of whether there is smoke or not; The pre-processed pump room monitoring video or image data is input into the trained multi-task deep network model, which outputs in real time the segmentation results of the complete smoke and fire area, the segmentation results of the core smoke and fire area, and the binary classification results of whether there is smoke and fire.

2. The method for detecting smoke and fire in pump rooms based on multi-task deep learning according to claim 1, characterized in that, The encoder network uses a ResNet network as its backbone network and includes multiple encoding modules for extracting features from the input image. The output of the encoder network undergoes feature fusion and upsampling through multiple decoder branch networks. Each decoder branch network includes multiple decoding modules, which contain upsampling layers, convolutional layers, batch normalization layers, and activation function layers. The output of the decoder branch network is concatenated with the feature maps of the corresponding encoding modules in the encoder network via skip connections to achieve feature information fusion. The output of the first decoder branch network is processed by convolutional layers and activation functions to generate the output for the complete fireworks region segmentation task. The output of the second decoder branch network is processed by convolutional layers and activation functions to generate the output for the fireworks core region segmentation task. The output of the third decoder branch network extracts global features through global average pooling, and then generates the output of the binary classification task of whether there are smoke or not through linear layers and activation functions.

3. The method for detecting smoke and fire in pump rooms based on multi-task deep learning according to claim 1, characterized in that, The multi-source training dataset was generated in the following way: a. Artificially set up smoke and fire in a simulated pump room scenario, collect video data, and extract frames to label the smoke areas; b. Clean and label flame images that match the pump room scene from the open-source dataset.

4. The method for detecting smoke and fire in pump rooms based on multi-task deep learning according to claim 1, characterized in that, The specific steps for parameter search include: With a fixed learning rate of 0.0001 and a batch size of 8, the loss weights of each task are adjusted in steps of 0.1 within the range of 0-1.5 using a grid search method to select the weight combination that optimizes the overall quality index Q of the validation set.

5. The method for detecting smoke and fire in pump rooms based on multi-task deep learning according to claim 1, characterized in that, The construction of the multi-task loss function specifically includes the following steps: Loss functions are constructed for semantic segmentation tasks of complete fireworks regions, core fireworks regions, and binary classification tasks based on the presence or absence of fireworks, respectively; among them, The loss function for semantic segmentation of complete fireworks regions is constructed using cross-entropy loss and Focal loss functions. L 1. Loss Function L 1 is represented as: ,in, ， ， In the formula, L CE Represents the cross-entropy loss function. y i Indicates the tag value. This represents the probability distribution of the predicted values; L Focal This represents the Focal loss function, where α is the weight assigned to the loss of samples from different classes. If there are few positive samples, the weight of the loss for positive samples is increased. γ is the adjustment factor for easy / hard samples, and γ is a hyperparameter that adjusts the weights of easy and hard samples. The role of γ is to adjust the sample prediction value. p t When the sample size is relatively large, that is, the easily separable sample, the loss value will be very small. In this way, the loss of easily separable samples will be significantly reduced, and the model will focus more on optimizing the loss of difficult samples. The loss function for semantic segmentation of the core region of fireworks is constructed using the cross-entropy loss function and the Focal loss function. L 2. Loss Function L 2= L 1; A loss function for a binary classification task involving smoke or no smoke is constructed using the binary cross-entropy loss function. L 3. Loss Function L 3 is represented as: in, y i Indicates sample i The labels are 1 for positive classes and 0 for negative classes. p i Indicates sample i The probability of predicting it as positive; The total loss function for the three tasks is constructed as follows: In the formula, λ1, λ2, and λ3 are the loss weights for the semantic segmentation task of the complete fireworks region, the semantic segmentation task of the core fireworks region, and the binary classification task of whether there are fireworks, respectively.

6. The method for detecting smoke and fire in pump rooms based on multi-task deep learning according to claim 1, characterized in that, The detection method further includes: evaluating the smoke detection results of the multi-task deep network model using multiple evaluation metrics, including at least the mean intersection-union ratio (mIOU), correctness (COR), integrity (COM), and overall quality (Q); and fine-tuning the multi-task deep network model based on the evaluation results; wherein, The intersection over union (mIOU) ratio is expressed as: in, p ij The actual value is indicated. i It is predicted to be j The number of pixels, k +1 represents the number of categories. p ii It's the actual quantity. p ij , p ii These respectively represent false positives and false negatives; Correctness of Reaction (COR) is the ratio of the correct detection value to the detection result in the area of ​​the firework, expressed as: Integrity COM is the ratio of the detected value to the marked value in the area of ​​smoke and fire, expressed as: Quality Q is a comprehensive metric calculated using both integrity (COM) and correctness (COR), and is expressed as: in, p ii , p ij and p ji These represent pixels that indicate true positives, false positives, and false negatives, respectively.

7. A pump room smoke and fire detection system based on multi-task deep learning, characterized in that, include: The dataset building module is used to build a multi-source training dataset containing smoke and flames; The model building module is used to construct a multi-task deep network model based on the U-Net architecture. This multi-task deep network model includes a shared encoder network and three independent decoder branch networks: a first decoder branch network, a second decoder branch network, and a third decoder branch network. The encoder network includes multiple encoding modules for extracting multi-level feature representations of the input image, which are shared by all decoder branch networks; The first decoder branch network receives the feature representation output from the encoder network, performs semantic segmentation of the complete fireworks region, and outputs the segmentation result representing the complete fireworks region. The second decoder branch network receives the feature representation output from the encoder network, performs semantic segmentation of the core fireworks region, and outputs the segmentation result representing the core fireworks region. The third decoder branch network receives the feature representation output from the encoder network, performs a binary classification task to determine whether there is smoke or fire, and outputs the result of whether there is smoke or fire. The training module uses a multi-source training dataset containing smoke and flames to train the constructed multi-task deep network model. During training, a multi-task loss function is constructed, which is a weighted sum of the losses of each decoding branch task. The weight coefficients of the multi-task loss function are determined through parameter search. Specifically, the loss functions for the semantic segmentation tasks of complete smoke and fire regions and the semantic segmentation tasks of core smoke and fire regions both adopt a combination of cross-entropy loss function and Focal loss function. The loss function for the binary classification task with and without smoke and fire adopts cross-entropy loss function. The detection module is used to receive monitoring video or image data from the pump room after the model training is completed, and to perform smoke and fire detection through the trained multi-task deep network model, outputting the complete smoke and fire area, the core smoke and fire area, and the judgment result of whether there is smoke and fire.

8. The pump room smoke and fire detection system based on multi-task deep learning according to claim 7, characterized in that, The model building module includes a loss function definition unit, used to define the loss function for the semantic segmentation task of the complete fireworks region. L 1. Loss function for semantic segmentation of the core region of fireworks L 2. Loss function for binary classification tasks with and without smoke. L 3, and based on L 1. L 2 and L 3. Construct the total loss function for the three tasks; where, The loss function for semantic segmentation of complete fireworks regions is constructed using cross-entropy loss and Focal loss functions. L 1. Loss Function L 1 is represented as: ,in, ， ， In the formula, L CE Represents the cross-entropy loss function. y i Indicates the tag value. This represents the probability distribution of the predicted values; L Focal This represents the Focal loss function, where α is the weight assigned to the loss of samples from different classes. If there are few positive samples, the weight of the loss for positive samples is increased. γ is the adjustment factor for easy / hard samples, and γ is a hyperparameter that adjusts the weights of easy and hard samples. The role of γ is to adjust the sample prediction value. p t When the sample size is relatively large, that is, the easily separable sample, the loss value will be very small. In this way, the loss of easily separable samples will be significantly reduced, and the model will focus more on optimizing the loss of difficult samples. The loss function for semantic segmentation of the core region of fireworks is constructed using the cross-entropy loss function and the Focal loss function. L 2. Loss Function L 2= L 1; A loss function for a binary classification task involving smoke or no smoke is constructed using the binary cross-entropy loss function. L 3. Loss Function L 3 is represented as: in, y i Indicates sample i The labels are 1 for positive classes and 0 for negative classes. p i Indicates sample i The probability of predicting it as positive; The total loss function for the three tasks is constructed as follows: In the formula, λ1, λ2, and λ3 are the loss weights for the semantic segmentation task of the complete fireworks region, the semantic segmentation task of the core fireworks region, and the binary classification task of whether there are fireworks, respectively.

9. The pump room smoke and fire detection system based on multi-task deep learning according to claim 7, characterized in that, The system further includes: an evaluation module, used to evaluate the performance of the trained multi-task deep network model using a test set after training is completed. Evaluation metrics include mean intersection-union ratio (mIOU), correctness (COR), integrity (COM), and overall quality (Q). The intersection over union (mIOU) ratio is expressed as: in, p ij The actual value is indicated. i It is predicted to be j The number of pixels, k +1 represents the number of categories. p ii It's the actual quantity. p ij , p ii These respectively represent false positives and false negatives; Correctness of Reaction (COR) is the ratio of the correct detection value to the detection result in the area of ​​the firework, expressed as: Integrity COM is the ratio of the detected value to the marked value in the area of ​​smoke and fire, expressed as: Quality Q is a comprehensive metric calculated using both integrity (COM) and correctness (COR), and is expressed as: in, p ii , p ij and p ji These represent pixels that indicate true positives, false positives, and false negatives, respectively.

10. The pump room smoke and fire detection system based on multi-task deep learning according to claim 7, characterized in that, The encoder network uses a ResNet network as its backbone network.

Images