Fire detection method, device and equipment

Abstract

The present application discloses a fire detection method, device and equipment, which relates to the field of fire image processing. The fire detection method includes: obtaining an image to be detected; inputting the image to be detected into a pre-trained fire detection model to obtain a target frame and a target category label of the detection target in the image to be detected; wherein the detection target includes flames and / or smoke; the pre-trained fire detection model includes a feature extraction network and a Transformer network. A feature suppression module is added after the feature extraction module to suppress features with low significance, which can reduce redundant calculations and memory overhead; a multi-scale deformable attention mechanism is adopted to sparsely sample key positions of features from multi-scale feature maps, realize detection of targets of different scales, and then accurately segment different targets to improve detection accuracy; at the same time, the use of FPN is avoided, which improves the simplicity and efficiency of the model.

Description

Technical Field

[0001] The present application relates to the field of fire image processing, and in particular to a fire detection method, device and equipment. Background Art

[0002] Fire is a disaster with great destructive power and suddenness, which brings great challenges to rescue work. With the rapid development of social economy, the frequency of fire accidents is also increasing, and traditional fire detection methods can no longer meet the needs of modern "intelligent firefighting". In order to achieve rapid and accurate detection of fire scenes, computer vision technology has played an important role. In recent years, image analysis and video image processing technology based on artificial intelligence have been widely used in the field of fire detection, which can detect fires in a timely and effective manner and have important academic and practical value. However, due to the complexity of the fire scene, there are great differences and uncertainties in the shape, size, ambiguity and generation order of the two types of targets, flames and smoke. At present, a variety of technologies based on target detection and semantic segmentation have been proposed at home and abroad to realize the recognition of flames and smoke. However, in some actual fire scenes, flames and smoke often appear at the same time (for example, the combustion of macromolecular organic compounds, such as chemicals, liquid fuels and solid fuels). This leads to a high degree of overlap between flame and smoke images, making it difficult to segment the contours of the target, which will reduce the accuracy of detection. At the same time, due to the significant feature differences between the two different types of targets, traditional target detection methods often have problems such as large accuracy deviation and excessive false detection when detecting flames and smoke respectively. Summary of the invention

[0003] The main purpose of the present application is to provide a fire detection method, device, equipment and computer equipment, aiming to solve the problem of low detection accuracy of the target detection model.

[0004] To achieve the above-mentioned purpose, the present application provides a fire detection method, comprising: acquiring an image to be detected; inputting the image to be detected into a pre-trained fire detection model to obtain a target frame and a target category label of a detection target in the image to be detected; wherein the detection target includes flame and / or smoke; the pre-trained fire detection model includes at least a feature extraction module and a feature suppression module, and the feature suppression module is used to suppress the output of the feature extraction module.

[0005] Optionally, the pre-trained fire detection model also includes an encoder module, a decoder module and a prediction head; the image to be detected is input into the pre-trained fire detection model to obtain a target box and a target category label of the detection target in the image to be detected, including: performing multi-scale feature extraction on the image to be detected through a feature extraction module to obtain a first feature map; performing feature suppression on the first feature map through a feature suppression module to obtain a second feature map; position encoding the second feature map through the encoder module, and using a self-attention mechanism to extract global dependencies on the position-encoded second feature map, and outputting a first feature vector; using a target query through a decoder module to interact with the first feature vector to obtain a prediction vector containing target category and bounding box information; and converting the prediction vector into a bounding box and category label of the target through a prediction head.

[0006] Optionally, the encoder module includes E encoder layers, and the decoder module includes F decoder layers, wherein E and F are both natural numbers, and E<F.

[0007] Optionally, the encoder module includes a first attention unit and a first transformation unit; the first attention unit is used to calculate the relationship between each feature in the second feature map through a self-attention mechanism, and output an initial feature map; the first transformation unit is used to transform the initial feature map and output the feature map; the decoder module includes a second attention unit, a third attention unit and a second transformation unit; the second attention unit is used to calculate the relationship between each initial target query using the self-attention mechanism, and output a target query containing a relationship with other initial target queries; the third attention unit is used to interact the target query with the transformed feature map using a cross-attention mechanism to obtain an initial prediction vector containing target category and bounding box information; the second transformation unit is used to transform the initial prediction vector and output a prediction vector.

[0008] Optionally, the self-attention mechanisms of the first attention unit and the second attention unit are both multi-scale deformable attention mechanisms.

[0009] Optionally, the training process of the pre-trained fire detection model includes: annotating the detection targets in the acquired fire images with target boxes and target categories to obtain a sample set; preprocessing the sample set; inputting the preprocessed sample set into a preset model to obtain predicted target boxes and target category labels; calculating a total loss value based on the predicted target boxes, target category labels and the actual target boxes, target category labels, and using a preset loss function, and updating the parameters of the preset model based on the total loss value until the preset model converges to obtain a pre-trained fire detection model.

[0010] Optionally, after acquiring the image to be detected, the method further includes: preprocessing the image to be detected, wherein the preprocessing process includes transformation, distortion, scaling, and denoising operations.

[0011] Optionally, the preset loss function includes a weighted sum of a classification loss function and a regression loss function, wherein the regression loss function includes a weighted sum of a GIoU loss function and a Smooth loss function.

[0012] In addition, to achieve the above objectives, the present application also provides a fire detection device, including:

[0013] An image acquisition module, used for acquiring an image to be detected;

[0014] The detection module is used to input the image to be detected into a pre-trained fire detection model to obtain the target frame and target category label of the detection target in the image to be detected.

[0015] In addition, to achieve the above-mentioned purpose, the present application also provides a detection device, including at least one processing unit, a system memory and a bus, and the above-mentioned fire detection method is implemented when the processing unit executes a program.

[0016] A fire detection method proposed in the present application adopts FasterNet in the feature extraction module, which can reduce the complexity of the backbone network and flexibly adjust according to the size of the input fire and the requirements of the fire detection task, thereby improving its versatility and detection efficiency; a feature suppression module is added after the feature extraction module to suppress features with low significance (including features in the background), thereby reducing redundant calculations and memory overhead, thereby improving computing efficiency and FLOPS; a multi-scale deformable attention mechanism is adopted in the Transformer network, which can sparsely sample key positions of features from multi-scale feature maps, realize the detection of flames (smoke) of different scales, and then accurately segment flames and smoke, thereby improving detection accuracy; at the same time, the use of FPN is avoided, thereby improving the simplicity and efficiency of the fire detection model. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A flow chart of a fire detection method for this application;

[0018] Figure 2 A schematic diagram of the structure of a pre-trained fire detection model in a fire detection method of the present application;

[0019] Figure 3 A detection process diagram of a fire detection model in a fire detection method of the present application;

[0020] Figure 4 A partial structural schematic diagram of a fire detection model in a fire detection method of the present application;

[0021] Figure 5 A flow chart of a model training process in a fire detection method of the present application;

[0022] Figure 6 is the original image of the first type of image to be detected;

[0023] Figure 7 An image obtained by processing a first type of image to be detected using a fire detection method of the present application;

[0024] Figure 8 is the original image of the second type of image to be detected;

[0025] Fig. 9 An image obtained by processing a second type of image to be detected using a fire detection method of the present application;

[0026] Fig.10 is the original image of the third type of image to be detected;

[0027] Fig.11 An image obtained by processing the third type of image to be detected using a fire detection method of the present application;

[0028] Fig.12 is the original image of the fourth type of image to be detected;

[0029] Fig.13 An image obtained by processing the fourth type of image to be detected using a fire detection method of the present application;

[0030] Fig.14 This is a structural schematic diagram of a fire detection device for this application;

[0031] Fig.15 A computer readable storage medium for this application is a CD.

[0032] Fig.16 The present application relates to a detection device;

[0033] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0034] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0035] Since the flames and smoke in the image overlap to a high degree, it is difficult for existing technologies to accurately segment the contours of the target, resulting in low detection accuracy and excessive false detections.

[0036] In order to solve the above technical problems, the present application provides a fire detection method. Figure 1 is a flow chart of a fire detection method according to an embodiment of the present application, such as Figure 1 As shown, the detection method includes:

[0037] Step S10, obtaining an image to be detected;

[0038] In this embodiment, the image to be detected can be obtained by reading a picture from a camera sub-stream or a video.

[0039] In an exemplary embodiment, after step S10, the method may further include the following steps:

[0040] The fire image is preprocessed, wherein the preprocessing process may include transformation, distortion, scaling, and denoising operations; specifically, the fire image I is subjected to translation transformation, rotation transformation, partial lens distortion, scaling, and denoising to obtain an output image O:

[0041]

[0042] Among them, I Bili represents the image after double nonlinear interpolation scaling, i and j represent the pixel coordinates, median represents the median filtering algorithm, replaces the central pixel with the median of the pixel values ​​in the 3×3 window, and then slides the window until all pixels are processed. After the above processing, the noise can be reduced and the detection accuracy can be improved.

[0043] Step S20, inputting the image to be detected into a pre-trained fire detection model to obtain a target frame and a target category label of the detection target in the image to be detected; wherein the detection target includes flame and / or smoke;

[0044] like Figure 2 As shown, the pre-trained fire detection model includes a feature extraction network 1, a feature suppression module 2, an encoder module 3, a decoder module 4 and a prediction head 5; wherein the feature extraction module 1 is used to extract features from the fire image to obtain a first feature map; the feature suppression module 2 is used to suppress the features with low significance output by the feature extraction module to obtain a second feature map. The feature suppression module 2 may include a channel attention module 21 and a spatial attention module 22. The encoder module 3 is used to position encode the second feature map, and use the self-attention mechanism to extract global dependencies from the position-encoded second feature map, and output a first feature vector. The decoder module 4 is used to interact with the feature vector using a target query to obtain a prediction vector containing target category and bounding box information. The prediction head 5 is used to convert the prediction vector into a bounding box and category label of the target.

[0045] In detail, Figure 3 As shown, the detection process of the fire detection model in step S20 may specifically include the following steps:

[0046] Step S21, performing multi-scale feature extraction on the image to be detected by the feature extraction module 1 to obtain a first feature map; for example, the features in the first feature map may include smoke and flame;

[0047] Step S22, suppressing the features with lower significance in the first feature map by the feature suppression module 2, highlighting the key features, and obtaining a second feature map; the feature suppression module 2 can reduce redundant calculations and memory overhead, and prune the weights with lower significance, thereby improving the computing efficiency and FLOPS;

[0048] Step S23, position encoding the second feature map through the encoder module 3, and using the self-attention mechanism to extract the global dependency of the position-encoded second feature map, and output a first feature vector; wherein the position-encoded second feature map contains flame (smoke) features and the positions of corresponding features in the fire image, and the first feature vector contains the relationship between each flame (smoke) feature in the encoded second feature map.

[0049] Step S24, using the target query to interact with the first feature vector through the decoder module 4 to obtain a prediction vector containing target category and bounding box information;

[0050] Step S25, convert the prediction vector into the bounding box and category label of the target through the prediction head 5.

[0051] In this embodiment, a feature suppression module 2 is added after the feature extraction module 1, and the feature suppression module 2 may include a normalization-based attention mechanism (NAM). NAM uses the scaling factor in batch normalization to measure the importance of channels and pixels, thereby suppressing significant features and reducing redundant calculations and memory overhead. In addition, NAM applies a sparse weight penalty to the attention module so that weights with lower significance can be pruned, thereby improving computational efficiency and FLOPS. Furthermore, the feature suppression module 2 may include a channel attention module and a spatial attention module. The channel attention module outputs M c And the spatial attention module output M s They are:

[0052]

[0053]

[0054]

[0055] M c =Sigmoid(W γ (BN(F1)))

[0056] M s =Sigmoid(W λ (BN S (F2)))

[0057] In the formula, μ B represents the mean, σ B represents the standard deviation, γ and β both represent trainable affine transformation parameters, i.e., scale and displacement, γ i Represents the proportional coefficient of each channel, w i represents the weight and λ represents the scaling factor.

[0058] Further, the feature extraction module 1 may include FasterNet. FasterNet is used as the backbone network. First, FasterNet reduces the computational complexity of the model and improves the computational efficiency by using partial convolution (PConv), especially for fire detection scenarios that require real-time analysis. Secondly, the design of PConv also reduces the memory usage of the model at runtime, allowing FasterNet to run on resource-constrained devices, such as mobile devices or embedded systems. In addition, FasterNet has a simple structure, is easy to implement and deploy, and is also convenient for model analysis and interpretation. FasterNet can also be flexibly adjusted according to the size of the input fire and the requirements of the fire detection task, thereby improving its versatility and adaptability. Specifically, the input image is first sent to the PConv layer, and the spatial features of the image are effectively extracted through partial convolution operations, while reducing the computational and memory requirements. The extracted spatial features are then passed to the subsequent layers of the FasterNet network for further analysis and processing; in these layers, the FasterNet network adjusts the size and step size of the convolution kernel according to the specific requirements of fire detection to adapt to input images of different sizes; and finally outputs the processed feature maps. These feature maps contain information that is critical to fire detection.

[0059] In an exemplary embodiment, if Figure 4As shown, the encoder module 3 includes a first attention unit 31 and a first transformation unit 32. The first attention unit 31 can be used to calculate the relationship between each feature in the second feature map through the self-attention mechanism and output the initial feature map; the first transformation unit 32 is used to transform the initial feature map and output the feature map. The decoder module 4 includes a second attention unit 41, a third attention unit 42 and a second transformation unit 43. The decoder module 4 is used to interact the query vector with the transformed feature map through the self-attention mechanism to obtain a prediction vector. The self-attention mechanism includes a multi-scale deformable attention mechanism; the second attention unit 41 is used to calculate the relationship between each initial target query using the self-attention mechanism and output the target query; the third attention unit 42 is used to interact the target query with the transformed feature map using the cross-attention mechanism, update the target query, and obtain an initial prediction vector containing the target category and bounding box information; the second transformation unit 43 is used to transform the initial prediction vector and output the prediction vector.

[0060] In this embodiment, the input (initial target query) of the second attention unit 41 in the first layer decoder module 4 is randomly generated through training and learning, and the target query is a set of learned embedding vectors; for example, 100 target queries can be preset, with a shape of (100, 256) and initialized to 0. In the second attention unit 41, each initial target query will pay attention to the relevant parts of other initial target queries in the second attention unit 41 through a deformable self-attention mechanism. For example, if a target query contains flame information, then the target query will pay attention to other initial target queries containing flame-related information, thereby capturing the relationship between each initial target query and outputting a target query, which contains the relationship with other initial target queries. In the third attention unit 42, the target query interacts with the output feature map of the encoder, that is, each target query extracts information related to flame or smoke from the picture according to its own area of ​​interest, and updates the target query. This allows each target query to dynamically pay attention to a small part of the encoder output containing flame or smoke according to its own state, thereby effectively aggregating information to predict the category and bounding box of the target.

[0061] In some exemplary embodiments, step S23 specifically includes the following steps:

[0062] Step S231, compressing the dimension of the feature map obtained by the feature extraction module along the channel direction and straightening the width and height dimensions, so that the two-dimensional feature map of each channel is converted into a one-dimensional vector to obtain a coding feature;

[0063] Step S232, generate position coding, add the position coding vector to the coding feature, capture the visual features of the flame and smoke, and obtain the position embedding vector; wherein, the method for generating the position coding is: encode according to the dimension index of each position in the feature map to obtain the position coding vector; specifically, for the positions with odd dimension indexes in the feature map, use the cosine function to create a vector, and for the positions with even dimension indexes in the feature map, use the sine function to create a vector.

[0064] Step S233, linearly transform the position embedding vector by corresponding linear matrices to obtain the first query vector z q , the first key vector, the first value vector, such as through W Q , W K , W V Perform a linear transformation on the position embedding vector; since the first attention unit is a multi-head attention mechanism, the first query vector, the first key vector and the first value vector are divided into m vectors respectively; the m first query vectors z q , the first key vector, and the first value vector are input into the first attention unit 31, and the sub-output vector is calculated using the self-attention mechanism; the m sub-output vectors are concatenated into the first output vector (initial feature map).

[0065] Step S234, performing a residual connection on the first output vector and the position embedding vector, and normalizing them to obtain a second output vector.

[0066] Step S235, input the second output vector into the first transformation unit 32 for nonlinear transformation to obtain a first nonlinear feature; connect the first nonlinear feature with the second output vector residual and normalize them to obtain a feature vector (feature map) containing the context relationship.

[0067] In some exemplary embodiments, step S24 specifically includes the following steps:

[0068] Step S241, linearly transforming the target query through a linear matrix to obtain a second query vector, a second key vector, and a second value vector, inputting the second query vector, the second key vector, and the second value vector into a second attention unit 41, and calculating an output matrix using a self-attention mechanism, the output matrix including updated target query information;

[0069] Step S242, calculate the third key vector and the third value vector according to the feature vector output by the encoder, use the output matrix as the third query vector, input the third key vector, the third value vector and the third query vector into the third attention unit 42, and use the cross attention mechanism to calculate the initial prediction vector containing the target category and the bounding box information. The third query vector in the first layer decoder is the target query.

[0070] Step S243, performing a residual connection on the updated target query and the output matrix input as the input of the second transformation unit 43, performing a nonlinear transformation on it, and obtaining a second nonlinear feature; performing a residual connection on the second nonlinear feature and the updated target query and outputting it to obtain a prediction vector, wherein the prediction vector includes the category information and the bounding box position information of the flame or smoke.

[0071] In an exemplary embodiment, the first transformation unit 32 and the second transformation unit 43 may be FFN, which includes further processing a feature map representing image semantics through two nonlinear transformations and an activation function to increase the nonlinear expression capability of the fire detection model.

[0072] In some exemplary embodiments, the first attention unit 31 and the second attention unit 41 both adopt a multi-scale deformable attention mechanism. In this embodiment, the input feature is converted into a new feature representation through a multi-scale deformable attention mechanism, and the new feature representation can highlight the most important information in the input data for the current task. Furthermore, a richer image representation can be obtained by taking the multi-scale features obtained by the feature extraction module 1 as input. Since the same normalized coordinates may correspond to different actual positions in the feature map at different scales, the function φ l Will Converting back to the actual size of the feature map of the lth layer can maintain the spatial consistency of the multi-scale feature maps and ensure that the attention mechanism can correctly correspond to the feature maps of each scale. Through this scaling, it can be ensured that feature points can be accurately located at all scales.

[0073] Furthermore, the calculation formula of the multi-scale deformable attention mechanism is:

[0074]

[0075] In the formula, x l Represents the input features of the lth layer, regarded as a value vector, z q represents the query vector, represents the position of the query vector after transformation or offset, M represents the number of attention heads, L represents the layer of the feature map, and W m represents the weight matrix of the mth attention head, W′ m represents the weight matrix of the value vector of the mth attention head, K represents the number of key vectors sampled by each attention head, and A mlqk represents the attention weight of the mth attention head, φ l represents the transformation function, Δp mlqk Represents the learned offset, which is used to adjust p q .

[0076] In this embodiment, the normalized coordinates are rescaled to the input feature map of the lth layer through a multi-scale deformable attention mechanism, and LK points are sampled from the multi-scale feature map instead of K points from the single-scale feature map. mlqk When the size of the query and key is reduced, all layers sample K points, and the features of different layers are integrated, which is similar to FPN, so the use of FPN is avoided, and the simplicity and efficiency of the model are improved. Therefore, the network model of this application not only accelerates the convergence of the model, but also improves the detection ability of multi-scale targets.

[0077] In an exemplary embodiment, the encoder module may include E encoder layers, and the decoder module may include F decoder layers, wherein E and F are both natural numbers, and E<F.

[0078] For example, E can be 3, and F can be 6, that is, the encoder module can include 3 encoder layers, and the decoder module can include 6 decoder layers. The encoder module 3 is mainly responsible for feature extraction, and the feature extraction part of the algorithm model has completed sufficient work. Therefore, this embodiment reduces the calculation amount and memory usage of the model by reducing the number of layers of the encoder module 3, which can better balance the accuracy and processing speed of the fire detection model. And it is verified by ablation experiments, and the results are as follows:

[0079] Ablation results of the number of encoder-decoder layers

[0080]

[0081] According to the experimental results in the table, the codec structure including 6 encoder layers and 6 decoder layers makes the detected AP 50 The value increased by 6.2%, which shows that the number of decoder layers has a greater impact on the performance of the fire detection model. However, the codec structure including 3 encoder layers and 6 decoder layers has a higher detection AP than that including 6 encoder layers and 6 decoder layers. 50 It is only reduced by 0.6%, but the number of parameters of the fire detection model is greatly reduced. Therefore, the encoder module of this embodiment includes 3 encoder layers and the decoder module includes 6 decoder layers, which can balance the complexity and accuracy of the model with more efficient calculation and memory usage.

[0082] In this embodiment, a feature suppression module 2 is added after the feature extraction module 1 to suppress features with low significance, reduce redundant calculations and memory overhead, and thus improve computational efficiency and FLOPS; a multi-scale deformable attention mechanism is used in the Transformer network, which can sparse features and sample key positions from multi-scale feature maps, realize the detection of targets of different scales, and then accurately segment different targets to improve detection accuracy; at the same time, the use of FPN is avoided, improving the simplicity and efficiency of the model. At the same time, changing the number of layers of the encoder module to 3 layers can balance the complexity and accuracy of the model with more efficient calculation and memory usage.

[0083] In some exemplary embodiments, in an exemplary embodiment, the prediction head 5 may include a first fully connected layer 51 and a second fully connected layer 52 .

[0084] Furthermore, the specific process of step S25 is as follows:

[0085] The first fully connected layer 51 can be used to obtain the prediction vector output by the decoder, which includes category information, calculates the score of each category through the ReLU activation function, converts the above score into probability through Softmax, and obtains the predicted category label in combination with the classification loss function;

[0086] The second fully connected layer 52 can be used to obtain a prediction vector output by the decoder, which includes the position information of the bounding box, calculate the center point, width and height of the bounding box based on the prediction information, and obtain the bounding box in combination with the regression loss function.

[0087] Based on the above embodiments, Figure 5 FIG. 1 is a flow chart of a fire detection method according to another embodiment of the present application. Figure 5 As shown, the fire detection method may include the following steps:

[0088] S210, annotating the detection targets in the fire image with target frames and target categories to obtain a sample set;

[0089] In this embodiment, the fire images include a real flame image group, a real smoke image group, and a flame and smoke image group. Each of the above image groups is a 24-bit RGB image with a resolution between 256×256 and 1920×1080, and is arranged according to a unified naming scheme. The training set and the test set are divided into 8:2 and come from different video clips. All data are annotated in MS COCO format. Specifically, the real flame image group has a total of 13,750 images, the real smoke image group has a total of 9,465 images, and the flame and smoke image group has a total of 2,895 images, including gray and black smoke, and the scenes include cities, forests, wastelands, industrial scenes, etc.

[0090] Specifically, a labeling tool (such as LabelImg or LabelMe) is used to label the target box and category label of the flame on each image in the real flame image group, to label the target box and category label of the smoke on each image in the real smoke image group, and to label the target box and category label corresponding to the flame and smoke on each image in the flame and smoke image group, and generate corresponding XML or JSON files.

[0091] S220, preprocessing the sample set;

[0092] S230, inputting the preprocessed sample set into a preset model to obtain a predicted target frame and a target category label;

[0093] S240, according to the predicted target frame, target category label and the actual target frame, target category label, and using a preset loss function to calculate the total loss value, and update the parameters of the preset model according to the total loss value until the preset model converges to obtain a pre-trained fire detection model.

[0094] During training, the deformable attention parameters can be set to M=8 and K=4, the size of each mini-batch can be set to 2, and a total of 50 epochs are trained. Using the Adam optimizer, the basic learning rate can be 2×10 -4 , the learning rate attenuation coefficient can be 0.1 at the 40th iteration, and the weight attenuation can be 10 -4 .

[0095] In some exemplary embodiments, the preset loss function includes a weighted sum of a regression loss function and a classification loss function, and the calculation formula is as follows:

[0096]

[0097]

[0098] Among them, the regression loss function is the weighted sum of the Smooth L1 loss function and the GIoU loss function:

[0099] L box =αL GIOU +βL SmoothL1

[0100] Where α and β represent the GIoU loss function L GIOU , Smooth L1 loss function L Smooth L1 The weight coefficients are defined in the embodiment as α=2 and β=5, y represents the Ground Truth, For the prediction box σ i The predicted value of The prediction index The classification probability, c i Represents the classification label, represents the prediction box, b σ(i) represents the desired box of the ith index, is the predicted box of the i-th index.

[0101]

[0102] Where b σ(i) represents the desired box of the ith index, is the predicted box of the i-th index.

[0103] GIoU can reflect the correlation between the predicted box and the expected box, and the expression is:

[0104]

[0105]

[0106] In the formula, Indicates that it contains b σ(i) and The minimum bounding box of Indicates that b is not covered σ(i) and area.

[0107] S240, obtaining an image to be detected, inputting the image to be detected into a fire detection model, and obtaining a target frame and a target category label of a detection target in the image to be detected.

[0108] In this embodiment, the image to be detected is as follows: Figure 6-Figure 9 As shown, Figure 6 For images including smoke, Figure 8 For images that include flames, Fig.10 and Fig.12 The above images are input into the fire detection model to obtain images including target boxes and target category labels, such as Figure 7 , Fig. 9 , Fig.11 , Fig.13 . 50 (Average Precision at 50% IoU) is used as an indicator to evaluate the accuracy. In this embodiment, the AP of flame and smoke detection is 50 The values ​​reached 81.7% and 76.0% respectively, and the detection speed was 0.036frame / s. The detection method of this embodiment is compared with the detection method of the traditional Transformer-based Deformable DETR model. The detection method provided by this embodiment has a better detection performance in the AP of flame and smoke detection. 50The values ​​are increased by 3.9% and 6.2% respectively, and the detection speed per frame is also increased by 0.023 seconds.

[0109] Based on the above embodiments, Fig.14 FIG. 1 is a structural diagram of a fire detection device, such as Fig.14 As shown, an embodiment of the present application also provides a fire detection device including: an image acquisition module, used to acquire an image to be detected; a detection module, used to input the image to be detected into a pre-trained fire detection model to obtain a target frame and a target category label of a detection target in the image to be detected.

[0110] In some exemplary embodiments, the detection module is also used to perform multi-scale feature extraction on the image to be detected through the feature extraction module 1 to obtain a first feature map; suppress the output of the feature extraction module 1 through the feature suppression module 2; position encode the second feature map through the encoder module 3, and use the self-attention mechanism to extract global dependencies on the position-encoded second feature map to output a first feature vector; interact with the first feature vector through the decoder module 4 using the target query to obtain a prediction vector containing the target category and bounding box information; and convert the prediction vector into the bounding box and category label of the target through the prediction head 5.

[0111] In some exemplary embodiments, the training process of the pre-trained fire detection model includes: annotating the acquired fire image with a target box and a target category to obtain a sample set; preprocessing the sample set; inputting the preprocessed sample set into a preset model to obtain a predicted target box and a target category label; calculating a total loss value based on the predicted target box and the target category label and the actual target box and the target category label using a preset loss function, and updating the parameters of the preset model based on the total loss value until the preset model converges to obtain a pre-trained fire detection model.

[0112] It should be understood that the fire detection device of the present application has the same beneficial effects as those described in any of the above fire detection method embodiments, which will not be described in detail here.

[0113] Based on the above embodiments, the present application also provides a computer-readable storage medium, referring to Fig.15, the computer-readable storage medium shown is a CD-ROM 6, on which a computer algorithm (i.e., an algorithm product) is stored. When the computer algorithm is executed by the processor, each step recorded in the above method implementation will be implemented, for example, obtaining an image to be detected; inputting the image to be detected into a pre-trained fire detection model to obtain a target frame and a target category label of the detection target in the image to be detected; wherein the detection target includes flame and / or smoke; the pre-trained fire detection model includes at least a feature extraction module and a feature suppression module, and the feature suppression module is used to suppress the output of the feature extraction module. The specific implementation method of each step will not be repeated here.

[0114] It should be noted that examples of computer-readable storage media may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical or magnetic storage media, which are not listed here one by one.

[0115] In addition, based on the above embodiments, the present application also provides a detection device. Fig.16 A block diagram of an exemplary detection device 7 suitable for implementing the embodiments of the present application is shown, and the detection device 7 may be a computer system or a server. Fig.16 The detection device 7 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0116] refer to Fig.16 The components of the detection device 7 may include, but are not limited to: one or more processors or processing units 71, a system memory 72, and a bus 73 connecting different system components (including the system memory 72 and the processing unit 71).

[0117] The detection device 7 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the detection device 7, including volatile and non-volatile media, removable and non-removable media.

[0118] The system memory 72 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 721 and / or cache memory 722. The detection device 7 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the ROM 723 may be used to read and write non-removable, non-volatile magnetic media ( Fig.16 is not shown in the Fig.16As shown in the figure, a disk drive for reading and writing a removable non-volatile disk (e.g., a "floppy disk") and an optical disk drive for reading and writing a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical media) can be provided. In these cases, each drive can be connected to a bus 73 connecting different system components through one or more data medium interfaces. The system memory 72 may include at least one algorithm product, which has a set (e.g., at least one) algorithm module, which is configured to perform the functions of each embodiment of the present application.

[0119] An algorithm / utility 725 having a set (at least one) of algorithm modules 724 may be stored, for example, in system memory 72, and such algorithm modules 724 include, but are not limited to, an operating system, one or more application algorithms, other algorithm modules, and algorithm data, each of which or some combination may include an implementation of a network environment. Algorithm modules 724 generally perform the functions and / or methods of the embodiments described herein.

[0120] The detection device 7 may also communicate with one or more external devices 74 (e.g., a keyboard, a pointing device, a display, etc.). Such communication may be performed via an input / output (I / O) interface 75. Furthermore, the detection device 7 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 76. Fig.16 As shown, the network adapter 76 communicates with other modules (such as the processing unit 71, etc.) of the detection device 7 via the bus 73 connecting different system components. Fig.16 Not shown, other hardware and / or software program modules may be used in conjunction with the detection device 7 .

[0121] The processing unit 71 executes various functional applications and data processing by running the algorithm stored in the system memory 72, for example, obtaining an image to be detected; inputting the image to be detected into a pre-trained fire detection model to obtain a target frame and a target category label of the detection target in the image to be detected; wherein the detection target includes flame and / or smoke; the pre-trained fire detection model includes at least a feature extraction module and a feature suppression module, and the feature suppression module is used to suppress the output of the feature extraction module. The specific implementation method of each step is not repeated here. It should be noted that although several units / modules or sub-units / sub-modules of the robot's motion trajectory processing device are mentioned in the above detailed description, this division is only exemplary and not mandatory. In fact, according to the embodiment of the present application, the features and functions of two or more units / modules described above can be concretized in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided into multiple units / modules to be concretized.

[0122] In the description of the present application, it should be noted that the terms "first", "second" and "third" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0123] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0124] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0125] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0126] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0127] If the function is implemented in the form of a software program functional unit and sold or used as an independent product, it can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software program product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., various media that can store algorithm codes.

[0128] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A fire detection method, characterized in that: include: Acquire the image to be detected; Inputting the image to be detected into a pre-trained fire detection model to obtain a target frame and a target category label of a detection target in the image to be detected, wherein the pre-trained fire detection model includes an encoder module, a decoder module and a prediction head; Wherein, the detection target includes flame and / or smoke; The pre-trained fire detection model comprises at least a feature extraction module and a feature suppression module, wherein the feature suppression module is used to suppress the output of the feature extraction module; The method of inputting the image to be detected into a pre-trained fire detection model to obtain a target frame and a target category label of a detection target in the image to be detected includes: performing multi-scale feature extraction on the image to be detected by the feature extraction module to obtain a first feature map; performing feature suppression on the first feature map by the feature suppression module to obtain a second feature map; performing position encoding on the second feature map by the encoder module, and performing global dependency extraction on the position-encoded second feature map by using a self-attention mechanism to output a first feature vector; using the decoder module to interact with the first feature vector by using a target query to obtain a prediction vector containing target category and bounding box information; and converting the prediction vector into a bounding box and category label of the target by using the prediction head.

2. The fire detection method according to claim 1, characterized in that: The encoder module includes E encoder layers, and the decoder module includes F decoder layers, wherein E and F are both natural numbers, and E<F.

3. The fire detection method according to claim 1, characterized in that: The encoder module includes a first attention unit and a first transformation unit; The first attention unit is used to calculate the relationship between the features in the second feature map through a self-attention mechanism, and output an initial feature map; The first transformation unit is used to transform the initial feature map and output a feature map; The decoder module includes a second attention unit, a third attention unit, and a second transformation unit; The second attention unit is used to calculate the relationship between each initial target query using the self-attention mechanism, and output the target query containing the relationship with other initial target queries; the third attention unit is used to interact the target query with the transformed feature map using the cross-attention mechanism to obtain an initial prediction vector containing target category and bounding box information; the second transformation unit is used to transform the initial prediction vector and output a prediction vector.

4. The fire detection method according to claim 3, characterized in that: The self-attention mechanisms of the first attention unit and the second attention unit are both multi-scale deformable attention mechanisms.

5. The fire detection method according to claim 1, characterized in that: The training process of the pre-trained fire detection model includes: The detected targets in the acquired fire images are labeled with bounding boxes and target categories to obtain a sample set; Preprocessing the sample set; Input the preprocessed sample set into a preset model to obtain a predicted target frame and a target category label; According to the predicted target frame, target category label and the actual target frame, target category label, a total loss value is calculated using a preset loss function, and the parameters of the preset model are updated according to the total loss value until the preset model converges to obtain a pre-trained fire detection model.

6. The fire detection method according to claim 1, characterized in that: After acquiring the image to be detected, the method further includes: The image to be detected is preprocessed, wherein the preprocessing process includes transformation, distortion, scaling, and denoising operations.

7. The fire detection method according to claim 5, characterized in that: The preset loss function includes a weighted sum of a classification loss function and a regression loss function, wherein the regression loss function includes a weighted sum of a GIoU loss function and a Smooth loss function.

8. A fire detection device, characterized in that: include: An image acquisition module, used for acquiring an image to be detected; A detection module, used for inputting the image to be detected into a pre-trained fire detection model to obtain a target frame and a target category label of a detection target in the image to be detected, wherein the pre-trained fire detection model includes an encoder module, a decoder module and a prediction head; Wherein, the detection target includes flame and / or smoke; The fire detection model at least includes a feature extraction module and a feature suppression module, wherein the feature suppression module is used to suppress the output of the feature extraction module; The method of inputting the image to be detected into a pre-trained fire detection model to obtain a target frame and a target category label of a detection target in the image to be detected includes: performing multi-scale feature extraction on the image to be detected by the feature extraction module to obtain a first feature map; performing feature suppression on the first feature map by the feature suppression module to obtain a second feature map; performing position encoding on the second feature map by the encoder module, and performing global dependency extraction on the position-encoded second feature map by using a self-attention mechanism to output a first feature vector; using the decoder module to interact with the first feature vector by using a target query to obtain a prediction vector containing target category and bounding box information; and converting the prediction vector into a bounding box and category label of the target by using the prediction head.

9. A detection device, characterized in that: The device comprises at least one processing unit and a readable storage medium, wherein the processing unit executes a program stored in the readable storage medium to implement the fire detection method according to any one of claims 1 to 7.

Images