Fire target detection method based on local form knowledge matching and related device

Abstract

The invention discloses a fire target detection method based on local form knowledge matching and a related device, and relates to the field of fire target detection, and the method comprises the steps: based on a fire local form feature knowledge base, employing a knowledge template matching technology to extract local fire form features of a preprocessed fire image; adopting a detection-oriented Transform backbone network to extract a plurality of fire features of different scales of the preprocessed fire image; and carrying out aggregation, alignment, fire target classification and bounding box regression on the local fire morphological features and the plurality of fire features of different scales to obtain fire target categories and mark box coordinates. According to the method, the prior knowledge base is constructed based on the unique local form of the flame target, the local form characteristics of the fire target in the fire image are mined by using the knowledge matching method, and the extracted multi-scale fire semantic characteristics with high-level information are combined, so that the accurate detection of the small target in the early stage of the fire is realized.

Description

Technical Field

[0001] The present application relates to the technical field of fire target detection, and in particular to a fire target detection method and related devices based on local morphological knowledge matching. Background Art

[0002] At present, the existing fire detection methods based on target detection can be divided into fire target detection methods based on region extraction, regression and Transformer (a deep learning model based on self-attention mechanism). Among them, the fire target detection method based on Transformer uses a powerful multi-head attention mechanism to simultaneously capture the global and local features of the fire image, and deeply analyzes the shape, texture and context information of the fire target, so as to achieve more accurate detection in complex scenes. However, the high number of parameters of the Transformer model also leads to an increase in computational complexity and memory consumption, which poses a challenge to the real-time performance of detection. DFFT (Decoder-Free Fully Transformer-BasedObject Detector) is a full Transformer target detection model without a decoder. It aims to achieve efficient and accurate target detection by simplifying the model structure and optimizing the feature extraction and fusion mechanism. This model is particularly suitable for scenarios with high real-time requirements and strict accuracy requirements such as fire detection. However, this model ignores the hidden and tiny characteristics of smoke or flame targets in the early stage of fire, and the detection effect of small-sized smoke or flame targets is still not ideal. Summary of the invention

[0003] The purpose of this application is to provide a fire target detection method and related devices based on local morphological knowledge matching, which can improve the detection effect of small-sized flames or smoke targets in the early stage of a fire.

[0004] To achieve the above objectives, this application provides the following solutions:

[0005] In a first aspect, the present application provides a fire target detection method based on local morphological knowledge matching, comprising:

[0006] The target fire image is preprocessed to obtain a preprocessed fire image; the target fire image is an image obtained by photographing the target fire area.

[0007] Based on a knowledge base of local fire morphology features, a knowledge template matching technique is used to extract local fire morphology features of the preprocessed fire image; the knowledge base of local fire morphology features includes a plurality of knowledge template images; the knowledge template images are images including different local flame morphologies selected from a fire image data set.

[0008] A deep fire detector is used to extract fire features of several different scales from the preprocessed fire image; the deep fire detector is a detection-oriented Transformer backbone network DOT.

[0009] A feature aggregation encoder is used to aggregate the local fire morphological features and a plurality of fire features of different scales to obtain a single-level fire semantic feature.

[0010] Based on the single-level fire semantic features, a task alignment encoder is adopted to generate aligned classification features and regression features.

[0011] According to the aligned classification features and regression features, fire target classification and bounding box regression are performed to obtain the fire target category and marked box coordinates.

[0012] In a second aspect, the present application provides a fire target detection device based on local morphological knowledge matching, comprising:

[0013] The preprocessing module is used to preprocess the target fire image to obtain a preprocessed fire image; the target fire image is an image obtained by photographing the target fire area.

[0014] The local fire morphology feature extraction module is used to extract the local fire morphology features of the preprocessed fire image based on the fire local morphology feature knowledge base and the knowledge template matching technology; the fire local morphology feature knowledge base includes a plurality of knowledge template images; the knowledge template images are images including different flame local morphologies screened from the fire image data set.

[0015] The multi-scale fire feature extraction module is used to extract fire features of several different scales from the pre-processed fire image using a deep fire detector; the deep fire detector is a detection-oriented Transformer backbone network DOT.

[0016] The aggregation module is used to aggregate the local fire morphological features and a plurality of fire features of different scales using a feature aggregation encoder to obtain a single-level fire semantic feature.

[0017] The task alignment module is used to generate aligned classification features and regression features based on the single-level fire semantic features by using a task alignment encoder.

[0018] The classification and regression module is used to perform fire target classification and bounding box regression according to the aligned classification features and regression features to obtain the fire target category and marked box coordinates.

[0019] In a third aspect, the present application provides a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-mentioned fire target detection based on local morphological knowledge matching.

[0020] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned fire target detection method based on local morphological knowledge matching.

[0021] In a fifth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the above-mentioned fire target detection method based on local morphological knowledge matching.

[0022] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0023] The present application provides a fire target detection method and related device based on local morphology knowledge matching. Based on the fire local morphology feature knowledge base, the knowledge template matching technology is used to extract the local fire morphology features of the preprocessed fire image; the fire local morphology feature knowledge base includes several knowledge template images, and the several knowledge template images are images including different local flame morphologies screened from the fire image data set; the detection-oriented Transformer backbone network DOT is used to extract several fire features of different scales of the preprocessed fire image; the local fire morphology features and several fire features of different scales are aggregated, aligned, fire target classified and bounding box regressed to obtain the fire target category and marking box coordinates. The present application constructs a priori knowledge base based on the unique local morphology of the flame target, uses the knowledge matching method to match the fire image with the knowledge template image in the local morphology knowledge base, mines the local morphology features of the fire target in the fire image, and combines the multi-scale fire semantic features with high-level information extracted based on the detection-oriented Transformer backbone network DOT to achieve accurate detection of small targets in the early stage of the fire. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0025] Figure 1 This is an application environment diagram of a fire target detection method based on local morphological knowledge matching in one embodiment of the present application;

[0026] Figure 2 A flowchart of a fire target detection method based on local morphological knowledge matching provided in one embodiment of the present application;

[0027] Figure 3 A flowchart of another fire target detection method based on local morphological knowledge matching provided by an embodiment of the present application;

[0028] Figure 4 A schematic diagram of mAP for small, medium and large size targets on Kaggle_flame_smoke provided in an embodiment of the present application;

[0029] Figure 5 A schematic diagram of the detection results of the LMKMFD on a small-sized target provided in an embodiment of the present application; Figure 5 (a) is a schematic diagram of the detection results of small-scale smoke. Figure 5 (b) is a schematic diagram of the detection results of small-scale flames. Figure 5 (c) is a schematic diagram of the detection results of another small-scale flame. Figure 5 (d) is a schematic diagram of the detection results of small-scale smoke and flame;

[0030] Figure 6 A schematic diagram for comparing multi-scale fire detection results provided by an embodiment of the present application;

[0031] Figure 7 A schematic diagram for comparing another multi-scale fire detection result provided by an embodiment of the present application;

[0032] Figure 8 A schematic diagram of functional modules of a fire target detection device based on local morphological knowledge matching provided in one embodiment of the present application;

[0033] Fig. 9 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0035] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0036] The following are the definitions of abbreviations and key terms used in this application:

[0037] 1) LMKMFD: Local Morphological Knowledge Matching based Fire Object Detector, which is the abbreviation of the method proposed in this application.

[0038] 2) Transformer: A deep learning model based on the self-attention mechanism, also referred to as the backbone network for multi-scale fire feature extraction in this application.

[0039] 3) DFFT: Decoder-Free Fully Transformer-Based Object Detector, a target detector that does not require a decoder, is a prior art close to this application. The fire target detection process of DFFT technology can be summarized into the following three stages: In the first stage, a Transformer backbone network for target detection is constructed to extract multi-scale features of fire images, introduce position information through an embedding module, and use a self-attention mechanism to capture long-distance dependencies in fire images; In the second stage, the multi-scale feature map output by the first stage is integrated into an overall feature map through a multi-scale feature fusion encoder; In the third stage, for the overall feature map, the self-attention mechanism of the grouped channel dimension is used to achieve feature decoupling and alignment of classification and regression tasks.

[0040] 4) DOT: Detection-Oriented Transformer backbone, detection-oriented Transformer backbone network, also referred to as DOT backbone network in this application.

[0041] 5)FAE: Feature-Aggregated Encoder, feature aggregation encoder.

[0042] 6)TAE: Task-Aligned Encoder, task-aligned encoder.

[0043] 7)SAA: Semantic-AugmentedAttention, semantically enhanced attention.

[0044] In the early stage of a fire, the random diffusion of smoke and the complex texture of the flames, coupled with the tiny nature of the fire targets, make it extremely difficult to accurately identify and locate the small target area of ​​the fire. Compared with other rigid objects, flames have unique local morphology, such as sharp tops, holes, and curved edges of flames. In view of the problem that the smoke and flame targets are small in the early stage of a fire, which makes it difficult to accurately locate small fire targets, this application proposes a fire target detection method based on local morphological knowledge matching, breaking through the concealment of small fire targets, and realizing the accurate positioning of small targets and other multi-scale targets in the early stage of a fire. First, based on the unique local morphology of flame targets, such as sharp tops, irregular edges, flows, and holes, a local morphology prior knowledge base (i.e., local morphology feature knowledge base) is constructed; secondly, the knowledge matching method is used to match the fire image with the knowledge template in the local morphology knowledge base to mine the local fire morphology features of the fire target in the image; then, a Transformer-based backbone network is used to efficiently extract multi-scale fire semantic features with high-level information; finally, through the feature aggregation encoder and the task alignment encoder, the above two fire features are deeply fused to achieve regional-level detection of fire targets. The experimental results on two datasets show that the method proposed in this application is superior to other comparative models, especially for small-sized fire targets. The accuracy of fire detection and adaptability to different scenes are improved, which is very suitable for early fire target detection with complex scenes.

[0045] The fire target detection method based on local morphological knowledge matching provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be set up separately, integrated on the server 104, or placed on the cloud or other servers. The terminal 102 can send the target fire image to be detected to the server 104. After the server 104 receives the target fire image, for the target fire image, the server 104 pre-processes the target fire image, and based on the fire local morphological feature knowledge base, uses the knowledge template matching technology to extract the local fire morphological features of the pre-processed fire image; uses the detection-oriented Transformer backbone network DOT to extract several fire features of different scales of the pre-processed fire image; aggregates, aligns, classifies fire targets and regresses bounding boxes on the local fire morphological features and several fire features of different scales to obtain the fire target category and the marked box coordinates. The server 104 can feedback the obtained fire target category and marked box coordinates to the terminal 102. In addition, in some embodiments, the fire target detection method based on local morphological knowledge matching can also be implemented independently by the server 104 or the terminal 102. For example, the terminal 102 can directly perform fire target category and marking frame coordinate detection on the target fire image, or the server 104 can obtain the target fire image from the data storage system and perform fire target category and marking frame coordinate detection on the target fire image.

[0046] The terminal 102 may be, but is not limited to, various desktop computers, laptop computers, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart vehicle-mounted devices, etc. The portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers, or may be a cloud server.

[0047] In an exemplary embodiment, Figure 2 and Figure 3 As shown, a fire target detection method based on local morphological knowledge matching is provided. The method is executed by a computer device, and can be executed by a computer device such as a terminal or a server alone, or by a terminal and a server together. In the embodiment of the present application, the method is applied to Figure 1 The server 104 in the example is used as an example to illustrate the method, which includes the following steps 201 to 208. Among them:

[0048] Step 201: preprocessing the target fire image to obtain a preprocessed fire image; the target fire image is an image obtained by photographing the target fire area.

[0049] In order to enhance the generalization ability of the fire target detection model and avoid model overfitting, this application uses a data enhancement method to preprocess the input fire image to obtain a data enhanced fire image. H and W are the height and width of the fire image after data enhancement, respectively. There are two types of classic data enhancement methods: geometric transformation and pixel transformation. Geometric transformation methods include flipping, rotation, cropping, scaling, translation, and jitter, while pixel transformation methods include adding Gaussian noise, applying Gaussian blur, adjusting contrast, brightness, saturation, and histogram equalization.

[0050] As an optional implementation, during preprocessing, the data enhancement methods used include scaling, horizontal flipping, padding, cropping, and normalization.

[0051] Step 202: Based on a knowledge base of local fire morphology features, a knowledge template matching technique is used to extract local fire morphology features of the preprocessed fire image; the knowledge base of local fire morphology features includes a plurality of knowledge template images; the knowledge template images are images including different local flame morphologies selected from a fire image data set.

[0052] This application constructs a knowledge base of local fire morphological features by extracting morphological prior knowledge from typical fire images. Flames have special shapes and diffusion patterns, such as sharp tops, irregular edges, cavities, flow diffusion, and edge bending. This application summarizes these characteristics of flames as morphological prior knowledge of flames. Then, using the knowledge matching method, the fire image is matched with the knowledge template in the local fire morphological knowledge base to mine the local fire morphological features of the fire target in the fire image.

[0053] As an optional implementation, based on the Our_flame_smoke dataset and the Kaggle_flame_smoke dataset, the morphological prior knowledge in typical fire images is extracted to construct a knowledge base of local morphological features of fire. Among them, t i is the i-th knowledge template image, and d is the number of knowledge template images. Each knowledge template image is obtained by intercepting a typical fire image in the data set, and represents different prior knowledge of the local shape of the flame. To ensure the accuracy of extraction, this step can be completed by humans.

[0054] Our_flame_smoke dataset: This dataset is self-constructed. A total of 5900 fire images were obtained by combining self-designed fire experiments and online collection. This dataset mainly consists of indoor fire scenes, and a small part is outdoor fire scenes. These images contain a series of challenging scenes, and the interference sources include colored fire-like targets, floating clouds, occlusions, and reflected light and shadows.

[0055] Kaggle_flame_smoke dataset: This dataset collects 23,730 images of smoke and flames in multiple stages, which are taken with mobile phones under various lighting conditions (indoors and outdoors) and weather conditions. The dataset also includes typical household scenes such as home cooking, farm crop burning, garbage burning, and plastic burning, covering a comprehensive range of fire scenarios.

[0056] Step 203: extracting fire features of several different scales from the preprocessed fire image using a deep fire detector; the deep fire detector is a detection-oriented Transformer backbone network DOT, also referred to as a Transformer-based backbone network or a DOT backbone network in this application.

[0057] Step 204: using a feature aggregation encoder to aggregate the local fire morphological features and a plurality of fire features of different scales to obtain a single-level fire semantic feature.

[0058] Step 205: Based on the single-level fire semantic features, a task alignment encoder is used to generate aligned classification features and regression features.

[0059] Step 206: Perform fire target classification and bounding box regression based on the aligned classification features and regression features to obtain the fire target category and marked box coordinates.

[0060] For the input fire image I, the image may contain multiple smoke or flame targets of different sizes. For the input fire image I, the output of this embodiment is a tuple in, represents the class labels of all fire targets detected from the fire image, k is the number of fire targets detected, Among them, smoke means smoke; flame means fire. represents the bounding box of all fire targets, Represents the bounding box coordinates of the i-th fire target detected. Among them, Represents the upper left corner coordinate of the fire target bounding box, Represents the coordinates of the lower right corner of the fire target bounding box. That is, the output of this application is the category of the fire target and its bounding box (the location of the fire target).

[0061] In another exemplary embodiment, in addition to the fire target category and the bounding box of the fire target, the output also includes the corresponding confidence level, which is used to determine the degree of certainty of the final output object (the detected target). in, and Same as the previous embodiment, represents the confidence of the model in the fire target detected from the fire image, It represents the confidence of the detected i-th fire target. It is a floating point number between 0 and 1, indicating the degree of certainty of the model for the detection result. The closer its value is to 1, the more confident the model is in the judgment.

[0062] As an optional implementation, step 202 specifically includes:

[0063] Step 202.1: Slide the knowledge template image in the fire local morphological feature knowledge base pixel by pixel on the preprocessed fire image, and during the pixel sliding process, calculate the correlation score of the overlapping area between the knowledge template image and the preprocessed fire image; the size of the overlapping area is the same as the size of the knowledge template image.

[0064] That is, by sliding the knowledge template image pixel by pixel on the fire image and calculating the feature similarity, the probability that the current slider image (i.e. the part of the fire image corresponding to the overlapping area) is the knowledge template image is determined. The specific implementation process is as follows:

[0065] The fire image I aug Grayscale image m gray Represented as function f(x,y), the grayscale image of the knowledge template image t gray It is expressed as a function t(x,y), whose size is M×N, where M and N are the height and width of the knowledge template image respectively. Different knowledge template images in the fire local morphology knowledge base may have different sizes, so the values ​​of M and N may also be different for different knowledge template images. Calculate the correlation R between the fire image and the knowledge template image:

[0066] R=match(m gray ,t gray );

[0067] Written in convolutional form:

[0068]

[0069] Among them, the normalized correlation coefficient matching is selected, Represents the normalized convolution of two functions; the normalized discrete convolution formula is expressed as follows:

[0070]

[0071] Among them, R(x,y) represents the correlation score between the knowledge template image and the fire image at the position (x,y), -1≤R(x,y)≤1, that is, the matching degree between the knowledge template image and the fire image in the overlapping area, where x and y refer to the coordinates of the upper left corner of the overlapping area of ​​the knowledge template image and the fire image, so its value range is (H-M+1, W-N+1). If the value of R(x,y) is close to 1, it means that there is a high correlation between the fire image and the knowledge template image at the position (x,y); if the value of R(x,y) is close to -1, it means that there is a high anti-correlation; if the value of R(x,y) is close to 0, it means that there is no obvious correlation or anti-correlation. t(i,j) and f(i,j) represent the pixel values ​​of the knowledge template image and the fire image at the position (i,j), respectively. R is a matrix with a shape of (H-M+1,W-N+1). x' and y' are intermediate variables, (x',y')∈(M,N). This formula is used to calculate the matching degree or similarity of the overlapping area of ​​the knowledge template image when the knowledge template image moves pixel by pixel on the fire image.

[0072] Step 202.2: If the correlation score of the overlapping area between the knowledge template image and the preprocessed fire image is within a predetermined threshold range, the portion overlapping with the overlapping area is extracted from the preprocessed fire image to generate a new fire image; the pixels of the new fire image corresponding to the portion outside the overlapping area are set to black, and the new fire image has the same size as the preprocessed fire image. The specific implementation process is as follows:

[0073] By observing the matching effect, the threshold range of the optimal match is set to [0.75, 1], and based on this threshold range, the position information corresponding to the effective match in the matching result is obtained. If R (x, y) is greater than 0.75, it is considered a match, and if R (x, y) is less than 0.75, it is considered a mismatch. The position information corresponding to the effective match is the upper left corner coordinate (x, y) of the effective matching area. According to the upper left corner coordinate and the height and width of the knowledge template image, the lower right corner coordinate (x+N, y+M) is calculated. The set of matched rectangular areas is:

[0074] Rect(I aug )=∪ x,y,i＝1,…,d {r i (x,y,x+N,y+M)|R(x,y)∈[0.75,1]};

[0075] Among them, r(x,y,x+N,y+M) represents the rectangular area formed by the upper left corner coordinates (x,y) and the lower right corner coordinates (x+N,y+M), that is, the valid matching area.

[0076] According to the matched rectangular area set, the part of the fire image corresponding to the rectangular area set is extracted, and the pixels not in the rectangular area set are set to black. The calculation formula is as follows:

[0077] I match =Region(Rect(I aug ),I aug );

[0078] Among them, I match is the generated new fire image, Region(·) is the extraction function, I match The size and I aug Consistent.

[0079] Step 202.3: Position-encode the new fire image to obtain the local fire morphological features. The specific implementation process is as follows:

[0080] According to the following formula, I match Perform position coding to obtain the local fire morphological characteristics F local :

[0081]

[0082] Among them, fea emb(·) It represents the feature encoding function, including position encoding and convolution operations. The encoding function is existing and comes from the prior art DFFT.

[0083] As an optional implementation, step 203 specifically includes:

[0084] A deep fire detector is used to extract first-scale fire features, second-scale fire features, third-scale fire features and fourth-scale fire features from the preprocessed fire image; wherein the first-scale fire features are fire features output by the network structure corresponding to the first stage of the detection-oriented Transformer backbone network DOT; the second-scale fire features are fire features output by the network structure corresponding to the second stage of the detection-oriented Transformer backbone network DOT; the third-scale fire features are fire features output by the network structure corresponding to the third stage of the detection-oriented Transformer backbone network DOT; the fourth-scale fire features are fire features output by the network structure corresponding to the fourth stage of the detection-oriented Transformer backbone network DOT; the network structure corresponding to the first stage includes a DOT block, and the DOT block includes several local spatial attention layers and a global channel attention block; the network structures corresponding to the second stage, the third stage and the fourth stage each include a DOT block and a semantic enhancement attention module.

[0085] This embodiment uses the DOT backbone network F to extract four multi-scale features of fire with high-level information. The DOT backbone network F stacks an embedding module and four DOT stages, each of which consists of a DOT block and a semantic enhancement attention module (SAA) (except the first stage). Among them, SAA aggregates the underlying semantic information of every two consecutive DOT stages. For each fire image after data enhancement, The four features of different scales extracted by the DOT backbone network F are expressed as follows:

[0086]

[0087] in, is the i-th feature, C i is the number of channels, i∈{1,2,3,4}.

[0088] The DOT backbone network structure and the working process of extracting multi-scale fire features include:

[0089] (1) Embedding module: This module divides the input image into multiple patches to adapt to the input data size of the Transformer-based backbone network. Specifically, given an enhanced image I aug , first divide it into These image blocks are then sent to linear projection to obtain the embedding of the image blocks. The calculation formula is as follows:

[0090]

[0091] Among them, F embed is the embedded function.

[0092] (2) DOT block: Each DOT phase contains a DOT block F block , which is used to effectively capture the local spatial and global semantic relations of fire at each scale. The DOT block contains multiple local spatial attention layers and a global channel attention block, and each global channel attention block contains an attention layer and a feedforward network layer. represents the output features of the DOT block in the i-th DOT stage. Placing a lightweight global channel attention layer after the consecutive local spatial attention layers can derive the overall fire target semantics at each scale. The local spatial attention in the DOT block is a multi-head self-attention mechanism for processing spatial information and aggregating features in local windows. For the input sequence Among them, B lsa is the batch size, L is the sequence length, C lsa is the number of feature channels. The self-attention in local spatial attention is defined as follows:

[0093]

[0094] in, is the number of columns of matrix Q and matrix V; Through X lsa With the weight matrix W Q , W K and W V Multiply to get; d v is the dimension of V, n and m are the number of positions in the local window and the length of the input sequence, respectively.

[0095] The global channel attention in the DOT block is a self-attention mechanism for aggregating information between different feature channels of a tensor. The self-attention mechanism in the global channel attention is performed in the channel direction. Among them, B gca is the batch size, H gca and W gca is the height and width of the feature map, C gca is the number of channels. Represents the feature map after reshaping, where N gca =H gca ×W gca , the feature vector of each pixel is expressed as The self-attention formula in global channel attention is:

[0096]

[0097] Among them, W P is the learnable parameter matrix.

[0098] (3) Semantic Enhanced Attention (SAA): Although the DOT block enhances the fire semantic information of low-level features through global channel attention, there is still room for further improvement to enhance the detection effect. Therefore, the SAA module F is used. se-att It promotes the exchange of semantic information between fire features at two adjacent scales and enhances their features. The SAA module contains an upsampling layer and a global channel attention. The SAA module is integrated into two adjacent DOT blocks. Formally, SAA takes the output of the current DOT block and the output of the previous DOT stage as input and returns semantically enhanced fire features. These features are sent to the next DOT stage and are used to evaluate the final multi-scale fire features. The output features of SAA in the i-th DOT stage are given by express.

[0099] (4) DOT stage: The DOT backbone network contains 4 DOT stages, each of which consists of a DOT block and an SAA module (except the first stage). Specifically, the first stage contains a DOT block but no SAA module because the input of the SAA module comes from two consecutive DOT stages. The remaining three stages respectively contain a patch merging module, a DOT block, a SAA module, and a downsampling layer for restoring the input dimension. The patch merging module is used to reduce the number of patches, thereby reducing the computational complexity and memory consumption of the model. The calculation formula of the DOT block in the i-th stage is defined as:

[0100]

[0101] Where down represents the downsampling function. The SAA module of the i-th stage is defined as:

[0102]

[0103] Among them, up represents the upsampling function. Finally, the multi-scale fire features obtained by the DOT backbone network for:

[0104]

[0105] As an optional implementation, step 204 specifically includes:

[0106] The local fire morphological features and the fire features of several different scales are aggregated according to a calculation formula to obtain a single-level fire semantic feature; the calculation formula is:

[0107]

[0108] s sae =s3;

[0109] Among them, Cat(·) represents the concatenation operation, Conv(·) represents the convolution operation, and They are the first scale fire characteristics, the second scale fire characteristics, the third scale fire characteristics and the fourth scale fire characteristics; S att represents the global channel attention operation, s sae represents the final aggregated features; up(·) represents the upsampling operation.

[0110] This step uses three FAE modules, which are connected in series. Each FAE module takes two features as input and aggregates these features step by step in all FAE blocks. The scale of the final aggregated feature is set to To balance detection accuracy and computational cost. The last FAE block upsamples the input features to

[0111] As an optional implementation, step 205 and step 206 specifically include:

[0112] The task alignment encoder TAE is used to simultaneously generate aligned classification features and regression features, perform classification and regression tasks, and obtain the fire target category and marking box coordinates. The encoder includes two channel attention blocks. First, the stacked group channel attention block T group Aggregate feature s sae Divided into two parts. Second, the global channel attention block T global Further encode the regression feature t reg , used for subsequent regression tasks. This process can be expressed as:

[0113] t1,t2=T group (s sae) ;

[0114] t cls =t1;

[0115] t reg =T global (t2);

[0116] in, are the two sub-features that are divided, and They are the two fire features that are ultimately used for fire target classification and bounding box regression tasks, respectively.

[0117] According to the obtained classification features and regression features, a fully connected layer is used to realize fire target classification and bounding box regression. This process uses two different functions G cls and G reg It is expressed as follows:

[0118]

[0119] During the training process, the difference between the output of these two functions and the true value is minimized to improve the fire detection accuracy.

[0120] The parameter updates involved in the training process are as follows: 1) The knowledge template matching process does not involve parameter updates. match The parameters of the feature encoding function corresponding to the encoding stage will be updated. 2) In the deep fire feature extraction stage, the parameters of the embedding module will be updated, the weight matrix and bias related to the Q, K, V calculation of the local spatial attention block and the global channel attention block in the DOT block, and the parameters of the feedforward network layer will be updated, and the weight matrix and bias related to the Q, K, V calculation in the attention in SAA will also be updated. 3) The parameters related to Satt in the FAE module will also be updated. 4) The parameters related to the group channel attention Tgroup and the global channel attention Tglobal in the task alignment encoder TAE will be updated. 5) The two different functions Gcls and Greg used in the fire target classification and bounding box regression process are implemented by the fully connected layer, and the parameters of its fully connected layer will also be updated.

[0121] The method proposed in this application performs single-level depth prediction on a single fire feature map, so the predefined anchor boxes are sparse, causing the detector to focus on large true boxes and ignore small true boxes during training. To overcome this problem, the unified matching strategy of YOLOF is used to ensure that all true boxes are consistently matched with the same number of positive anchor boxes, regardless of their size. Therefore, the loss function of this application is It consists of the Focal loss for classification and the Generalized Intersection over Union (GIOU) loss for regression:

[0122]

[0123] in, Focal loss for predicted and true categories, is the GIoU loss between the predicted bounding box and the true bounding box. cls and λ giou are the coefficients of each component.

[0124] The following cases are used to verify the technical effect of the method proposed in this application:

[0125] Firstly, a quantitative analysis of baseline performance comparison is performed from both global and local perspectives on a private dataset (Our_flame_smoke) and a public dataset (Kaggle_flame_smoke). Then, a qualitative visualization analysis of fire detection results for small, medium, and large-scale fire targets is performed.

[0126] Experimental settings and evaluation indicators: The parameter selection and details of the model training process are described as follows: the experiment is deployed on a computer with an NVIDIA GeForce RTX 3090 GPU; the Pytorch deep learning framework is used for modeling, and CUDA, Cudnn and OpenCV libraries are called to train and test the fire detection model; in the experiment, the image size is set to 1333×800 after data augmentation; the fire detection model is optimized using the Adam optimizer, with an initial learning rate of 0.02 and a batch size of 6; in both datasets, the images are randomly divided into training set: validation set: test set with a ratio of 8:1:1; the evaluation indicator used in the experiment is AP smoke 、AP flame The larger the values ​​of the three indicators, the better the recognition effect. Among them, AP (Average Precision) is the average precision, which is a commonly used indicator in target detection; AP smoke Refers to the average precision of smoke detection, AP flame Refers to the average accuracy of flame detection; meanAverage Precision (mAP) is the average accuracy of each target under a certain detection threshold (set to 0.5 in this case).

[0127] 1) Quantitative analysis

[0128] Table 1 Results of LMKMFD and comparison methods on two datasets (test sets)

[0129]

[0130] Note: The values ​​in bold in Table 1 are the optimal index values, and the values ​​in underline are the suboptimal index values. The values ​​in brackets represent the values ​​that the model proposed in this application improves over the suboptimal model. Table 2 is the same as Table 1.

[0131] The experimental results of the method proposed in this application and the baseline method are shown in Table 1, where LMKMFD represents the method proposed in this application. By observation, the following information can be obtained:

[0132] On the Our_flame_smoke dataset, LMKMFD outperforms all the comparison models in all three metrics. Specifically, LMKMFD has a smokeIt performs better than the region extraction based model Faster RCNN, improving by 2.43%. Compared with the regression based model, LMKMFD has a higher AP smoke ,AP flame In terms of mAP and mAP, they are 3.34%, 0.44%, and 1.89% higher than the best model YOLOF, respectively. Compared with the Transformer-based methods DETR, Deformable DETR, AdaMixer, DFFT, and DINO, LMKMFD improves mAP by 3.68%, 9.65%, 2.05%, 0.98%, and 2.55%, respectively.

[0133] On the Kaggle_flame_smoke dataset, LMKMFD has the best AP smoke Compared with the region extraction model Faster RCNN, LMKMFD outperforms the comparison model in AP and mAP indicators. smoke Compared with the regression-based model FCOS, LMKMFD has a 4.23% improvement in AP and a 3.47% improvement in mAP. smoke Compared with DINO, the best Transformer-based model, LMKMFD improves by more than 1.3% in three indicators. Compared with DFFT, LMKMFD improves by 6.6% in AP smoke and mAP indicators respectively achieved 4.19% and 2% growth. flame The performance on is not optimal, but considering the small test sample size, the difference between it and DFFT is actually small.

[0134] Overall, the above experimental results show that LMKMFD outperforms other comparison models on both datasets, demonstrating its effectiveness and generalization in fire target detection tasks. The LMKMFD model performs better in detecting flames because the transparency and movement of smoke usually cause it to blend with the background, and it is challenging to distinguish smoke-like targets from noise interference. In contrast, flames have distinct features such as color and edges that help with accurate positioning.

[0135] Table 2 mAP of small, medium and large size targets on Kaggle_flame_smoke

[0136]

[0137] The method proposed in this application performs well in small-sized smoke and flame detection. Table 2 and Figure 4The mAP comparison results of different methods on fire targets of different scales are shown. According to the proportion of fire targets in the fire images, the fire images are classified into three different scales: small targets (0%-1%), medium targets (1%-10%), and large targets (10%-100%). Only the results on the Kaggle_flame_smoke dataset are shown here because it contains more fire scenes than Our_flame_smoke. The results show that LMKMFD performs best in terms of detection accuracy of small-scale fires, and also achieves the best results in the detection of medium-scale and large-scale fires.

[0138] 2) Visual qualitative analysis

[0139] Figure 5 The effect of LMKMFD in early fire detection is intuitively demonstrated. For fire images containing only small smoke or flames, such as Figure 5 (a) Figure 5 (b) and Figure 5 As shown in (c), LMKMFD accurately marks the detection box for each fire target without redundancy. In addition, the detection box closely fits the actual area of ​​the fire target, and LMKMFD correctly identifies the category of each fire target. For images containing both small smoke and flames, such as Figure 5 As shown in (d), LMKMFD distinguishes between smoke and flames and marks different detection boxes according to the category. Even in the presence of interference factors such as gray sky and walls, which are similar to smoke in color level, LMKMFD will mask the interference features and extract important fire features for detection. In the early stages of a fire, fire targets usually appear as small-scale smoke or flames. The LMKMFD model has high accuracy in identifying and locating small targets and is suitable for early fire detection tasks.

[0140] The method proposed in this application is suitable for processing fire images of different scales. Figure 6 and Figure 7 The visualization results of the proposed method (LMKMFD) and the comparative methods (Faster RCNN, FCOS, YOLOF, DETR, Deformable DETR, AdaMixer, DFFT and DINO) in processing fire images of different scales (small, medium and large) are shown. The first, second and third rows in the figure represent small, medium and large-scale targets, respectively. Specifically, in the case of medium-scale fires, LMKMFD is the only method that can detect flames under smoke occlusion. For small and large fires, the bounding boxes generated by LMKMFD can better adapt to the fire targets and there are no redundant bounding boxes. Figure 4The results are consistent with those shown in Figure 2, showing that LMKMFD outperforms multiple comparison models in identifying complex fires of various sizes and scales.

[0141] Typical and challenging images from the two datasets were selected for the visual qualitative analysis. Figure 5 (a) and (b) are from the Kaggle_flame_smoke dataset. Figure 5 (c) and (d) are from the dataset Our_flame_smoke. Figure 6 and Figure 7 The first row of images comes from the dataset Our_flame_smoke, and the second and third rows of images come from the dataset Kaggle_flame_smoke. The numbers after smoke and flame in the figure refer to the confidence level, which is the degree of certainty of the model in its single prediction result, that is, the credibility of the result of judging it as smoke or flame. The larger the value, the more confident the model is in the judgment.

[0142] The present application also provides an application scenario, which applies the above-mentioned fire target detection method based on local morphological knowledge matching. Specifically: the fire target detection method based on local morphological knowledge matching provided in this embodiment can be applied in a fire detection scenario. The fire detection scenario includes a fire image acquisition link, a fire image transmission link and a fire image detection link. In the fire image acquisition link, a fire image of the fire area is obtained by shooting, and enters the fire image detection link through the fire image transmission link. In the fire image detection link, fire target detection is performed on the obtained fire image to obtain the fire target category and marking box coordinates. The fire target detection method based on local morphological knowledge matching provided in this embodiment belongs to the fire image detection link. Specifically, in the process of detecting the obtained fire image, local morphological knowledge matching and multi-scale fire feature extraction are combined to achieve accurate detection of small targets in the early stage of the fire.

[0143] Based on the same inventive concept, the embodiment of the present application also provides a device for implementing the fire target detection method based on local morphological knowledge matching involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more embodiments of the fire target detection device based on local morphological knowledge matching provided below can refer to the limitations of the fire target detection method based on local morphological knowledge matching above, and will not be repeated here.

[0144] In an exemplary embodiment, Figure 8 As shown, a fire target detection device based on local morphological knowledge matching is provided, comprising:

[0145] The preprocessing module M1 is used to preprocess the target fire image to obtain a preprocessed fire image; the target fire image is an image obtained by photographing the target fire area.

[0146] The local fire morphology feature extraction module M2 is used to extract the local fire morphology features of the preprocessed fire image based on the fire local morphology feature knowledge base and the knowledge template matching technology; the fire local morphology feature knowledge base includes a plurality of knowledge template images; the knowledge template images are images including different flame local morphologies screened out from the fire image data set.

[0147] The multi-scale fire feature extraction module M3 is used to extract fire features of several different scales from the pre-processed fire image using a deep fire detector; the deep fire detector is a detection-oriented Transformer backbone network DOT.

[0148] The aggregation module M4 is used to aggregate the local fire morphological features and a plurality of fire features of different scales using a feature aggregation encoder to obtain a single-level fire semantic feature.

[0149] The task alignment module M5 is used to generate aligned classification features and regression features based on the single-level fire semantic features using a task alignment encoder.

[0150] The classification and regression module M6 is used to perform fire target classification and bounding box regression according to the aligned classification features and regression features to obtain the fire target category and marking box coordinates.

[0151] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Fig. 9 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store fire image data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a fire target detection method based on local morphological knowledge matching is implemented.

[0152] Those skilled in the art will understand that Fig. 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components. In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0153] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0154] In an exemplary embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0155] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0156] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0157] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0158] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0159] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A fire target detection method based on local morphological knowledge matching, characterized in that: The fire target detection method based on local morphological knowledge matching includes: Preprocessing the target fire image to obtain a preprocessed fire image; the target fire image is an image obtained by photographing the target fire area; Based on a knowledge base of local fire morphology features, a knowledge template matching technique is used to extract local fire morphology features of the preprocessed fire image; the knowledge base of local fire morphology features includes a plurality of knowledge template images; the knowledge template images are images including different local flame morphologies selected from a fire image data set; A deep fire detector is used to extract fire features of several different scales from the preprocessed fire image; the deep fire detector is a detection-oriented Transformer backbone network DOT; A feature aggregation encoder is used to aggregate the local fire morphological features and a plurality of fire features of different scales to obtain a single-level fire semantic feature; Based on the single-level fire semantic features, a task alignment encoder is used to generate aligned classification features and regression features; According to the aligned classification features and regression features, fire target classification and bounding box regression are performed to obtain the fire target category and marked box coordinates.

2. The fire target detection method based on local morphological knowledge matching according to claim 1 is characterized in that: Based on the fire local morphological feature knowledge base, the local fire morphological features of the preprocessed fire image are extracted using the knowledge template matching technology, specifically including: Slide the knowledge template image in the fire local morphological feature knowledge base pixel by pixel on the preprocessed fire image, and calculate the correlation score of the overlapping area between the knowledge template image and the preprocessed fire image during the pixel by pixel sliding process; the size of the overlapping area is the same as the size of the knowledge template image; If the correlation score of the overlapping area between the knowledge template image and the preprocessed fire image is within a predetermined threshold range, the portion overlapping with the overlapping area is extracted from the preprocessed fire image to generate a new fire image; the pixel points of the new fire image corresponding to the portion outside the overlapping area are set to black, and the new fire image has the same size as the preprocessed fire image; Position encoding is performed on the new fire image to obtain the local fire morphological features.

3. The fire target detection method based on local morphological knowledge matching according to claim 2 is characterized in that: Calculate the correlation score of the overlapping area between the knowledge template image and the preprocessed fire image, including: According to the normalized discrete convolution formula, the correlation score of the overlapping area between the knowledge template image and the preprocessed fire image is calculated; the normalized discrete convolution formula is as follows: Among them, R(x,y) is the correlation score of the overlapping area between the knowledge template image and the preprocessed fire image, (x,y) is the coordinate of the upper left corner of the overlapping area, (x,y)∈(H-M+1,W-N+1), H and W are the height and width of the preprocessed fire image, M and N are the height and width of the knowledge template image, respectively; t(i,j) and f(i,j) represent the pixel values ​​of the knowledge template image and the preprocessed fire image at position (i,j), respectively; x' and y' are intermediate variables, (x',y')∈(M,N).

4. The fire target detection method based on local morphological knowledge matching according to claim 2 is characterized in that: The predetermined threshold range is [0.75, 1].

5. The fire target detection method based on local morphological knowledge matching according to claim 1 is characterized in that: A deep fire detector is used to extract fire features of several different scales from the preprocessed fire image, specifically including: A deep fire detector is used to extract first-scale fire features, second-scale fire features, third-scale fire features and fourth-scale fire features from the preprocessed fire image; wherein the first-scale fire features are fire features output by the network structure corresponding to the first stage of the detection-oriented Transformer backbone network DOT; the second-scale fire features are fire features output by the network structure corresponding to the second stage of the detection-oriented Transformer backbone network DOT; the third-scale fire features are fire features output by the network structure corresponding to the third stage of the detection-oriented Transformer backbone network DOT; the fourth-scale fire features are fire features output by the network structure corresponding to the fourth stage of the detection-oriented Transformer backbone network DOT; the network structure corresponding to the first stage includes a DOT block, and the DOT block includes several local spatial attention layers and a global channel attention block; the network structures corresponding to the second stage, the third stage and the fourth stage each include a DOT block and a semantic enhancement attention module.

6. The fire target detection method based on local morphological knowledge matching according to claim 5 is characterized in that: The local fire morphological features and the fire features of several different scales are aggregated by a feature aggregation encoder to obtain a single-level fire semantic feature, specifically including: The local fire morphological features and the fire features of several different scales are aggregated according to a calculation formula to obtain a single-level fire semantic feature; the calculation formula is: and sae =s3; Among them, Cat(·) represents the concatenation operation, Conv(·) represents the convolution operation, and They are the first scale fire characteristics, the second scale fire characteristics, the third scale fire characteristics and the fourth scale fire characteristics; S att represents the global channel attention operation, s sae represents the final aggregated features; up(·) represents the upsampling operation.

7. A fire target detection device based on local morphological knowledge matching, characterized in that: The fire target detection device based on local morphological knowledge matching comprises: A preprocessing module is used to preprocess the target fire image to obtain a preprocessed fire image; the target fire image is an image obtained by photographing the target fire area; A local fire morphology feature extraction module is used to extract the local fire morphology features of the preprocessed fire image by using a knowledge template matching technology based on a fire local morphology feature knowledge base; the fire local morphology feature knowledge base includes a plurality of knowledge template images; the knowledge template images are images including different flame local morphologies selected from a fire image data set; A multi-scale fire feature extraction module, used to extract fire features of several different scales from the pre-processed fire image using a deep fire detector; the deep fire detector is a detection-oriented Transformer backbone network DOT; An aggregation module, used for aggregating the local fire morphological features and a plurality of fire features of different scales using a feature aggregation encoder to obtain a single-level fire semantic feature; A task alignment module, used for generating aligned classification features and regression features based on the single-level fire semantic features by using a task alignment encoder; The classification and regression module is used to perform fire target classification and bounding box regression according to the aligned classification features and regression features to obtain the fire target category and marked box coordinates.

8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the fire target detection method based on local morphological knowledge matching as described in any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the fire target detection method based on local morphological knowledge matching described in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the fire target detection method based on local morphological knowledge matching described in any one of claims 1 to 6 is implemented.

Images