A flame and smoke recognition method, system and storage medium

Abstract

The application discloses a flame and smoke identification method, and relates to the technical field of computer vision, and the method comprises the following steps: generating a boundary box of a flame and smoke target through a YOLOv target detection framework; extracting multi-scale visual features by using an image encoder of a SegmentAnything Model (SAM); encoding boundary box coordinates into a prompt embedding vector; fusing the visual features and the prompt embedding by using a mask decoder to generate a segmentation mask and a confidence score; confirming the target when the score exceeds a threshold value, and superimposing the boundary box and the segmentation mask and outputting to a fire-fighting early warning system.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, specifically to a real-time flame and smoke recognition method based on deep learning, applicable to fire early warning systems. Background Technology

[0002] Firefighting is a crucial safety guarantee system in modern society, its importance manifesting in multiple dimensions. It not only concerns individual life and property safety but also relates to social order and economic development. Construction sites are complex environments, presenting numerous fire hazards at different stages of construction, such as material storage, temporary electrical connections, and hot work operations. The complexity and necessity of fire prevention at these sites pose significant challenges to the construction industry.

[0003] Traditional fire protection technology mainly uses sensors to detect flames or smoke. However, for the construction industry, this type of technology is only applicable to relatively simple environmental factors and is only at the detection stage. It is difficult to meet the construction industry's needs for the entire process of fire safety prevention, detection, and response. Summary of the Invention

[0004] In view of the technical defects and drawbacks existing in the prior art, embodiments of the present invention provide a flame and smoke identification method, system, and storage medium to overcome the above problems or at least partially solve the above problems, the specific solutions of which are as follows:

[0005] See Figure 1 As shown, an embodiment of the present invention provides a flame and smoke recognition method, which includes the following steps:

[0006] Step 1, Target Recognition Model: The input image is processed using the YOLOv target detection framework to generate bounding boxes for flame and smoke targets. YOLOv includes, but is not limited to, the latest YOLOv11 and YOLOv12.

[0007] Step 2, Image Feature Extraction: The input image is processed using the SegmentAnything Model (SAM) image encoder to generate the visual feature embedding of the image;

[0008] Step 3, Cue Encoding: Use SAM's cue encoder to encode the bounding box coordinates generated in Step 1 into cue embedding vectors;

[0009] Step 4, Target Segmentation and Scoring: Use the SAM mask decoder to fuse the visual feature embeddings from Step 2 and the cue embedding vectors from Step 3 to generate the target segmentation mask and its corresponding confidence score.

[0010] Step 5, Target Confirmation: When the confidence score generated in Step 4 exceeds the preset threshold, the presence of a flame or smoke target is confirmed.

[0011] Step 6, Output the result: Overlay the bounding box from Step 1 with the segmentation mask from Step 4, and output the result to the fire alarm system.

[0012] Furthermore, the multi-scale feature extraction of the image encoder in step 2 includes the following steps:

[0013] (a1) Extracting multi-level feature maps: Four levels of feature maps are extracted from the backbone network of the image encoder, with spatial resolutions of the four levels as follows:

[0014] Level 1: 1 / 4 of the input image resolution;

[0015] Level 2: 1 / 8 of the input image resolution;

[0016] Level 3: 1 / 16 of the input image resolution;

[0017] Level 4: 1 / 32 of the input image resolution;

[0018] (b1) Level 1 coordinate alignment: Use the IntermediateLayerGetter module to perform coordinate alignment on the feature map of level 1 so that it shares the same spatial coordinate reference with the feature maps of level 2, level 3 and level 4;

[0019] (c1) Channel dimension unification: A 4x4 convolutional kernel with a stride of 1 is used to process the feature maps of the four levels respectively, and their channel dimensions are uniformly mapped to 256 dimensions.

[0020] Furthermore, the mask decoder processing flow in step 4 includes the following steps in sequence:

[0021] (i) Image embedding preprocessing:

[0022] The image embedding is spatially deformed to generate a deformable feature tensor.

[0023] The deformable feature tensor is processed using the first transposed convolutional layer: convolutional kernel 4x4, stride 2, output channels reduced to 128 dimensions;

[0024] The output of the first transposed convolutional layer is processed using a second transposed convolutional layer: convolutional kernel 4x4, stride 2, output channels reduced to 32 dimensions;

[0025] (ii) Hints on embedding separation and confidence score generation:

[0026] Separate the mask token and confidence token from the cue embedding vector;

[0027] The confidence token is sequentially input into three fully connected layers for prediction. The number of neurons in the fully connected layers are 128, 64 and 1, respectively, and finally a confidence score for the existence of the target is generated.

[0028] (iii) Target segmentation mask generation:

[0029] Perform matrix multiplication on the 32-dimensional feature tensor output in step (i) and the mask token separated in step (ii);

[0030] Apply the Sigmoid activation function to the matrix multiplication result to generate a segmentation mask for the target;

[0031] (iv) Output determination:

[0032] When the confidence score generated in step (ii) is greater than the preset threshold of 0.85, the segmentation mask generated in step (iii) and the bounding box corresponding to step 1 are output to the early warning system.

[0033] In the above embodiments, end-to-end efficiency is achieved by simultaneously outputting the segmentation mask and quality rating in a single feedforward, parameter quantity is reduced by 75% by channel compression, and high-precision segmentation is achieved by using Sigmoid output to adapt to the soft edge characteristics of flame / smoke.

[0034] Furthermore, the deformable attention mechanism employed by the image encoder in step 2 includes the following steps:

[0035] (a2) Query initialization:

[0036] Extract the query vector Z with spatial location index q from the 256-dimensional feature map output by the image encoder. q ;

[0037] Get and query vector Z q The corresponding two-dimensional reference point coordinates p q =(x q ,y q The coordinates are obtained by transforming the center point coordinates of the bounding box predicted by YOLOv in step 1;

[0038] (b2) Dynamic offset calculation:

[0039] Through a fully connected layer, for each query vector Z q Generate M=8 attention heads, each corresponding to K=4 learnable offsets Δp mqk Where m∈[1,8] identifies the attention head number, and k∈[1,4] identifies the sampling point number;

[0040] Each offset is constrained to satisfy ||Δp mqk ∥≤3 (with reference point p) q(pixel distance from the origin);

[0041] (c2) Feature-weighted aggregation:

[0042] Calculate the output features for query position q using the following formula:

[0043]

[0044] Among them, W m and W m 'represent the output projection matrix and feature transformation matrix of the m-th attention head, respectively.

[0045] x(p q +Δp mqk ) represents the position (x) on the feature map x. q +Δx mqk ,y q +Δy mqk The eigenvalues ​​at point () are obtained through bilinear interpolation;

[0046] SAM qk The attention weight for the m-th head and the k-th sampling point is calculated using the following formula:

[0047]

[0048] Among them, W m " is the query transformation matrix of the m-th attention head, and d is the feature dimension.

[0049] Furthermore, the YOLOv object detection framework is trained using a dynamic area-weighted loss function, the construction of which includes the following steps:

[0050] (a3) For each bounding box i in the training batch, calculate its dynamic area weight:

[0051]

[0052] in:

[0053] A i =w i ×h i Represents the pixel area of ​​the target box i ((w i h is the width. i (for height)

[0054] This represents the average pixel area of ​​all N bounding boxes in the current training batch;

[0055] The clamp(value,min,max) function means that when value exceeds the range [min,max], it will be forcibly truncated to the boundary value of the interval (i.e., if it is less than min, take min, and if it is greater than max, take max);

[0056] (b3) Calculate the weighted L1 loss:

[0057] Calculate the coordinates of the prediction box with the coordinates of the true bounding box b i =(x i ,y i ,w i ,h i The coordinate error of ) is calculated, and the area weight is applied using step (a3):

[0058]

[0059] Special Limited Edition L L1 For weighted L1 loss: the area weight area_weight i Only applies to the width error term and height error term Error terms not applied to center point coordinates and

[0060] Furthermore, the total bounding box regression loss L of the YOLOv object detection framework box It is composed of the following three loss terms weighted together:

[0061]

[0062] in:

[0063] (a4) Definition of Loss Item

[0064] L L1 : The weighted L1 loss (including area weight) as defined in claim 5;

[0065] L GIoU The generalized crossover ratio loss is calculated using the following formula:

[0066]

[0067] Where b is the predicted bounding box, b gt C is the bounding box between b and b'. gt The minimum convex hull region;

[0068] L mse The formula for calculating the mean square error loss of dimensions is:

[0069]

[0070] Where N is the number of targets in the batch. To predict the width and height of bounding box i, w i h i Let i be the width and height of the actual bounding box.

[0071] (b4) Applying constraints:

[0072] When the target frame area A i <100px 2 At that time, L mse The item's weight is automatically increased to 1.5;

[0073] When calculating LGIoU, outlier bounding boxes with an aspect ratio difference greater than 2:1 are ignored; (i.e.) (target bounding box);

[0074] (c4) Gradient balancing mechanism:

[0075] During the optimization process, the total loss L box The gradient is truncated, and its L2 norm is constrained to satisfy... To prevent L mse Lead the optimization direction.

[0076] Furthermore, in step S3, when the encoder is prompted to process the mask input, the following operations are performed:

[0077] (a5) Mask resolution compression:

[0078] Input mask M in ∈R H×W A compressed mask Mcomp is generated by bilinear interpolation downsampling, with its height maintained at H and its width compressed to [value missing]. (i.e., the resolution becomes) );

[0079] (b5) Feature extraction and downsampling:

[0080] Compress mask M comp Input to the first convolutional layer: kernel size 2x2, stride 2, output channels 64, generating feature maps.

[0081] Input the feature map F1 into the second convolutional layer: kernel size 2x2, stride 2, output channels 128, to generate a feature map.

[0082] (c5) Feature Dimension Enhancement:

[0083] The feature map F2 is input into a 1x1 convolutional layer, increasing the channel dimension from 128 to 256 to generate the final cue embedding vector.

[0084] The prompt embedding vector E p resolution It has the same spatial resolution as the feature map output by the image encoder.

[0085] As a second aspect of the present invention, a flame and smoke recognition system is provided, comprising:

[0086] Image acquisition module: A multispectral camera array deployed at construction sites;

[0087] Edge computing module: A GPU-embedded device equipped with a model corresponding to any of the flame and smoke recognition methods described above;

[0088] Early warning response module: Triggers the activation of the fire extinguishing device when the confidence score exceeds a threshold;

[0089] The edge computing module is connected to the cloud server via a 5G private network to enable dynamic updates of model parameters.

[0090] Furthermore, the edge computing module includes dual processing channels:

[0091] The first channel processes the visible light video stream at 30fps and performs flame recognition.

[0092] The second channel processes the infrared video stream at 15fps to perform smoke detection.

[0093] The dual channels share the image encoder output of SAM through a feature-level fusion network.

[0094] As a third aspect of the present invention, a computer-readable storage medium is provided, wherein a computer program is stored therein, and when executed by a computer, the computer program causes the computer to perform the flame and smoke recognition method as described above.

[0095] The present invention has the following beneficial effects:

[0096] 1. Improved detection accuracy: Small target detection is optimized by using a dynamic area-weighted loss function, which improves the training convergence speed.

[0097] 2. Segmentation edge optimization: By employing multi-level feature alignment combined with deformable attention, the segmentation mask IoU is improved by 15.2%.

[0098] 3. Optimized computational efficiency: The lightweight design of the mask decoder enables an inference speed of 45 FPS (1080P resolution). Attached Figure Description

[0099] Figure 1This is a flowchart illustrating a flame and smoke recognition method provided in an embodiment of the present invention. Detailed Implementation

[0100] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0101] As a first aspect of the present invention, a flame and smoke identification method is provided, comprising the following steps:

[0102] Step 1, Target Recognition Model: The YOLOv target detection framework is used to process the input image to generate bounding boxes for flame and smoke targets;

[0103] Step 2, Image Feature Extraction: The input image is processed using the SegmentAnything Model (SAM) image encoder to generate the visual feature embedding of the image;

[0104] Step 3, Cue Encoding: Use SAM's cue encoder to encode the bounding box coordinates generated in Step 1 into cue embedding vectors;

[0105] Step 4, Target Segmentation and Scoring: Use the SAM mask decoder to fuse the visual feature embeddings from Step 2 and the cue embedding vectors from Step 3 to generate the target segmentation mask and its corresponding confidence score.

[0106] Step 5, Target Confirmation: When the confidence score generated in Step 4 exceeds the preset threshold, the presence of a flame or smoke target is confirmed.

[0107] Step 6, Output the result: Overlay the bounding box from Step 1 with the segmentation mask from Step 4, and output the result to the fire alarm system.

[0108] This invention employs a two-stage detection-segmentation architecture. The YOLOv detector quickly locates the bounding box of the flame / smoke (coarse localization). The SAM segmentation module's image encoder extracts multi-scale visual features, and the cue encoder converts the bounding box into an embedding vector. The mask decoder fuses the visual and cue features, outputting a segmentation mask and confidence score. An early warning is triggered only when the confidence score is greater than a threshold, and the bounding box and mask are superimposed in the output. This approach is compatible with existing fire alarm systems. YOLOv ensures real-time performance, while SAM improves segmentation precision.

[0109] In view of the characteristics of flames and smoke in the early stage of fire, which are small and difficult to identify, the flame and smoke identification method of the present invention can accurately identify flames and smoke, reduce manual monitoring, reduce labor intensity, and improve the detection efficiency and accuracy of flame and smoke identification.

[0110] Preferably, the multi-scale feature extraction of the image encoder in step 2 includes the following steps:

[0111] (a1) Extracting multi-level feature maps: Four levels of feature maps are extracted from the backbone network of the image encoder, with spatial resolutions of the four levels as follows:

[0112] Level 1: 1 / 4 of the input image resolution;

[0113] Level 2: 1 / 8 of the input image resolution;

[0114] Level 3: 1 / 16 of the input image resolution;

[0115] Level 4: 1 / 32 of the input image resolution;

[0116] (b1) Level 1 coordinate alignment: Use the IntermediateLayerGetter module to perform coordinate alignment on the feature map of level 1 so that it shares the same spatial coordinate reference with the feature maps of level 2, level 3 and level 4;

[0117] (c1) Channel dimension unification: A 4x4 convolutional kernel with a stride of 1 is used to process the feature maps of the four levels respectively, and their channel dimensions are uniformly mapped to 256 dimensions.

[0118] In the above embodiments, high-resolution level 1 features are retained to improve the capture of smoke / flame details, thereby enhancing small target detection. By unifying the number of channels and coordinate reference, subsequent decoding operations are simplified, and feature fusion compatibility is achieved.

[0119] Preferably, the mask decoder processing flow in step 4 includes the following steps in sequence:

[0120] (i) Image embedding preprocessing:

[0121] The image embedding is spatially deformed to generate a deformable feature tensor.

[0122] The deformable feature tensor is processed using the first transposed convolutional layer: convolutional kernel 4x4, stride 2, output channels reduced to 128 dimensions;

[0123] The output of the first transposed convolutional layer is processed using a second transposed convolutional layer: convolutional kernel 4x4, stride 2, output channels reduced to 32 dimensions;

[0124] (ii) Hints on embedding separation and confidence score generation:

[0125] Separate the mask token and confidence token from the cue embedding vector;

[0126] The confidence token is sequentially input into three fully connected layers for prediction. The number of neurons in the fully connected layers are 128, 64 and 1, respectively, and finally a confidence score for the existence of the target is generated.

[0127] (iii) Target segmentation mask generation:

[0128] Perform matrix multiplication on the 32-dimensional feature tensor output in step (i) and the mask token separated in step (ii);

[0129] Apply the Sigmoid activation function to the matrix multiplication result to generate a segmentation mask for the target;

[0130] (iv) Output determination:

[0131] When the confidence score generated in step (ii) is greater than the preset threshold of 0.85, the segmentation mask generated in step (iii) and the bounding box corresponding to step 1 are output to the early warning system.

[0132] In the above embodiments, end-to-end efficiency is achieved by simultaneously outputting the segmentation mask and quality rating in a single feedforward, parameter quantity is reduced by 75% by channel compression, and high-precision segmentation is achieved by using Sigmoid output to adapt to the soft edge characteristics of flame / smoke.

[0133] Preferably, the deformable attention mechanism employed by the image encoder in step 2 includes the following steps:

[0134] (a2) Query initialization:

[0135] Extract the query vector Z with spatial location index q from the 256-dimensional feature map output by the image encoder. q ;

[0136] Get and query vector Z q The corresponding two-dimensional reference point coordinates p q =(x q ,y q The coordinates are obtained by transforming the center point coordinates of the bounding box predicted by YOLOv in step 1;

[0137] (b2) Dynamic offset calculation:

[0138] Through a fully connected layer, for each query vector Z q Generate M=8 attention heads, each corresponding to K=4 learnable offsets Δp mqkWhere m∈[1,8] identifies the attention head number, and k∈[1,4] identifies the sampling point number;

[0139] Each offset is constrained to satisfy ||Δp mqk ∥≤3 (with reference point p) q (pixel distance from the origin);

[0140] (c2) Feature-weighted aggregation:

[0141] Calculate the output features for query position q using the following formula:

[0142]

[0143] Among them, W m and W m 'represent the output projection matrix and feature transformation matrix of the m-th attention head, respectively.

[0144] x(p q +Δp mqk ) represents the position (x) on the feature map x. q +Δx mqk ,y q +Δy mqk The eigenvalues ​​at point () are obtained through bilinear interpolation;

[0145] SAM qk The attention weight for the m-th head and the k-th sampling point is calculated using the following formula:

[0146]

[0147] Among them, W m " is the query transformation matrix of the m-th attention head, and d is the feature dimension.

[0148] In the above embodiments, offset learning enables the model to focus on the irregular edges of flames / smoke, adapting to target deformation. Sparse sampling (only 4 points / head) improves efficiency by 50%+ compared to global attention, reducing computational overhead. Local feature aggregation enhances the recognition of some visible targets and improves anti-occlusion capability.

[0149] Preferably, the YOLOv object detection framework is trained using a dynamic area-weighted loss function, the construction of which includes the following steps:

[0150] (a3) For each bounding box i in the training batch, calculate its dynamic area weight:

[0151]

[0152] in:

[0153] Ai =w i ×h i Represents the pixel area of ​​the target box i ((w i h is the width. i (for height)

[0154] This represents the average pixel area of ​​all N bounding boxes in the current training batch;

[0155] The clamp(value,min,max) function means that when value exceeds the range [min,max], it will be forcibly truncated to the boundary value of the interval (i.e., if it is less than min, take min, and if it is greater than max, take max);

[0156] (b3) Calculate the weighted L1 loss:

[0157] Calculate the coordinates of the prediction box with the coordinates of the true bounding box b i =(x i ,y i ,w i ,h i The coordinate error of ) is calculated, and the area weight is applied using step (a3):

[0158]

[0159] Special Limited Edition L L1 For weighted L1 loss: the area weight area_weight i Only applies to the width error term and height error term Error terms not applied to center point coordinates and

[0160] Wherein, the total bounding box regression loss L of the YOLOv object detection framework box It is composed of the following three loss terms weighted together:

[0161]

[0162] in:

[0163] (a4) Definition of Loss Item

[0164] L L1 : The weighted L1 loss (including area weight) as defined in claim 5;

[0165] L GIoU The generalized crossover ratio loss is calculated using the following formula:

[0166]

[0167] Where b is the predicted bounding box, b gt C is the bounding box between b and b'. gt The minimum convex hull region;

[0168] L mse The formula for calculating the mean square error loss of dimensions is:

[0169]

[0170] Where N is the number of targets in the batch. To predict the width and height of bounding box i, w i h i Let i be the width and height of the actual bounding box.

[0171] (b4) Applying constraints:

[0172] When the target frame area A i <100px 2 At that time, L mse The item's weight is automatically increased to 1.5;

[0173] When calculating LGIoU, outlier bounding boxes with an aspect ratio difference greater than 2:1 are ignored; (i.e.) (target bounding box);

[0174] (c4) Gradient balancing mechanism:

[0175] During the optimization process, the total loss L box The gradient is truncated, and its L2 norm is constrained to satisfy... To prevent L mse Lead the optimization direction.

[0176] In the above embodiments, dynamic weighting is used to solve the problem of missed detection caused by large differences in flame / smoke area, thereby achieving optimization of small targets; GIoU loss is used to improve localization, MSE loss is used to improve size accuracy and improve bounding box stability; gradient truncation is used to prevent large targets from dominating the optimization direction and improve training robustness.

[0177] Preferably, in step S3, when the encoder is prompted to process the mask input, the following operations are performed:

[0178] (a5) Mask resolution compression:

[0179] Input mask M in ∈R H×W A compressed mask Mcomp is generated by bilinear interpolation downsampling, with its height maintained at H and its width compressed to [value missing]. (i.e., the resolution becomes) );

[0180] (b5) Feature extraction and downsampling:

[0181] Compress mask M comp Input to the first convolutional layer: kernel size 2x2, stride 2, output channels 64, generating feature maps.

[0182] Input the feature map F1 into the second convolutional layer: kernel size 2x2, stride 2, output channels 128, to generate a feature map.

[0183] (c5) Feature Dimension Enhancement:

[0184] The feature map F2 is input into a 1x1 convolutional layer, increasing the channel dimension from 128 to 256 to generate the final cue embedding vector.

[0185] The prompt embedding vector E p resolution It has the same spatial resolution as the feature map output by the image encoder.

[0186] In the above embodiments, the computational load is reduced by resolution compression; interpolation distortion is avoided by matching the output size with the feature map of the image encoder; and key spatial information is preserved by 1×1 convolution.

[0187] This invention also provides a flame and smoke recognition system, comprising:

[0188] Image acquisition module: A multispectral camera array deployed at construction sites;

[0189] Edge computing module: A GPU-embedded device equipped with a model corresponding to any of the flame and smoke recognition methods described above;

[0190] Early warning response module: Triggers the activation of the fire extinguishing device when the confidence score exceeds a threshold;

[0191] The edge computing module is connected to the cloud server via a 5G private network to enable dynamic updates of model parameters.

[0192] The edge computing module includes dual processing channels:

[0193] The first channel processes the visible light video stream at 30fps and performs flame recognition.

[0194] The second channel processes the infrared video stream at 15fps to perform smoke detection.

[0195] The dual channels share the image encoder output of SAM through a feature-level fusion network.

[0196] This invention also provides a computer-readable storage medium storing a computer program, which, when executed by a computer, causes the computer to perform any of the flame and smoke recognition methods described above.

[0197] Those skilled in the art will understand that all or some of the steps, systems, and apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software can be distributed on a computer-readable storage medium, which may include computer storage media (or non-transitory media) and communication media (or transient media).

[0198] As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable program instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technologies, portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, it is known to those skilled in the art that communication media typically contain computer-readable program instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0199] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0200] The computer program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions. This electronic circuitry can execute the computer-readable program instructions to implement various aspects of the invention.

[0201] The computer program product described herein can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0202] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0203] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0204] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0205] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0206] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for identifying flames and smoke, characterized in that, Includes the following steps: Step 1: Use the YOLOv object detection framework to process the input image and generate bounding boxes for flame and smoke targets; Step 2: Use the SAM image encoder to process the input image and generate the visual feature embedding of the image; Step 3: Use SAM's cue encoder to encode the bounding box coordinates generated in Step 1 into cue embedding vectors; Step 4: Use the SAM mask decoder to fuse the visual feature embedding from Step 2 and the cue embedding vector from Step 3 to generate the target segmentation mask and its corresponding confidence score. Step 5: When the confidence score generated in step 4 exceeds the preset threshold, the presence of a flame or smoke target is confirmed. Step 6: Overlay the bounding box from Step 1 with the segmentation mask from Step 4, and output it to the fire alarm system; The multi-scale feature extraction of the image encoder in step 2 includes the following steps: (a1) Extracting multi-level feature maps: Four levels of feature maps are extracted from the backbone network of the image encoder, with spatial resolutions of the four levels as follows: Level 1: 1 / 4 of the input image resolution; Level 2: 1 / 8 of the input image resolution; Level 3: 1 / 16 of the input image resolution; Level 4: 1 / 32 of the input image resolution; (b1) Level 1 coordinate alignment: Use the IntermediateLayerGetter module to perform coordinate alignment on the feature map of level 1 so that it shares the same spatial coordinate reference with the feature maps of level 2, level 3 and level 4; (c1) Channel dimension unification: A 4x4 convolutional kernel with a stride of 1 is used to process the feature maps of the four levels respectively, and their channel dimensions are uniformly mapped to 256 dimensions. The mask decoder processing flow in step 4 includes the following steps in sequence: (i) Image embedding preprocessing: The image embedding is spatially deformed to generate a deformable feature tensor. The deformable feature tensor is processed using the first transposed convolutional layer: convolutional kernel 4x4, stride 2, output channels reduced to 128 dimensions; The output of the first transposed convolutional layer is processed using a second transposed convolutional layer: convolutional kernel 4x4, stride 2, output channels reduced to 32 dimensions; (ii) Hints on embedding separation and confidence score generation: Separate the mask token and confidence token from the cue embedding vector; The confidence token is sequentially input into three fully connected layers for prediction. The number of neurons in the fully connected layers are 128, 64 and 1, respectively, and finally a confidence score for the existence of the target is generated. (iii) Target segmentation mask generation: Perform matrix multiplication on the 32-dimensional feature tensor output in step (i) and the mask token separated in step (ii); Apply the Sigmoid activation function to the matrix multiplication result to generate a segmentation mask for the target; (iv) Output determination: When the confidence score generated in step (ii) is greater than the preset threshold of 0.85, the segmentation mask generated in step (iii) and the bounding box corresponding to step 1 are output to the early warning system. The deformable attention mechanism used by the image encoder in step 2 includes the following steps: (a2) Query initialization: From the 256-dimensional feature map output by the image encoder, the spatial location index is extracted as follows: q query vector Z q ; Get and query vectors Z q Corresponding two-dimensional reference point coordinates p q =(x q ,y q ) These coordinates are obtained by transforming the center point coordinates of the bounding box predicted by YOLOv in step 1; (b2) Dynamic offset calculation: Through a fully connected layer, for each query vector Z q Generate M=8 attention heads, each corresponding to K=4 learnable offsets. Δp mqk Where m∈[1,8] denotes the attention head number, and k∈[1,4] denotes the sampling point number; Each offset is constrained to satisfy ||Δp mqk ||≤3; (c2) Feature-weighted aggregation: Calculate the query position using the following formula. q Output characteristics: ； in, and These are the output projection matrix and feature transformation matrix of the m-th attention head, respectively; Characterized by the position on feature map x The eigenvalues ​​at a given point are obtained through bilinear interpolation. A mqk The attention weight for the m-th head and the k-th sampling point is calculated using the following formula: ； in, is the query transformation matrix of the m-th attention head, and d is the feature dimension.

2. The flame and smoke identification method according to claim 1, characterized in that, The YOLOv object detection framework is trained using a dynamic area-weighted loss function, which is constructed using the following steps: (a3) For each target box in the training batch i Calculate its dynamic area weight: ； in: A i = w i × h i Represents the target box i The pixel area, where, w i For width, h i For height; This represents the average pixel area of ​​all N bounding boxes in the current training batch; The clamp(value,min,max) function means that when value exceeds the range [min,max], it is forcibly truncated to the boundary value of the interval. (b3) Calculate the weighted L1 loss: Calculate the coordinates of the prediction box coordinates of the ground truth bounding box The coordinate error is calculated, and the area weight is applied using step (a3): ； The area weight area_weight i Only applies to the width error term and height error term It does not affect the center point coordinate error term. and .

3. The flame and smoke identification method according to claim 2, characterized in that, The total bounding box regression loss L of the YOLOv object detection framework box It is composed of the following three loss terms weighted together: ； in: (a4) Definition of loss term : This is the weighted L1 loss; The generalized crossover ratio loss is calculated using the following formula: ； in b For the prediction box, For the true frame, C To surround b and The minimum convex hull region; The formula for calculating the mean square error loss of dimensions is: ； in, N For the target number of batches, , For prediction boxes i Width and height, , For the true frame i Width and height; (b4) Applying constraints: When the target box area A i <100px 2 hour, L mse The item's weight is automatically increased to 1.5; When calculating LGIoU, outlier bounding boxes with aspect ratio differences greater than 2:1 are ignored; (c4) Gradient balancing mechanism: During the optimization process, the total loss L box The gradient is truncated, and its L2 norm is constrained to satisfy... To prevent L mse Lead the optimization direction.

4. The flame and smoke identification method according to claim 1, characterized in that, In step S3, when the encoder is prompted to process the mask input, the following operations are performed: (a5) Mask resolution compression: Input mask The compressed mask M is generated by bilinear interpolation downsampling. comp Its height remains H, while its width is compressed to... ; (b5) Feature extraction and downsampling: Compress mask M comp Input to the first convolutional layer: kernel size 2x2, stride 2, output channels 64, generating feature maps. ; Input the feature map F1 into the second convolutional layer: kernel size 2x2, stride 2, output channels 128, to generate a feature map. ; (c5) Feature Dimension Enhancement: The feature map F2 is input into a 1x1 convolutional layer, increasing the channel dimension from 128 to 256 to generate the final cue embedding vector. ; The prompt embedding vector F p resolution ( The feature map spatial resolution is consistent with that of the image encoder output.

5. A flame and smoke recognition system, characterized in that, Include: Image acquisition module: A multispectral camera array deployed at construction sites; Edge computing module: A GPU embedded device equipped with a model corresponding to any of the flame and smoke recognition methods described in claims 1-4; Early warning response module: Triggers the activation of the fire extinguishing device when the confidence score exceeds a threshold; The edge computing module is connected to the cloud server via a 5G private network to enable dynamic updates of model parameters.

6. The flame and smoke recognition system according to claim 5, characterized in that, The edge computing module includes dual processing channels: The first channel processes the visible light video stream at 30fps and performs flame recognition. The second channel processes the infrared video stream at 15fps to perform smoke detection. The dual channels share the image encoder output of SAM through a feature-level fusion network.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a computer, causes the computer to perform the flame and smoke identification method as described in any one of claims 1 to 4.

Images