Intelligent fire detection and positioning method based on visual recognition

Abstract

The application relates to the technical field of fire monitoring, in particular to an intelligent fire detection and positioning method based on visual recognition, which comprises the following steps: improving a YOLOv8 model, combining an image segmentation and depth estimation model, training a multi-stage flame depth recognition model through a multi-element flame data set; identifying a flame monitoring image through the multi-stage flame depth recognition model, comprising the following steps: determining a warning area in the flame monitoring image through a target detection algorithm, identifying the warning area through a Mobile SAM model to obtain a distribution area and a segmentation mask; identifying the distribution area and the segmentation mask through a DepthAnything model to obtain flame depth data; calibrating internal and external parameters of a monocular camera, converting a flame pixel coordinate system to a world coordinate system in combination with the flame depth data, and obtaining a three-dimensional space position of the flame; and identifying a fire warning coefficient according to the three-dimensional space position of the flame to accurately perform fire warning.

Description

Technical Field

[0001] This invention relates to the field of fire monitoring technology, specifically to an intelligent fire detection and location method based on visual recognition. Background Technology

[0002] Current visual sensing-based fire detection systems commonly employ deep learning frameworks for flame recognition. Mainstream methods include YOLO-based bounding box detection algorithms and pixel-level segmentation techniques based on fully convolutional networks. These methods achieve flame recognition by analyzing spectral features in RGB images, demonstrating good real-time detection capabilities in typical scenarios.

[0003] However, existing systems lack the ability to perceive the dynamic characteristics of flames in multiple dimensions, and cannot determine the type of burning material and the stage of fire development by observing the trend of radiation intensity changes, thus restricting the accurate formulation of fire extinguishing strategies. For example, monocular vision systems have difficulty accurately resolving the three-dimensional spatial distribution of flames, resulting in a lack of depth information; they are prone to misjudgment when affected by sudden changes in ambient light or smoke obstruction; and the complex network structure makes the deployment of models on embedded platforms face computational resource bottlenecks.

[0004] Especially when precise fire extinguishing location is required, existing methods have significant shortcomings in terms of flame contour segmentation accuracy and 3D coordinate calculation. Rectangular box detection cannot provide a fine mask of the burning area, while monocular ranging methods based on 2D images suffer from accumulated spatial positioning errors due to the lack of depth perception. These problems severely restrict the environmental adaptability and positioning reliability of intelligent fire detection systems in complex scenarios.

[0005] To address this, a visual recognition-based intelligent fire detection and location method is proposed. Summary of the Invention

[0006] The purpose of this invention is to provide a visual recognition-based intelligent fire detection and localization method. It improves the YOLOv8 model by combining image segmentation and depth estimation models to obtain a multi-stage flame depth recognition model. This model is trained using a multivariate flame dataset. The method then identifies flame monitoring images using this model, including determining warning regions in the images through target detection algorithms, identifying these regions using the Mobile SAM model to obtain distribution areas and segmentation masks, and using the DepthAnything model to identify the distribution areas and segmentation masks to obtain flame depth data. Finally, it calibrates the intrinsic and extrinsic parameters of a monocular camera and, combined with the flame depth data, transforms the flame pixel coordinate system to the world coordinate system to obtain the three-dimensional spatial position of the flame. Based on the three-dimensional spatial position of the flame, this invention identifies a fire warning coefficient, enabling accurate fire warning.

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] A visual recognition-based intelligent fire detection and location method includes:

[0009] Collect flame images from diverse scenes, identify and label the flame images to obtain a diverse flame dataset;

[0010] A multi-stage flame depth recognition model was obtained by training the improved YOLOv8 model on a multi-dimensional flame dataset, including a fire detection module and a fire depth recognition module; the improvements to the YOLOv8 model include the GSConv module and the CFB module.

[0011] Real-time monitoring using camera devices yields flame monitoring images; fire detection modules then identify these flame monitoring images to obtain fire fusion characteristics.

[0012] A fire depth recognition module is constructed to identify fire fusion features; the fire warning area of ​​the fire monitoring image is determined by the target detection algorithm, and the fire warning area is identified by the Mobile SAM model to obtain the fire distribution area and the fire segmentation mask; the fire distribution area and the fire segmentation mask are identified by the DepthAnything model to obtain the fire depth data.

[0013] A flame depth map is obtained based on the flame depth data. Based on the depth values ​​of the pixels in the flame depth map and the camera device parameters, the coordinates of each pixel in the flame distribution area are converted into world coordinates to obtain the three-dimensional spatial position.

[0014] The system acquires time-series data of the three-dimensional spatial location of the flame, identifies the fire early warning coefficient based on the ignition point material, real-time environmental data, and time-series data, and then issues an early warning.

[0015] The multivariate flame dataset includes flame images, image annotation information, and image depth information; its acquisition process is as follows:

[0016] To obtain flame images, images of fire scenes captured in diverse environments are obtained.

[0017] The flame region in the flame image is labeled, including boundary labeling and shape labeling, to obtain image labeling information;

[0018] Flame region features and non-flame region features in flame images are identified to obtain image depth information.

[0019] Model improvements based on the GSConv module include replacing the convolutional layers in the YOLOv8 model with the GSConv module; the GSConv module divides the convolutional operation into multiple groups for processing, which is used to capture local spatial features.

[0020] Its formula is expressed as:

[0021]

[0022] Among them, X out This represents the fused feature map obtained by concatenating the outputs of multiple groups; G represents the number of groups; X g Let K represent the g-th input feature map. g X represents g The corresponding convolution kernel; * indicates the convolution operation; concat() indicates the concatenation function.

[0023] Model improvements based on the CFB module include: replacing the C2f layer in the YOLOv8 model with the CFB module;

[0024] The CFB module includes a main branch and a flame feature branch;

[0025] The main branch is a C2f structure, which performs feature recognition on the fused feature map to obtain the first flame feature data;

[0026] The flame feature branch includes a multi-scale feature extraction structure that uses convolutional kernels of three scales: 3×3, 5×5, and 7×7 to capture fire features of different sizes simultaneously. Then, a dynamic weighting mechanism is used to adaptively adjust the features at each scale to obtain multi-scale fire feature data. It also includes a color perception structure that extracts color information based on 1×1 convolution and obtains fire color feature data through Sigmoid activation.

[0027] The CFB module performs feature fusion based on the first flame feature data, fire multi-scale feature data, and fire color feature data to obtain fire fused features.

[0028] A flame depth recognition module is constructed, including a flame region recognition layer, a flame region segmentation layer, and a flame depth prediction layer;

[0029] The flame area identification layer identifies fire fusion features using a target detection algorithm to obtain the flame warning area;

[0030] The flame region segmentation layer identifies the flame warning region using a lightweight Mobile SAM model and segments the flame warning region using a transfer learning method to obtain the flame distribution region and the flame segmentation mask.

[0031] The flame depth prediction layer uses the DepthAnything depth estimation model to identify the flame distribution area and the flame segmentation mask, thereby obtaining the depth value corresponding to each pixel in the flame distribution area and thus obtaining the flame depth data.

[0032] The process of obtaining a three-dimensional spatial location from a flame depth map includes:

[0033] Transform the pixel coordinates of the flame distribution area to camera coordinates based on the flame depth map:

[0034]

[0035] Among them, (X) c ,Y c Z c () represents the coordinates in the camera coordinate system; (u,v) represents the pixel coordinates in the flame depth map; D(u,v) is the depth value of the pixel; c x and c y Principal point coordinates;

[0036] Then, based on the camera's extrinsic parameters, the camera coordinates are transformed to world coordinates to obtain the three-dimensional spatial position of the flame:

[0037]

[0038] Among them, (X) w, Y w Z w ) represents the coordinates in the world coordinate system; R and t are the camera extrinsic parameters, where R represents the rotation matrix and t represents the translation vector.

[0039] The process for identifying the fire early warning coefficient is as follows:

[0040] The ignition point is determined based on three-dimensional spatial coordinates, and the material at the ignition point and real-time environmental data are identified.

[0041] Feature identification is performed on the time series data of the three-dimensional spatial coordinates of the flame to obtain the spatial density distribution, surface curvature change, centroid displacement trajectory, vertical height distribution, and point cloud topological connectivity; then, combined with the ignition point material and real-time environmental data, a fire early warning coefficient is obtained for early warning.

[0042] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0043] 1. This invention improves the YOLOv8 model by using the GSConv and CFB modules to obtain a fire detection module. The GSConv module is an improved convolution method that, by dividing the convolution operation into multiple groups, can better capture local spatial features, enhancing the model's sensitivity to local details while preserving global information. It also improves computational efficiency; compared to traditional convolution, GSConv is more computationally efficient, reducing computational load, shortening training time, and maintaining model performance while increasing adaptability. Through this replacement, YOLOv8 can better capture local flame features when processing fire images, while reducing computational overhead and improving detection speed.

[0044] 2. This invention uses a CFB module to identify flame features. The CFB module employs a dual-branch architecture design. The main branch maintains the original C2f structure to ensure basic feature extraction capabilities, while the fire feature branch is specifically optimized for fire features. A dynamic fusion mechanism enables adaptive feature combination. For flame features of different scales, a multi-scale feature extraction branch is designed, capable of simultaneously capturing fire features of varying sizes. A dynamic weighting mechanism adaptively adjusts features at each scale. Combining color perception and spatial attention to capture shape features, accurate flame feature identification is achieved.

[0045] 3. This invention constructs a fire depth recognition module for fire fusion feature recognition; it determines the fire warning area of ​​the fire monitoring image through a target detection algorithm, identifies the fire warning area through the Mobile SAM model, and obtains the fire distribution area and fire segmentation mask; it then identifies the fire distribution area and fire segmentation mask through the DepthAnything model, obtaining fire depth data. This invention effectively combines target detection bounding boxes (YOLOv8), lightweight SAM (MobileSAM), accurate mask depth estimation (DepthAnything), and depth map to 3D point cloud conversion technologies. The advantage of this combined solution is that it can simultaneously provide high-precision fire segmentation, depth estimation, and 3D localization in different environments and on different devices.

[0046] 4. This invention determines the ignition point based on three-dimensional spatial coordinates and identifies the ignition point material and real-time environmental data; it performs feature recognition on the time series data of the flame's three-dimensional spatial coordinates to obtain spatial density distribution, surface curvature change, centroid displacement trajectory, vertical height distribution, and point cloud topological connectivity; and then combines the ignition point material and real-time environmental data to identify the fire early warning coefficient, thereby accurately identifying and warning of changes in the fire. Attached Figure Description

[0047] Figure 1This is a flowchart illustrating an intelligent fire detection and location method based on visual recognition proposed in this invention.

[0048] Figure 2 A flowchart illustrating the steps involved in intelligent fire detection and location.

[0049] Figure 3 This is a diagram of the YOLOv8 network structure.

[0050] Figure 4 Here is a structural diagram of the GSConv module;

[0051] Figure 5 Here is a diagram of the CFB module structure;

[0052] Figure 6 This is a structural diagram of the fire characteristic branch module;

[0053] Figure 7 This is a structural diagram of a fire detection module;

[0054] Figure 8 A rectangular box indicating the fire warning area;

[0055] Figure 9 This is a segmentation map of the flame distribution area within the target box;

[0056] Figure 10 This is a location map showing the transformation of flames into a three-dimensional point cloud within a fire environment. Detailed Implementation

[0057] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0058] Example 1

[0059] This invention proposes an intelligent fire detection and location method based on visual recognition, the process of which is as follows: Figure 1 As shown, it includes:

[0060] Collect flame images from diverse scenes, identify and label the flame images to obtain a diverse flame dataset;

[0061] The multivariate flame dataset includes flame images, image annotation information, and image depth information; its acquisition process is as follows:

[0062] Capture fire scenes in diverse environments to obtain flame images;

[0063] The flame region in the flame image is labeled, including boundary labeling and shape labeling, to obtain image labeling information;

[0064] Flame region features and non-flame region features in flame images are identified to obtain image depth information.

[0065] Furthermore, the steps involved in intelligent fire detection and location are as follows: Figure 2 As shown, the steps for acquiring the multivariate flame dataset in this embodiment are as follows:

[0066] Flame images were captured using an ORBBEC Astra S depth camera in various environments (such as indoors, outdoors, nighttime, and daytime). The image data needed to cover a wide range of flame colors, shapes, sizes, and complex backgrounds to ensure data diversity and representativeness.

[0067] Flame image annotation: Mark the precise boundaries of the flame region for each image to ensure high-quality segmentation results.

[0068] Background information acquisition: The ORBBEC Astra S depth camera was also used to acquire depth information of the fire scene, providing raw data for subsequent depth estimation.

[0069] A multi-stage flame depth recognition model was obtained by training the improved YOLOv8 model on a multivariate flame dataset, including a fire detection module and a fire depth recognition module. The improvements to the YOLOv8 model include the GSConv module and the CFB module. The YOLOv8 network structure is as follows: Figure 3 As shown.

[0070] Model improvements based on the GSConv module include replacing the convolutional layers in the YOLOv8 model with the GSConv module; the GSConv module divides the convolutional operation into multiple groups for processing, which is used to capture local spatial features.

[0071] Its formula is expressed as:

[0072]

[0073] Among them, X out This represents the fused feature map obtained by concatenating the outputs of multiple groups; G represents the number of groups; X g Let K represent the g-th input feature map. g X represents g The corresponding convolution kernel; * indicates the convolution operation; concat() indicates the concatenation function.

[0074] To further improve the model's performance in fire detection, this invention replaces the original convolutional layer (Conv) with a GSConv (Group Spatial Convolution) module, such as... Figure 4 As shown.

[0075] The recognition process of the GSConv module includes feature map input, grouped convolution, and feature concatenation:

[0076] Feature map input: The input feature map is X∈R C×H×W Where C is the number of channels, H is the height of the feature map, W is the width of the feature map, and the convolution kernel is... Where G is the number of groups, C / G represents the number of channels in each group, and K... h and K w These are the height and width of the convolution kernel, representing the spatial dimensions of the convolution kernel.

[0077] Grouped convolution: converts input features Figure X Divide the input feature maps into G groups, each with a size of C / G. Perform a convolution operation on each group. Each convolution operation is performed on a corresponding subset of the input feature maps. For the g-th group (g∈{1,2,…,G}), the convolution operation can be represented as: Among them, X g ∈R (C / G)×H×W This is the g-th input feature map. This is the corresponding convolution kernel, and * indicates the convolution operation.

[0078] Concatenating convolution outputs: combining the outputs of each group splicing together to form a new feature Figure X out :

[0079]

[0080] The number of channels in the output feature map will be restored to C, i.e., X. out ∈R C×H×W concat() represents a concatenation function.

[0081] The GSConv module is an improved convolutional method with the following advantages: enhanced spatial locality – GSConv processes convolutional operations in multiple groups, better capturing local spatial features. It improves the model's sensitivity to local details while preserving global information; improved computational efficiency – compared to traditional convolution, GSConv is more computationally efficient, reducing computational cost and training time while maintaining model performance; and greater adaptability – the GSConv module can flexibly handle images of different scales and complexities, making it particularly suitable for extracting key features in special scenarios such as fires. Through this replacement, YOLOv8 can better capture local flame features when processing fire images, while reducing computational overhead and improving detection speed.

[0082] Model improvements based on the CFB module include: replacing the C2f layer in the YOLOv8 model with the CFB module;

[0083] The CFB module includes a main branch and a flame feature branch;

[0084] The main branch is a C2f structure, which performs feature recognition on the fused feature map to obtain the first flame feature data;

[0085] The flame feature branch includes a multi-scale feature extraction structure that uses convolutional kernels of three scales: 3×3, 5×5, and 7×7 to capture fire features of different sizes simultaneously. Then, a dynamic weighting mechanism is used to adaptively adjust the features at each scale to obtain multi-scale fire feature data. It also includes a color perception structure that extracts color information based on 1×1 convolution and obtains fire color feature data through Sigmoid activation.

[0086] The CFB module performs feature fusion based on the first flame feature data, fire multi-scale feature data, and fire color feature data to obtain fire fused features.

[0087] Flame features possess certain unique characteristics. First, flames are irregular in shape, highly variable, and lack fixed geometric features. Second, flames exhibit diverse scales, ranging widely from small to large fire sizes. Color characteristics are also quite pronounced, with flames typically displaying reddish-yellow hues. Furthermore, flame texture features are unique, exhibiting distinctive fluid textures. Additionally, existing target detection models have limitations; conventional feature extraction lacks specificity and attention to fire-specific features, and single-scale feature extraction yields limited results.

[0088] Based on the above analysis, this invention designs a CFB (C2fFireBranch) module, such as... Figure 5As shown, to address these issues, the CFB module employs a dual-branch architecture. The main branch maintains the original C2f structure, ensuring basic feature extraction capabilities. The Fire Feature Branch (FFB) is specifically optimized for fire features, and a dynamic fusion mechanism enables adaptive feature combination. The CFB module designs multi-scale feature extraction branches for flame features at different scales, using convolutional kernels of 3×3, 5×5, and 7×7 scales to simultaneously capture fire features of different sizes. A dynamic weighting mechanism adaptively adjusts features at each scale. The CFB module also incorporates a color perception mechanism with a dedicated color-sensitive branch that extracts color information through 1×1 convolutions, while sigmoid activation highlights the importance of color features. The CFB module uses spatial attention to capture shape features and channel attention to enhance feature representation, employing a cascaded combination of attention mechanisms.

[0089] Input features X∈R C×H×W First, the main branch is calculated; the first convolution: F1 = Conv(X); where F1 represents the first convolution data; Conv() represents the convolutional layer;

[0090] Perform feature segmentation: F 1a ,F 1b =Split(F1); where F 1a Indicates the first segment of data; F 1b This represents the second segment of data; Split() represents the feature segmentation operation;

[0091] Bottleneck Operation: B i =Bottleneck(B i-1 ), i∈[1,n], where B0=F 1b B i This represents the bottleneck data at the i-th position; Bottleneck() represents the Bottleneck operation.

[0092] Final result: F main =Conv(concat(F 1a ,F 1b ,B1,...,B n )); where F main This represents the first flame characteristic data.

[0093] For the Fire Feature Branch (FFB), such as Figure 6 As shown: First, the input feature X∈R C ×H×W Multi-scale feature extraction is performed to obtain the basic convolutional feature data F. b ;

[0094] Multi-scale recognition is performed on basic convolutional feature data. Multi-scale features: M i =Convi(F b ), i∈{3,5,7}, where i corresponds to the size of the convolution kernel; M i Represents multi-scale feature data of size i; Convi() represents multi-scale convolution.

[0095] Color feature extraction: F c =Sigmoid(Conv(Conv(X))); F c This represents fire color characteristic data; Sigmoid() represents the Sigmoid activation function.

[0096] Feature fusion is then performed:

[0097] F combined =concat(F b *W0,M3*W1,M5*W2,M7*W3);

[0098] F color =F combined *F c ;

[0099] F fire =Conv(concat(F combined ,F color ]));

[0100] F final =Conv(concat(F main *W f0 ,F fire *W f1 ));

[0101] Among them, F combined This represents multi-scale feature data of fire; W0, W1, W2, and W3 are the multi-scale weights, respectively; F color Indicates color blending data; F fire Indicates the first fire fusion data; F final Indicates fire fusion characteristics; F main Represents the first flame characteristic data; W f0 W f1 These are the feature weights.

[0102] The CFB module is designed with full consideration of the special needs of fire detection. Through innovative technical solutions, it achieves highly targeted and high-performance feature extraction capabilities, adapting to fire scenarios in various environments.

[0103] After improvement, the structure of the fire detection module is as follows: Figure 7As shown; in this embodiment, X∈R C×H×W This refers to input features, which represent different inputs in different scenarios. The specific input-output relationship can be determined based on... Figure 6 The structure is determined.

[0104] To verify the recognition effect of the fire detection module, verification was conducted; verification model one, verification model two, verification model three, and verification model four were constructed.

[0105] Among them, verification model one is the fire detection module of this application, which improves the YOLOv8 model through the GSConv module and the CFB module; verification model two improves the YOLOv8 model only through the GSConv module; verification model three improves the YOLOv8 model only through the CFB module; verification model four is the YOLOv8 model; the similarity between the identified feature data and the actual feature data is used as the accuracy.

[0106] The data obtained after multiple tests are shown in Table 1.

[0107] Table 1 Model Validation Data

[0108] Model GSConv module CFB module accuracy Recognition speed Validation Model 1 include include 0.924 53ms Verification Model 2 include Excluding 0.887 87ms Validation Model 3 Excluding include 0.905 89ms Validation Model 4 Excluding Excluding 0.856 134ms

[0109] According to the data in Table 1, after the improvements to the GSConv and CFB modules, the accuracy and speed of the fire detection module have been significantly improved.

[0110] Real-time monitoring using camera devices yields flame monitoring images; fire detection modules then identify these flame monitoring images to obtain fire fusion characteristics.

[0111] A fire depth recognition module is constructed to identify fire fusion features; the fire warning area of ​​the fire monitoring image is determined by the target detection algorithm, and the fire warning area is identified by the Mobile SAM model to obtain the fire distribution area and the fire segmentation mask; the fire distribution area and the fire segmentation mask are identified by the DepthAnything model to obtain the fire depth data.

[0112] A fire depth identification module is constructed, including a flame area identification layer, a flame area segmentation layer, and a flame depth prediction layer;

[0113] The flame area identification layer identifies fire fusion features using a target detection algorithm to obtain the flame warning area;

[0114] The target detection algorithm model can accurately identify the location of flames in an image and generate a rectangular box prompt for each flame target, such as... Figure 8As shown. These detection boxes contain the approximate location and extent of the flame in the image, serving as input for subsequent segmentation. Assume the flame detection box output by the target detection algorithm is B = (x1, y1, x2, y2), where (x1, y1) are the coordinates of the top-left corner of the box, and (x2, y2) are the coordinates of the bottom-right corner of the box, representing the boundary of the flame warning area.

[0115] The flame region segmentation layer identifies the flame warning region using a lightweight Mobile SAM model and segments the flame warning region using a transfer learning method to obtain the flame distribution region and the flame segmentation mask.

[0116] MobileSAM is a version optimized for low computational resources. Through techniques such as transfer learning, it can perform accurate segmentation while maintaining high efficiency. During input, the flame-shaped bounding box B serves as a box cue, providing spatial constraints on the target. MobileSAM then focuses on the region within this box based on the cue, performing further precise segmentation.

[0117] After receiving a bounding box prompt, MobileSAM uses feature extraction and image understanding capabilities to identify the flame region within the box and generate a corresponding flame segmentation mask. This segmentation mask precisely marks the pixel position of the flame, enabling the segmentation of the flame distribution area from the image, such as... Figure 9 As shown, during the segmentation process, MobileSAM not only relies on bounding box cues but also further refines the flame details in the image, such as the flame's edge, shape, and color features, through its embedded deep feature learning mechanism, thereby ensuring the accuracy of the segmentation results.

[0118] The segmentation result is output as a binary mask image containing the precise pixel locations of the flame distribution. This mask can be used for subsequent tasks such as flame depth estimation, location localization, and point cloud transformation, providing accurate data support for fire detection systems.

[0119] Furthermore, an improved segmentation network can be used to replace SAM; MobileSAM, as a lightweight image segmentation model, plays an important role in this invention. However, other lightweight segmentation networks (such as DeepLabV3+, Fast-SCNN, etc.) can also be used as alternatives. These networks also have high segmentation performance and different design and optimization directions, which may provide better performance in certain hardware or application scenarios.

[0120] The flame depth prediction layer uses the DepthAnything depth estimation model to identify the flame distribution area and the flame segmentation mask, thereby obtaining the depth value corresponding to each pixel in the flame distribution area and thus obtaining the flame depth data.

[0121] This invention proposes a method for depth estimation and 3D information generation that combines flame segmentation masks with a depth estimation model. By using accurate flame segmentation masks, this method can effectively extract useful depth information from fire scenes and generate high-quality 3D data support.

[0122] In the target detection stage, an improved YOLOv8 model is used to detect targets in the fire image, obtaining bounding boxes containing flame warning areas. A lightweight SAM model (MobileSAM) is then used for accurate flame segmentation, resulting in a high-quality binary flame segmentation mask. This mask clearly identifies the pixel locations of the flame distribution areas, ensuring high accuracy for subsequent depth estimation.

[0123] In the process of creating depth datasets, interpolation methods can be used to fill in regions with holes in the depth map. This paper uses the nearest neighbor interpolation method to estimate the depth values ​​of the hole regions. This step ensures that the depth map is not affected by the hole data during training, enhancing the continuity and stability of depth estimation.

[0124] The DepthAnything model learns the depth of flame regions in a fire scene, estimating the depth value for each pixel and generating depth maps for different areas of the flame. The depth estimation model, based on the input image and mask information, outputs a depth D. f ∈R H×W Each pixel value D f (x, y) represents the depth value corresponding to each pixel. The mask M for the flame distribution area is used. f (Binarization) is used to filter valid depth values. Specifically, only depth data within the masked area is preserved. This step ensures that depth estimation focuses only on the flame distribution area, thereby obtaining the flame depth value.

[0125]

[0126] in, For the depth map after applying the mask, the mask M f Ensure that only the flame depth value is retained.

[0127] Furthermore, this invention uses the DepthAnything depth estimation model. While this model performs well in this invention, other depth estimation techniques, such as MiDaS (Monocular Depth Estimation) or MonoDepth, could be considered. These models can also perform monocular depth estimation and may be more suitable for certain applications.

[0128] The core idea of ​​this invention is to improve the efficiency and accuracy of fire monitoring and location systems by combining technical modules such as target detection boxes, lightweight SAM models (MobileSAM), accurate flame segmentation masks, depth estimation, and 3D point cloud generation.

[0129] A flame depth map is obtained based on the flame depth data. Based on the depth values ​​of the pixels in the flame depth map and the camera device parameters, the coordinates of each pixel in the flame distribution area are converted into world coordinates to obtain the three-dimensional spatial position.

[0130] Depth map generated by depth estimation model By combining the camera's intrinsic and extrinsic parameters, the depth information of the flame can be converted into a three-dimensional spatial position; this three-dimensional spatial position is represented by three-dimensional spatial coordinates. This process involves the following steps:

[0131] Depth to 3D coordinate transformation: Based on the depth value of each pixel in the depth map, each pixel is converted into coordinates in 3D space using the camera's intrinsic and extrinsic parameters.

[0132] The process of obtaining a three-dimensional spatial location from a flame depth map includes:

[0133] Transform the pixel coordinates of the flame distribution area to camera coordinates based on the flame depth map:

[0134]

[0135] Among them, (X) c ,Y c Z c () represents the coordinates in the camera coordinate system; (u,v) represents the pixel coordinates in the flame depth map; D(u,v) is the depth value of the pixel; c x and c y The principal point coordinates are the intersection of the camera's optical axis and the image plane, usually close to the image center.

[0136] Then, based on the camera's extrinsic parameters, the camera coordinates are transformed to world coordinates to obtain the three-dimensional spatial position of the flame:

[0137]

[0138] Among them, (X) w, Y w Z w ) represents the coordinates in the world coordinate system; R and t are the camera extrinsic parameters, where R represents the rotation matrix and t represents the translation vector.

[0139] Point cloud generation: Through the above transformation, the depth values ​​of all flame regions are converted into three-dimensional spatial coordinates, representing the position of the flame in three-dimensional space, such as... Figure 10 As shown.

[0140] By combining precise masking with depth estimation, this invention effectively provides accurate three-dimensional spatial positioning of flames. Addressing holes in the depth map ensures the continuity of depth data, thus avoiding the impact of inaccurate depth estimation on subsequent processing. Simultaneously, through precise flame depth estimation, this invention can provide accurate three-dimensional flame location data for subsequent automatic fire extinguishing devices, improving the efficiency of fire emergency response.

[0141] Variations of the 3D spatial positioning method: This invention employs a method for converting depth maps to 3D point clouds. This method is highly suitable for data acquired by depth cameras, but 3D spatial data can be acquired in different ways if other types of sensors (such as LiDAR, lidar) are used. In this case, the generation of flame depth information and the method of 3D positioning may differ. An alternative could be to generate a 3D point cloud based on LiDAR data and use a similar depth estimation algorithm for depth supplementation.

[0142] The system acquires time-series data of the three-dimensional spatial coordinates of the flame, identifies fire early warning coefficients based on the ignition point material, real-time environmental data, and time-series data, and then issues early warnings.

[0143] The process for identifying the fire early warning coefficient is as follows:

[0144] The ignition point is determined based on three-dimensional spatial coordinates, and the material at the ignition point and real-time environmental data are identified.

[0145] Feature identification is performed on the time series data of the three-dimensional spatial coordinates of the flame to obtain the spatial density distribution, surface curvature change, centroid displacement trajectory, vertical height distribution, and point cloud topological connectivity; then, combined with the ignition point material and real-time environmental data, a fire early warning coefficient is obtained for early warning.

[0146] The flame volume growth rate is obtained by identifying the spatial density distribution, which is used to quantify the fire spread rate; the combustion intensity index is obtained by identifying the surface curvature change, which is used to reflect the turbulence intensity on the flame surface; the spread direction angle is obtained by identifying the centroid displacement trajectory, which is used to predict the fire propagation path; the roof jet risk coefficient is obtained by identifying the vertical height distribution, which is used to assess the probability of vertical dangerous spread; and the firebreak breach warning is obtained by identifying the point cloud topological connectivity, which is used to determine whether the flame has crossed the physical barrier.

[0147] By recognizing the three-dimensional spatial coordinates of flames in multiple dimensions, it is possible to accurately determine how a fire changes over time and provide early warnings.

[0148] This invention improves the YOLOv8 model to obtain a multi-stage flame depth recognition model. The multi-stage flame depth recognition model is trained using a multivariate flame dataset. It then uses this model to recognize flame monitoring images, including determining warning regions in the images using a target detection algorithm, identifying these regions using a Mobile SAM model to obtain distribution areas and segmentation masks, and using a DepthAnything model to identify the distribution areas and segmentation masks to obtain flame depth data. Based on this data, a flame depth map is generated, and each pixel is converted into coordinates in three-dimensional space to obtain the three-dimensional spatial coordinates of the flame. This invention uses the three-dimensional spatial coordinates of the flame to identify fire warning coefficients, enabling accurate fire warnings.

[0149] Example 2

[0150] This invention proposes an intelligent fire detection and location method based on visual recognition, such as... Figure 1 As shown, it includes:

[0151] Collect flame images from diverse scenes, identify and label the flame images to obtain a diverse flame dataset;

[0152] The multivariate flame dataset includes flame images, image annotation information, and image depth information; its acquisition process is as follows:

[0153] Capture fire scenes in diverse environments to obtain flame images;

[0154] The flame region in the flame image is labeled, including boundary labeling and shape labeling, to obtain image labeling information;

[0155] Flame region features and non-flame region features in flame images are identified to obtain image depth information.

[0156] A multi-stage flame depth recognition model was obtained by training the improved YOLOv8 model on a multi-dimensional flame dataset, including a fire detection module and a fire depth recognition module; the improvements to the YOLOv8 model include the GSConv module and the CFB module.

[0157] Model improvements based on the GSConv module include replacing the convolutional layers in the YOLOv8 model with the GSConv module; the GSConv module divides the convolutional operation into multiple groups for processing, which is used to capture local spatial features.

[0158] Its formula is expressed as:

[0159]

[0160] Among them, X outThis represents the fused feature map obtained by concatenating the outputs of multiple groups; G represents the number of groups; X g Let K represent the g-th input feature map. g X represents g The corresponding convolution kernel; * indicates the convolution operation; concat() indicates the concatenation function.

[0161] Model improvements based on the CFB module include: replacing the C2f layer in the YOLOv8 model with the CFB module;

[0162] The CFB module includes a main branch and a flame feature branch;

[0163] The main branch is a C2f structure, which performs feature recognition on the fused feature map to obtain the first flame feature data;

[0164] The flame feature branch includes a multi-scale feature extraction structure that uses convolutional kernels of three scales: 3×3, 5×5, and 7×7 to capture fire features of different sizes simultaneously. Then, a dynamic weighting mechanism is used to adaptively adjust the features at each scale to obtain multi-scale fire feature data. It also includes a color perception structure that extracts color information based on 1×1 convolution and obtains fire color feature data through Sigmoid activation.

[0165] The CFB module performs feature fusion based on the first flame feature data, fire multi-scale feature data, and fire color feature data to obtain fire fused features.

[0166] Real-time monitoring using camera devices yields flame monitoring images; fire detection modules then identify these flame monitoring images to obtain fire fusion characteristics.

[0167] A fire depth recognition module is constructed to identify fire fusion features; the fire warning area of ​​the fire monitoring image is determined by the target detection algorithm, and the fire warning area is identified by the Mobile SAM model to obtain the fire distribution area and the fire segmentation mask; the fire distribution area and the fire segmentation mask are identified by the DepthAnything model to obtain the fire depth data.

[0168] A flame depth recognition module is constructed, including a flame region recognition layer, a flame region segmentation layer, and a flame depth prediction layer;

[0169] The flame area identification layer identifies fire fusion features using a target detection algorithm to obtain the flame warning area;

[0170] The flame region segmentation layer identifies the flame warning region using a lightweight Mobile SAM model and segments the flame warning region using a transfer learning method to obtain the flame distribution region and the flame segmentation mask.

[0171] The flame depth prediction layer uses the DepthAnything depth estimation model to identify the flame distribution area and the flame segmentation mask, thereby obtaining the depth value corresponding to each pixel in the flame distribution area and thus obtaining the flame depth data.

[0172] A flame depth map is obtained based on the flame depth data. Based on the depth values ​​of the pixels in the flame depth map and the camera device parameters, the coordinates of each pixel in the flame distribution area are converted into world coordinates to obtain the three-dimensional spatial position.

[0173] The process of obtaining a three-dimensional spatial location from a flame depth map includes:

[0174] Transform the pixel coordinates of the flame distribution area to camera coordinates based on the flame depth map:

[0175]

[0176] Among them, (X) c ,Y c Z c () represents the coordinates in the camera coordinate system; (u,v) represents the pixel coordinates in the flame depth map; D(u,v) is the depth value of the pixel; c x and c y Principal point coordinates;

[0177] Then, based on the camera's extrinsic parameters, the camera coordinates are transformed to world coordinates to obtain the three-dimensional spatial position of the flame:

[0178]

[0179] Among them, (X) w, Y w Z w ) represents the coordinates in the world coordinate system; R and t are the camera extrinsic parameters, where R represents the rotation matrix and t represents the translation vector.

[0180] The system acquires time-series data of the three-dimensional spatial coordinates of the flame, identifies fire early warning coefficients based on the ignition point material, real-time environmental data, and time-series data, and then issues early warnings.

[0181] The process for identifying the fire early warning coefficient is as follows:

[0182] The ignition point is determined based on three-dimensional spatial coordinates, and the material at the ignition point and real-time environmental data are identified.

[0183] Feature identification is performed on the time series data of the three-dimensional spatial coordinates of the flame to obtain the spatial density distribution, surface curvature change, centroid displacement trajectory, vertical height distribution, and point cloud topological connectivity; then, combined with the ignition point material and real-time environmental data, a fire early warning coefficient is obtained for early warning.

[0184] By recognizing the three-dimensional spatial coordinates of flames in multiple dimensions, it is possible to accurately determine how a fire changes over time and provide early warnings.

[0185] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A visual recognition-based intelligent fire detection and positioning method, characterized in that, The method comprises the following steps: Collecting flame images of multiple scenes, identifying and labeling the flame images to obtain a multi-element flame dataset; Training an improved YOLOv8 model through the multi-element flame dataset to obtain a multi-stage flame depth identification model, including a fire detection module and a fire depth identification module; The improvement of the YOLOv8 model includes a GSConv module and a CFB module; The model improvement according to the CFB module includes replacing the C2f layer in the YOLOv8 model with the CFB module; The CFB module includes a main branch and a flame feature branch; The main branch is a C2f structure, which identifies the fused feature map to obtain first flame feature data; The flame feature branch includes a multi-scale feature extraction structure, which uses 3x3, 5x5, and 7x7 convolution kernels to capture different sizes of fire features, and then adaptively adjusts each scale feature through a dynamic weight mechanism to obtain fire multi-scale feature data; it also includes a color perception structure, which extracts color information based on 1x1 convolution and processes it through Sigmoid activation to obtain fire color feature data; The CFB module fuses the first flame feature data, the fire multi-scale feature data, and the fire color feature data to obtain fire fusion features; Real-time monitoring is performed through a camera device to obtain a flame monitoring image; the fire detection module identifies the flame monitoring image to obtain fire fusion features; The fire depth identification module identifies the fire fusion features; a target detection algorithm is used to determine a flame warning area of the flame monitoring image, a Mobile SAM model is used to identify the flame warning area to obtain a flame distribution area and a flame segmentation mask; a DepthAnything model is used to identify the flame distribution area and the flame segmentation mask to obtain flame depth data; A flame depth map is obtained according to the flame depth data, the depth values of the pixels in the flame depth map, and the camera device parameters, the pixel coordinates of the flame distribution area are converted into world coordinates to obtain a three-dimensional spatial position; Time series data of the flame three-dimensional spatial position are obtained, a fire warning coefficient is identified according to the ignition point material, real-time environmental data, and the time series data, and a warning is given.

2. The intelligent fire detection and positioning method based on visual recognition according to claim 1, wherein: The multi-element flame dataset includes flame images, image labeling information, and image depth information; The acquisition process is as follows: Pictures of fire scenes taken in multiple environments are obtained to obtain flame images; The flame regions in the flame images are labeled, including boundary labeling and morphological labeling, to obtain image labeling information; The features of the flame regions and the features of the non-flame regions in the flame images are identified to obtain image depth information.

3. The intelligent fire detection and positioning method based on visual recognition according to claim 1, wherein: The model improvement according to the GSConv module includes replacing the convolution layer in the YOLOv8 model with the GSConv module; The GSConv module divides the convolution operation into multiple groups for processing, for capturing local features in space; the formula is expressed as: ; wherein, represents a fused feature map obtained by splicing outputs of multiple groups; represents the number of divided groups; represents the first group input feature map, represents a corresponding convolution kernel; * represents a convolution operation; represents a splicing function.

4. The intelligent fire detection and positioning method based on visual recognition according to claim 1, wherein: The flame depth recognition module is constructed, and comprises a flame region recognition layer, a flame region segmentation layer and a flame depth prediction layer. The flame region recognition layer identifies the fire fusion features through a target detection algorithm to obtain a flame warning region. The flame region segmentation layer identifies the flame warning region through a lightweight Mobile SAM model, and segments the flame warning region through a transfer learning method to obtain a flame distribution region and a flame segmentation mask. The flame depth prediction layer identifies the flame distribution region and the flame segmentation mask through a DepthAnything depth estimation model to obtain a depth value corresponding to each pixel in the flame distribution region, and obtains flame depth data.

5. The intelligent fire detection and positioning method based on visual recognition according to claim 1, wherein: the process of obtaining the three-dimensional space position according to the flame depth map comprises:

6. The intelligent fire detection and positioning method based on visual recognition according to claim 1, wherein: the identification process of the fire warning coefficient comprises: determining a fire point according to the three-dimensional space coordinates, and identifying the fire point material and real-time environment data; identifying the time series data of the flame three-dimensional space coordinates to obtain the spatial density distribution, the surface curvature change, the centroid displacement trajectory, the vertical height distribution and the point cloud topological connectivity; and then combining the fire point material and the real-time environment data to identify the fire warning coefficient and perform warning. Converting flame distribution area pixel coordinates to camera coordinates according to flame depth map: ; wherein, represents a coordinate in the camera coordinate system; represents a pixel coordinate in the flame depth map; is a depth value for a pixel; and is a principal point coordinate; According to the camera external parameter, the camera coordinates are converted to the world coordinates to obtain the three-dimensional space position of the flame: ; wherein, represents a coordinate in a world coordinate system; , is a camera extrinsic parameter, represents a rotation matrix, represents a translation vector. ​ ​

Images