Intelligent fire detection and positioning method based on visual identification

Abstract

The invention relates to the technical field of fire monitoring, in particular to an intelligent fire detection and positioning method based on visual identification, which comprises the following steps: improving a YOLOv8 model, and obtaining a multi-stage flame depth identification model through multi-element flame data set training in combination with image segmentation and a depth estimation model; identifying the flame monitoring image through a multi-stage flame depth identification model, including determining an early warning area in the flame monitoring image through a target detection algorithm, and identifying the early warning area through a Mobile SAM model to obtain a distribution area and a segmentation mask; the distribution area and the segmentation mask are identified through a DepthAnything model, and flame depth data are obtained; internal and external parameter calibration is carried out on the monocular camera, flame depth data is combined, a flame pixel coordinate system is converted into a world coordinate system, and the three-dimensional space position of flame is obtained; the fire early warning coefficient is identified according to the three-dimensional space position of the flame, and fire early warning is accurately carried out.

Description

Technical Field

[0001] The present invention relates to the technical field of fire monitoring, and in particular to an intelligent fire detection and positioning method based on visual recognition. Background Art

[0002] Current visual sensing-based fire detection systems generally employ deep learning frameworks for flame recognition. Mainstream approaches include rectangular box detection algorithms based on the YOLO family and pixel-level segmentation techniques based on fully convolutional networks. These methods analyze spectral features in RGB images to identify flames, demonstrating strong real-time detection capabilities in common scenarios.

[0003] However, existing systems lack the ability to perceive the dynamic characteristics of flames in multiple dimensions. They are unable to identify the type of burning material and the stage of fire development based on trends in radiation intensity, hindering the precise formulation of firefighting strategies. For example, monocular vision systems struggle to accurately interpret the three-dimensional spatial distribution of flames, resulting in a loss of depth information. They are prone to misjudgment due to sudden changes in ambient lighting or obstructions caused by smoke. Furthermore, complex network structures pose computing resource bottlenecks for model deployment on embedded platforms.

[0004] Especially when precise fire extinguishing positioning is required, existing methods have significant shortcomings in flame outline segmentation accuracy and three-dimensional coordinate calculation. Rectangular frame detection cannot provide a detailed burn area mask, while monocular ranging methods based on two-dimensional images lack depth perception, leading to cumulative spatial positioning errors. These issues seriously restrict the environmental adaptability and positioning reliability of intelligent fire detection systems in complex scenarios.

[0005] Therefore, an intelligent fire detection and location method based on visual recognition is proposed. Summary of the Invention

[0006] The purpose of the present invention is to provide an intelligent fire detection and positioning method based on visual recognition, which improves the YOLOv8 model, combines image segmentation and depth estimation models, obtains a multi-stage flame depth recognition model, and trains the multi-stage flame depth recognition model using a multivariate flame data set; uses the multi-stage flame depth recognition model to recognize flame monitoring images, including determining warning areas in the flame monitoring images through a target detection algorithm, identifying the warning areas through a Mobile SAM model, and obtaining distribution areas and segmentation masks; using a DepthAnything model to identify the distribution areas and segmentation masks to obtain flame depth data; calibrates the internal and external parameters of a monocular camera, and combines the flame depth data to convert the flame pixel coordinate system into a world coordinate system to obtain the three-dimensional spatial position of the flame; the present invention obtains a fire warning coefficient based on the three-dimensional spatial position of the flame, and accurately performs fire warning.

[0007] To achieve the above object, the present invention provides the following technical solutions:

[0008] An intelligent fire detection and positioning method based on visual recognition, comprising:

[0009] Collect flame images of multiple scenes, identify and annotate the flame images, and obtain a multi-dimensional flame dataset;

[0010] By training the improved YOLOv8 model on a multi-element flame dataset, a multi-stage flame depth recognition model was obtained, including a fire detection module and a fire depth recognition module. The improvements to the YOLOv8 model include the GSConv module and the CFB module.

[0011] The flame monitoring image is obtained by real-time monitoring through the camera device; the flame monitoring image is recognized by the fire detection module to obtain the fire fusion feature;

[0012] A fire depth recognition module is constructed to identify fire fusion features. The flame warning area of the flame monitoring image is determined through the target detection algorithm. The flame warning area is identified through the Mobile SAM model to obtain the flame distribution area and flame segmentation mask. The flame distribution area and flame segmentation mask are identified through the DepthAnything model to obtain flame depth data.

[0013] A flame depth map is obtained based on the flame depth data, and the coordinates of each pixel in the flame distribution area are converted into world coordinates based on the depth values of the pixels in the flame depth map and the parameters of the camera device to obtain the three-dimensional spatial position;

[0014] The time series data of the flame's three-dimensional spatial position is obtained, and the fire warning coefficient is obtained based on the ignition point material, real-time environmental data and time series data to issue an early warning.

[0015] The multivariate flame dataset includes flame images, image annotation information, and image depth information; the acquisition process is as follows:

[0016] Obtain pictures of fire scenes taken in multiple environments to obtain flame images;

[0017] Annotate the flame area in the flame image, including boundary annotation and morphology annotation, to obtain image annotation information;

[0018] The flame area features and non-flame area features in the flame image are identified to obtain the image depth information.

[0019] Model improvements based on the GSConv module include replacing the convolutional layers in the YOLOv8 model with the GSConv module; the GSConv module divides the convolution operation into multiple groups for processing to capture local features in space;

[0020] Its formula is:

[0021]

[0022] Among them, X out Represents the fusion feature map obtained by splicing the outputs of multiple groups; G represents the number of groups divided; X g represents the g-th group of input feature maps, K g Represents X g The corresponding convolution kernel; * indicates the convolution operation; concat() indicates the concatenation function.

[0023] Model improvements based on the CFB module include: replacing the C2f layer in the YOLOv8 model with the CFB module;

[0024] The CFB module includes a main branch and a flame signature branch;

[0025] The main branch is a C2f structure, which performs feature recognition on the fusion feature map to obtain first flame feature data;

[0026] The flame feature branch includes a multi-scale feature extraction structure, which uses convolution kernels of three scales: 3×3, 5×5, and 7×7 to simultaneously capture fire features of different sizes. It then adaptively adjusts the features of each scale through a dynamic weight mechanism to obtain multi-scale fire feature data. It also includes a color perception structure, which extracts color information based on 1×1 convolution and obtains fire color feature data through Sigmoid activation processing.

[0027] The CFB module performs feature fusion based on the first flame feature data, the fire multi-scale feature data and the fire color feature data to obtain a fire fusion feature.

[0028] Construct a flame depth recognition module, including a flame area recognition layer, a flame area segmentation layer, and a flame depth prediction layer;

[0029] The flame area recognition layer identifies the fire fusion features through the target detection algorithm to obtain the flame warning area;

[0030] The flame area segmentation layer uses a lightweight Mobile SAM model to identify the flame warning area and segments the flame warning area using a transfer learning method to obtain the flame distribution area and flame segmentation mask.

[0031] The flame depth prediction layer identifies the flame distribution area and the flame segmentation mask through the DepthAnything depth estimation model, obtains the depth value corresponding to each pixel in the flame distribution area, and obtains flame depth data.

[0032] The process of obtaining the three-dimensional spatial position based on the flame depth map includes:

[0033] Convert the pixel coordinates of the flame distribution area to camera coordinates according to the flame depth map:

[0034]

[0035] Among them, (X c ,Y c ,Z c ) represents the coordinates in the camera coordinate system; (u, v) represents the pixel coordinates in the flame depth map; D(u, v) is the depth value of the pixel; c x and c y is the main point coordinate;

[0036] Then, the camera coordinates are converted to world coordinates based on the camera external parameters to obtain the three-dimensional spatial position of the flame:

[0037]

[0038] Among them, (X w, Y w ,Z w ) represents the coordinates in the world coordinate system; R, t are the camera extrinsic parameters, R represents the rotation matrix, and t represents the translation vector.

[0039] The identification process of the fire warning coefficient is:

[0040] Determine the fire point based on the three-dimensional spatial coordinates, and identify the fire point material and real-time environmental data;

[0041] Feature recognition is performed on the time series data of the flame's three-dimensional spatial coordinates to obtain the spatial density distribution, surface curvature change, center of mass displacement trajectory, vertical height distribution and point cloud topological connectivity; then, combined with the ignition point material and real-time environmental data identification, the fire warning coefficient is obtained for early warning.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] 1. The present invention improves the YOLOv8 model through the GSConv module and the CFB module to obtain a fire detection module. The GSConv module is an improved convolution method. By dividing the convolution operation into multiple groups for processing, it can better capture local features in space, and can enhance the model's sensitivity to local details while retaining global information. It also improves computational efficiency. Compared with traditional convolution, GSConv is computationally more efficient, can reduce the amount of calculation and shorten the training time, while maintaining the performance of the model and making it more adaptable. Through this replacement, YOLOv8 can better capture the local features of flames when processing fire images, while reducing computational overhead and improving detection speed.

[0044] 2. This invention uses a CFB module to identify flame characteristics. The CFB module adopts a dual-branch architecture. The main branch maintains the original C2f structure to ensure basic feature extraction capabilities. The fire feature branch is specifically optimized for fire characteristics, and a dynamic fusion mechanism enables adaptive feature combination. A multi-scale feature extraction branch is designed to simultaneously capture fire characteristics of varying scales. A dynamic weighting mechanism adaptively adjusts features at each scale. Combining color perception mechanisms with spatial attention to capture shape features, accurate flame identification is achieved.

[0045] 3. The present invention constructs a fire depth recognition module to identify fire fusion features; determines the flame warning area of the flame monitoring image through a target detection algorithm, identifies the flame warning area through the Mobile SAM model, and obtains the flame distribution area and flame segmentation mask; identifies the flame distribution area and flame segmentation mask through the DepthAnything model to obtain flame depth data; the present invention effectively combines technical solutions such as target detection box (YOLOv8), lightweight SAM (MobileSAM), accurate mask depth estimation (DepthAnything), and conversion of depth map to 3D point cloud. The advantage of this combined solution is that it can simultaneously provide high-precision flame segmentation, depth estimation, and 3D positioning in different environments and devices.

[0046] 4. The present invention determines the ignition point based on the three-dimensional spatial coordinates, and identifies the ignition point material and real-time environmental data; performs feature recognition on the time series data of the flame's three-dimensional spatial coordinates to obtain spatial density distribution, surface curvature changes, center of mass displacement trajectory, vertical height distribution and point cloud topological connectivity; and then combines the ignition point material and real-time environmental data to obtain a fire warning coefficient, thereby accurately identifying and warning of fire changes. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1This is a flow chart of an intelligent fire detection and location method based on visual recognition proposed by the present invention;

[0048] Figure 2 A diagram showing the steps for intelligent fire detection and location;

[0049] Figure 3 This is the YOLOv8 network structure diagram;

[0050] Figure 4 This is the GSConv module structure diagram;

[0051] Figure 5 This is the CFB module structure diagram;

[0052] Figure 6 This is the structure diagram of the fire feature branch module;

[0053] Figure 7 This is the structural diagram of the fire detection module;

[0054] Figure 8 It is a rectangular frame prompt image of the flame warning area;

[0055] Figure 9 It is the segmentation map of the flame distribution area within the target frame;

[0056] Figure 10 Convert the flame location map into a 3D point cloud in the fire environment. DETAILED DESCRIPTION

[0057] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0058] Example 1

[0059] The present invention proposes an intelligent fire detection and positioning method based on visual recognition, the process of which is as follows: Figure 1 Shown, including:

[0060] Collect flame images of multiple scenes, identify and annotate the flame images, and obtain a multi-dimensional flame dataset;

[0061] The multivariate flame dataset includes flame images, image annotation information, and image depth information; the acquisition process is as follows:

[0062] Obtain fire scenes shot in multiple environments to obtain flame images;

[0063] Annotate the flame area in the flame image, including boundary annotation and morphology annotation, to obtain image annotation information;

[0064] The flame area features and non-flame area features in the flame image are identified to obtain the image depth information.

[0065] Furthermore, the steps of intelligent fire detection and location are as follows: Figure 2 As shown, the steps of acquiring the multivariate flame data set in this embodiment are:

[0066] Flame image capture uses an ORBBEC Astra S depth camera to capture fire scenes in various environments (e.g., indoors, outdoors, at night, and during the day). The image data must cover a variety of flame colors, shapes, sizes, and complex background environments to ensure data diversity and representativeness.

[0067] Flame image annotation: Mark the precise boundaries of the flame area for each image to ensure high quality of the segmentation results.

[0068] Background information collection: The ORBBEC Astra S depth camera is also used to obtain depth information of the fire scene, providing raw data for subsequent depth estimation.

[0069] The improved YOLOv8 model is trained on a multi-element flame dataset to obtain a multi-stage flame depth recognition model, including a fire detection module and a fire depth recognition module. The improvements to the YOLOv8 model include the GSConv module and the CFB module. The YOLOv8 network structure is as follows: Figure 3 shown.

[0070] Model improvements based on the GSConv module include replacing the convolutional layers in the YOLOv8 model with the GSConv module; the GSConv module divides the convolution operation into multiple groups for processing to capture local features in space;

[0071] Its formula is:

[0072]

[0073] Among them, X out Represents the fusion feature map obtained by splicing the outputs of multiple groups; G represents the number of groups divided; X g represents the g-th group of input feature maps, K g Represents X g The corresponding convolution kernel; * indicates the convolution operation; concat() indicates the concatenation function.

[0074] In order to further improve the performance of the model in fire detection, the present invention replaces the original convolution layer (Conv) with the GSConv (Group Spatial Convolution) module, such as Figure 4 shown.

[0075] The recognition process of the GSConv module includes feature map input, group convolution and feature splicing:

[0076] Feature map input: Input feature map is X∈R C×H×W , where C is the number of channels, H is the height of the feature map, W is the width of the feature map, and the convolution kernel is Where G is the number of groups, C / G represents the number of channels in each group, and K h and K w are the height and width of the convolution kernel, respectively, indicating the spatial size of the convolution kernel.

[0077] Grouped convolution: Input features Figure X Divide into G groups, each group size is C / G. Perform convolution operations on each group separately. Each convolution operation is performed on the corresponding subset of the input feature map. For the g-th group (g∈{1,2,…,G}), its convolution operation can be expressed as: Among them, X g ∈R (C / G)×H×W is the g-th group of input feature maps, is the corresponding convolution kernel, and * indicates the convolution operation.

[0078] Concatenation of convolution outputs: concatenate the output of each group Splice into a new feature Figure X out :

[0079]

[0080] The number of channels of the output feature map will be restored to C, that is, X out ∈R C×H×W ;concat() represents the concatenation function.

[0081] The GSConv module is an improved convolution method with the following advantages: It enhances spatial locality. By dividing the convolution operation into multiple groups, GSConv can better capture local features in space. It can retain global information while increasing the model's sensitivity to local details. It improves computational efficiency. Compared with traditional convolution, GSConv is more efficient, reducing the amount of computation and shortening training time while maintaining model performance. It is more adaptable. The GSConv module can flexibly process images of different scales and complexities, making it particularly suitable for extracting key features in special scenarios such as fires. Through this replacement, YOLOv8 can better capture the local characteristics of flames when processing fire images, while reducing computational overhead and improving detection speed.

[0082] Model improvements based on the CFB module include: replacing the C2f layer in the YOLOv8 model with the CFB module;

[0083] The CFB module includes a main branch and a flame signature branch;

[0084] The main branch is a C2f structure, which performs feature recognition on the fusion feature map to obtain first flame feature data;

[0085] The flame feature branch includes a multi-scale feature extraction structure, which uses convolution kernels of three scales: 3×3, 5×5, and 7×7 to simultaneously capture fire features of different sizes. It then adaptively adjusts the features of each scale through a dynamic weight mechanism to obtain multi-scale fire feature data. It also includes a color perception structure, which extracts color information based on 1×1 convolution and obtains fire color feature data through Sigmoid activation processing.

[0086] The CFB module performs feature fusion based on the first flame feature data, the fire multi-scale feature data and the fire color feature data to obtain a fire fusion feature.

[0087] Flame characteristics are unique. First, their shape is irregular and variable, lacking fixed geometric features. Furthermore, flames vary in scale, ranging from small to large. Their color is also distinct, typically exhibiting a reddish-yellow hue. Flames also have unique textures, characterized by a fluid flow. Furthermore, existing object detection models have limitations. Conventional feature extraction is not targeted enough, lacking focus on fire-specific characteristics, and single-scale feature extraction is ineffective.

[0088] According to the above analysis, the present invention designs a CFB (C2fFireBranch) module, such as Figure 5As shown, to address these issues, the CFB module adopts a dual-branch architecture. The main branch maintains the original C2f structure to ensure basic feature extraction capabilities, while the fire feature branch (FFB) is specifically optimized for fire characteristics. A dynamic fusion mechanism enables adaptive feature combination. The CFB module designs a multi-scale feature extraction branch to address flame characteristics of different scales. It uses convolution kernels of three sizes: 3×3, 5×5, and 7×7 to simultaneously capture fire characteristics of different sizes. A dynamic weighting mechanism adaptively adjusts features at each scale. The CFB module also incorporates a color perception mechanism. A dedicated color-sensitive branch extracts color information through 1×1 convolution and sigmoid activation to highlight the importance of color features. The CFB module uses spatial attention to capture shape features and channel attention to enhance feature representation, creating a cascaded combination of attention mechanisms.

[0089] Input features X∈R C×H×W , first is the calculation process of the main branch; the first convolution: F1 = Conv(X); where F1 represents the first convolution data; Conv() represents the convolution layer;

[0090] Perform feature segmentation: F 1a ,F 1b =Split(F1); where F 1a Indicates the first segmentation data; F 1b Represents the second segmentation data; Split() represents the feature segmentation operation;

[0091] Bottleneck Operation: B i =Bottleneck(B i-1 ), i∈[1,n], where B0=F 1b ; B i Represents the bottleneck data of the i-th node; Bottleneck() represents the Bottleneck operation;

[0092] Final result: F main =Conv(concat(F 1a ,F 1b ,B1,...,B n )); where F main Indicates the first flame characteristic data.

[0093] For the flame feature branch FFB (FireFeatureBranch), such as Figure 6 As shown: First, input feature X∈R C ×H×W , perform multi-scale feature extraction to obtain basic convolution feature data F b ;

[0094] Perform multi-scale recognition on basic convolution feature data, multi-scale features: M i =Convi(F b ),i∈{3,5,7}, where i corresponds to the size of the convolution kernel; M i Represents multi-scale feature data of size i; Convi() represents multi-scale convolution.

[0095] Color feature extraction: F c =Sigmoid(Conv(Conv(X))); F c Represents fire color feature data; Sigmoid() represents the Sigmoid activation function;

[0096] Then perform feature fusion:

[0097] F combined =concat(F b *W0,M3*W1,M5*W2,M7*W3);

[0098] F color =F combined *F c ;

[0099] F fire =Conv(concat(F combined ,F color ]));

[0100] F final =Conv(concat(F main *W f0 ,F fire *W f1 ));

[0101] Among them, F combined Represents the multi-scale characteristic data of fire; W0, W1, W2, W3 are multi-scale weights respectively; F color Indicates color fusion data; F fire represents the first fire fusion data; F final Indicates fire fusion characteristics; F main Represents the first flame characteristic data; W f0 、W f1 are feature weights respectively.

[0102] The design of the CFB module fully considers the special needs of fire detection. Through innovative technical solutions, it realizes highly targeted and high-performance feature extraction capabilities, adapting to the detection of fire scenes in various environments.

[0103] After improvement, the structure of the fire detection module is as follows Figure 7As shown; in this embodiment, X∈R C×H×W Generally refers to input features, representing different inputs in different scenarios. The specific input-output relationship can be determined based on Figure 6 The structure is determined.

[0104] In order to verify the recognition effect of the fire detection module, it is verified; verification model 1, verification model 2, verification model 3 and verification model 4 are constructed;

[0105] Among them, verification model 1 is the fire detection module of this application, which improves the YOLOv8 model through the GSConv module and the CFB module; verification model 2 improves the YOLOv8 model only through the GSConv module; verification model 3 improves the YOLOv8 model only through the CFB module; verification model 4 is the YOLOv8 model; the similarity between the identified feature data and the actual feature data is used as the accuracy.

[0106] After multiple tests, the data obtained are shown in Table 1.

[0107] Table 1 Model validation data table

[0108] Model GSConv module CFB module Accuracy Recognition speed Verification Model 1 include include 0.924 53ms Verification Model 2 include Not included 0.887 87ms Verification Model 3 Not included include 0.905 89ms Verification Model 4 Not included Not included 0.856 134ms

[0109] According to the data in Table 1, after the improvements of the GSConv module and the CFB module, the recognition accuracy and speed of the fire detection module are significantly improved.

[0110] The flame monitoring image is obtained by real-time monitoring through the camera device; the flame monitoring image is recognized by the fire detection module to obtain the fire fusion feature;

[0111] A fire depth recognition module is constructed to identify fire fusion features. The flame warning area of the flame monitoring image is determined through the target detection algorithm. The flame warning area is identified through the Mobile SAM model to obtain the flame distribution area and flame segmentation mask. The flame distribution area and flame segmentation mask are identified through the DepthAnything model to obtain flame depth data.

[0112] Construct a fire depth recognition module, including a flame area recognition layer, a flame area segmentation layer, and a flame depth prediction layer;

[0113] The flame area recognition layer identifies the fire fusion features through the target detection algorithm to obtain the flame warning area;

[0114] The target detection algorithm model can accurately identify the location of the flame in the image and generate a rectangular frame prompt of the flame warning area for each flame target, such as Figure 8As shown in Figure 2. These detection boxes contain the approximate location and range of the flame in the image and serve as input for the subsequent segmentation process. Assume that the flame detection box output by the target detection algorithm is B = (x1, y1, x2, y2), where (x1, y1) is the coordinate of the upper left corner of the box and (x2, y2) is the coordinate of the lower right corner of the box, indicating the boundary of the flame warning area.

[0115] The flame area segmentation layer uses a lightweight Mobile SAM model to identify the flame warning area and segments the flame warning area using a transfer learning method to obtain the flame distribution area and flame segmentation mask.

[0116] MobileSAM is optimized for low-computing resources. Using techniques like transfer learning, it achieves precise segmentation while maintaining high efficiency. During input, the flame box B serves as a hint, providing spatial constraints on the target. Based on this hint, MobileSAM focuses on the area within the box for further precise segmentation.

[0117] After receiving the frame prompt, MobileSAM uses feature extraction and image understanding capabilities to identify the flame area within the frame and generate a corresponding flame segmentation mask. This segmentation mask accurately calibrates the pixel position of the flame and can segment the flame distribution area from the image, such as Figure 9 During the segmentation process, MobileSAM not only relies on box cues, but also further refines flame details in the image, such as the edge, shape, and color features of the flame through its embedded deep feature learning mechanism, thereby ensuring the accuracy of the segmentation results.

[0118] The segmentation result is output as a binary mask that contains the precise pixel locations of the flame distribution. This mask can be used for subsequent tasks such as flame depth estimation, position positioning, and point cloud conversion, providing accurate data support for fire detection systems.

[0119] Furthermore, an improved segmentation network can be used to replace SAM; MobileSAM, as a lightweight image segmentation model, plays an important role in this invention. However, other lightweight segmentation networks (such as DeepLabV3+, Fast-SCNN, etc.) can also be used as alternatives. These networks also have high segmentation performance and have different designs and optimization directions, which may provide better performance in certain hardware or application scenarios.

[0120] The flame depth prediction layer identifies the flame distribution area and the flame segmentation mask through the DepthAnything depth estimation model, obtains the depth value corresponding to each pixel in the flame distribution area, and obtains flame depth data.

[0121] This paper proposes a method for depth estimation and 3D information generation that combines a flame segmentation mask with a depth estimation model. By using an accurate flame segmentation mask, this method can effectively extract useful depth information from fire scenes and generate high-quality 3D data support.

[0122] During the target detection phase, an improved YOLOv8 model is used to detect objects in fire images, obtaining a frame hint containing the flame warning area. A lightweight SAM model (MobileSAM) is then used to accurately segment the flames, generating a high-quality binary flame segmentation mask. This mask clearly demarcates the pixel locations of the flame distribution area, ensuring high-precision depth estimation.

[0123] When creating a depth dataset, interpolation can be used to fill holes in the depth map. This paper uses the nearest neighbor interpolation method to estimate the depth values in these holes. This step ensures that the depth map is not affected by holes during training, enhancing the continuity and stability of depth estimation.

[0124] The DepthAnything model can estimate the corresponding depth value for each pixel by learning the depth of the flame area in the fire scene, and then generate a depth map of the flame area. The depth estimation model outputs a depth D based on the input image and mask information. f ∈R H×W , where each pixel value D f (x,y) represents the depth value corresponding to each pixel. Use the mask M of the flame distribution area f (Binarization) is used to filter valid depth values. Specifically, only the depth data within the masked area is retained. This step ensures that the depth estimation focuses only on the flame distribution area, thereby obtaining the depth value of the flame.

[0125]

[0126] in, is the depth map after applying the mask, mask M f Make sure only the depth value of the flame is preserved.

[0127] Furthermore, the present invention uses the DepthAnything depth estimation model. While this model performs well in this application, other depth estimation techniques, such as MiDaS (Monocular Depth Estimation) or MonoDepth, can be considered. These models can also perform monocular depth estimation and may be more suitable in some applications.

[0128] The core idea of this invention is to improve the efficiency and accuracy of the fire monitoring and positioning system by combining technical modules such as target detection box, lightweight SAM model (MobileSAM), accurate flame segmentation mask, depth estimation and 3D point cloud generation.

[0129] A flame depth map is obtained based on the flame depth data, and the coordinates of each pixel in the flame distribution area are converted into world coordinates based on the depth values of the pixels in the flame depth map and the parameters of the camera device to obtain the three-dimensional spatial position;

[0130] Depth map generated by the depth estimation model , combined with the internal and external parameters of the camera, the depth information of the flame can be converted into a three-dimensional spatial position; the three-dimensional spatial position is a three-dimensional spatial coordinate. This process involves the following steps:

[0131] Depth to 3D coordinate conversion: According to the depth value of each pixel in the depth map, each pixel is converted into coordinates in 3D space through the intrinsic and extrinsic parameters of the camera.

[0132] The process of obtaining the three-dimensional spatial position based on the flame depth map includes:

[0133] Convert the pixel coordinates of the flame distribution area to camera coordinates according to the flame depth map:

[0134]

[0135] Among them, (X c ,Y c ,Z c ) represents the coordinates in the camera coordinate system; (u, v) represents the pixel coordinates in the flame depth map; D(u, v) is the depth value of the pixel; c x and c y The principal point coordinates are the intersection of the camera optical axis and the image plane, usually close to the center of the image.

[0136] Then, the camera coordinates are converted to world coordinates based on the camera external parameters to obtain the three-dimensional spatial position of the flame:

[0137]

[0138] Among them, (X w, Y w ,Z w ) represents the coordinates in the world coordinate system; R, t are the camera extrinsic parameters, R represents the rotation matrix, and t represents the translation vector.

[0139] Point cloud generation: Through the above transformation, the depth values of all flame areas are converted into three-dimensional space coordinates to represent the position of the flame in three-dimensional space, such as Figure 10 shown.

[0140] The combination of precise masking and depth estimation in this invention effectively provides accurate three-dimensional spatial positioning of flames. Addressing holes in the depth map ensures the continuity of depth data, thereby preventing the impact of inaccurate depth estimation on subsequent processing. Furthermore, through accurate flame depth estimation, this invention can provide precise three-dimensional flame position data for subsequent automatic fire extinguishing devices, improving the efficiency of fire emergency response.

[0141] Variations of the 3D spatial positioning method: This invention uses a method for converting depth maps into 3D point clouds. This method is well suited for data acquired by depth cameras, but if other types of sensors (such as LiDAR, laser radar) are used, 3D spatial data can also be acquired in different ways. In this case, the generation of flame depth information and the 3D positioning method may be different. An alternative approach is to consider generating a 3D point cloud based on LiDAR data and using a similar depth estimation algorithm for depth supplementation.

[0142] The time series data of the flame's three-dimensional spatial coordinates are obtained, and the fire warning coefficient is obtained based on the ignition point material, real-time environmental data and time series data identification to issue an early warning.

[0143] The identification process of the fire warning coefficient is:

[0144] Determine the fire point based on the three-dimensional spatial coordinates, and identify the fire point material and real-time environmental data;

[0145] Feature recognition is performed on the time series data of the flame's three-dimensional spatial coordinates to obtain the spatial density distribution, surface curvature change, center of mass displacement trajectory, vertical height distribution and point cloud topological connectivity; then, combined with the ignition point material and real-time environmental data identification, the fire warning coefficient is obtained for early warning.

[0146] The flame volume growth rate is obtained through spatial density distribution identification to quantify the fire expansion speed; the combustion intensity index is obtained through surface curvature change identification to reflect the turbulence intensity of the flame surface; the spread direction angle is obtained through center of mass displacement trajectory identification to predict the fire propagation path; the ceiling jet risk coefficient is obtained through vertical height distribution identification to evaluate the probability of dangerous spread in the vertical direction; the isolation zone breakthrough warning is obtained through point cloud topological connectivity identification to determine whether the flame has crossed the physical barrier.

[0147] Through multi-dimensional flame three-dimensional spatial coordinate recognition, the change of fire over time can be accurately obtained and early warning can be issued.

[0148] The present invention improves the YOLOv8 model to obtain a multi-stage flame depth recognition model, and trains the multi-stage flame depth recognition model using a multivariate flame data set; uses the multi-stage flame depth recognition model to recognize flame monitoring images, including determining a warning area in the flame monitoring image using a target detection algorithm, identifying the warning area using a Mobile SAM model to obtain a distribution area and a segmentation mask; using a DepthAnything model to identify the distribution area and the segmentation mask to obtain flame depth data; obtains a flame depth map based on the flame depth data, converts each pixel into a coordinate in three-dimensional space, and obtains the three-dimensional spatial coordinates of the flame; and obtains a fire warning coefficient based on the three-dimensional spatial coordinates of the flame, thereby accurately performing a fire warning.

[0149] Example 2

[0150] The present invention proposes an intelligent fire detection and positioning method based on visual recognition, such as Figure 1 Shown, including:

[0151] Collect flame images of multiple scenes, identify and annotate the flame images, and obtain a multi-dimensional flame dataset;

[0152] The multivariate flame dataset includes flame images, image annotation information, and image depth information; the acquisition process is as follows:

[0153] Obtain fire scenes shot in multiple environments to obtain flame images;

[0154] Annotate the flame area in the flame image, including boundary annotation and morphology annotation, to obtain image annotation information;

[0155] The flame area features and non-flame area features in the flame image are identified to obtain the image depth information.

[0156] By training the improved YOLOv8 model on a multi-element flame dataset, a multi-stage flame depth recognition model was obtained, including a fire detection module and a fire depth recognition module. The improvements to the YOLOv8 model include the GSConv module and the CFB module.

[0157] Model improvements based on the GSConv module include replacing the convolutional layers in the YOLOv8 model with the GSConv module; the GSConv module divides the convolution operation into multiple groups for processing to capture local features in space;

[0158] Its formula is:

[0159]

[0160] Among them, X outRepresents the fusion feature map obtained by splicing the outputs of multiple groups; G represents the number of groups divided; X g represents the g-th group of input feature maps, K g Represents X g The corresponding convolution kernel; * indicates the convolution operation; concat() indicates the concatenation function.

[0161] Model improvements based on the CFB module include: replacing the C2f layer in the YOLOv8 model with the CFB module;

[0162] The CFB module includes a main branch and a flame signature branch;

[0163] The main branch is a C2f structure, which performs feature recognition on the fusion feature map to obtain first flame feature data;

[0164] The flame feature branch includes a multi-scale feature extraction structure, which uses convolution kernels of three scales: 3×3, 5×5, and 7×7 to simultaneously capture fire features of different sizes. It then adaptively adjusts the features of each scale through a dynamic weight mechanism to obtain multi-scale fire feature data. It also includes a color perception structure, which extracts color information based on 1×1 convolution and obtains fire color feature data through Sigmoid activation processing.

[0165] The CFB module performs feature fusion based on the first flame feature data, the fire multi-scale feature data and the fire color feature data to obtain a fire fusion feature.

[0166] The flame monitoring image is obtained by real-time monitoring through the camera device; the flame monitoring image is recognized by the fire detection module to obtain the fire fusion feature;

[0167] A fire depth recognition module is constructed to identify fire fusion features. The flame warning area of the flame monitoring image is determined through the target detection algorithm. The flame warning area is identified through the Mobile SAM model to obtain the flame distribution area and flame segmentation mask. The flame distribution area and flame segmentation mask are identified through the DepthAnything model to obtain flame depth data.

[0168] Construct a flame depth recognition module, including a flame area recognition layer, a flame area segmentation layer, and a flame depth prediction layer;

[0169] The flame area recognition layer identifies the fire fusion features through the target detection algorithm to obtain the flame warning area;

[0170] The flame area segmentation layer uses a lightweight Mobile SAM model to identify the flame warning area and segments the flame warning area using a transfer learning method to obtain the flame distribution area and flame segmentation mask.

[0171] The flame depth prediction layer identifies the flame distribution area and the flame segmentation mask through the DepthAnything depth estimation model, obtains the depth value corresponding to each pixel in the flame distribution area, and obtains flame depth data.

[0172] A flame depth map is obtained based on the flame depth data, and the coordinates of each pixel in the flame distribution area are converted into world coordinates based on the depth values of the pixels in the flame depth map and the parameters of the camera device to obtain the three-dimensional spatial position;

[0173] The process of obtaining the three-dimensional spatial position based on the flame depth map includes:

[0174] Convert the pixel coordinates of the flame distribution area to camera coordinates according to the flame depth map:

[0175]

[0176] Among them, (X c ,Y c ,Z c ) represents the coordinates in the camera coordinate system; (u, v) represents the pixel coordinates in the flame depth map; D(u, v) is the depth value of the pixel; c x and c y is the main point coordinate;

[0177] Then, the camera coordinates are converted to world coordinates based on the camera external parameters to obtain the three-dimensional spatial position of the flame:

[0178]

[0179] Among them, (X w, Y w ,Z w ) represents the coordinates in the world coordinate system; R, t are the camera extrinsic parameters, R represents the rotation matrix, and t represents the translation vector.

[0180] The time series data of the flame's three-dimensional spatial coordinates are obtained, and the fire warning coefficient is obtained based on the ignition point material, real-time environmental data and time series data identification to issue an early warning.

[0181] The identification process of the fire warning coefficient is:

[0182] Determine the fire point based on the three-dimensional spatial coordinates, and identify the fire point material and real-time environmental data;

[0183] Feature recognition is performed on the time series data of the flame's three-dimensional spatial coordinates to obtain the spatial density distribution, surface curvature change, center of mass displacement trajectory, vertical height distribution and point cloud topological connectivity; then, combined with the ignition point material and real-time environmental data identification, the fire warning coefficient is obtained for early warning.

[0184] Through multi-dimensional flame three-dimensional spatial coordinate recognition, the change of fire over time can be accurately obtained and early warning can be issued.

[0185] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. An intelligent fire detection and positioning method based on visual recognition, characterized in that: include: Collect flame images of multiple scenes, identify and annotate the flame images, and obtain a multi-dimensional flame dataset; By training the improved YOLOv8 model on a multi-element flame dataset, a multi-stage flame depth recognition model was obtained, including a fire detection module and a fire depth recognition module. The improvements to the YOLOv8 model include the GSConv module and the CFB module. The flame monitoring image is obtained by real-time monitoring through the camera device; the flame monitoring image is recognized by the fire detection module to obtain the fire fusion feature; Fire fusion feature recognition based on fire depth recognition module; The flame warning area of the flame monitoring image is determined by the target detection algorithm. The flame warning area is identified by the Mobile SAM model to obtain the flame distribution area and flame segmentation mask. The flame distribution area and flame segmentation mask are identified by the DepthAnything model to obtain the flame depth data. A flame depth map is obtained based on the flame depth data, and pixel coordinates of the flame distribution area are converted into world coordinates based on the depth values of pixels in the flame depth map and the camera device parameters to obtain a three-dimensional spatial position; The time series data of the flame's three-dimensional spatial position is obtained, and the fire warning coefficient is obtained based on the ignition point material, real-time environmental data and time series data to issue an early warning.

2. The intelligent fire detection and location method based on visual recognition according to claim 1, characterized in that: The multivariate flame dataset includes flame images, image annotation information and image depth information; The acquisition process is: Obtain pictures of fire scenes taken in multiple environments to obtain flame images; Annotate the flame area in the flame image, including boundary annotation and morphology annotation, to obtain image annotation information; The flame area features and non-flame area features in the flame image are identified to obtain the image depth information.

3. The intelligent fire detection and location method based on visual recognition according to claim 1, characterized in that: Model improvements based on the GSConv module include replacing the convolutional layers in the YOLOv8 model with the GSConv module; The GSConv module divides the convolution operation into multiple groups for processing to capture local features in space; the formula is expressed as: Among them, X out Represents the fusion feature map obtained by splicing the outputs of multiple groups; G represents the number of groups divided; X g represents the g-th group of input feature maps, K g Represents X g The corresponding convolution kernel; * indicates the convolution operation; concat() indicates the concatenation function.

4. The intelligent fire detection and location method based on visual recognition according to claim 3, characterized in that: Model improvements based on the CFB module include: replacing the C2f layer in the YOLOv8 model with the CFB module; The CFB module includes a main branch and a flame signature branch; The main branch is a C2f structure, which performs feature recognition on the fusion feature map to obtain first flame feature data; The flame feature branch includes a multi-scale feature extraction structure, which uses convolution kernels of three scales: 3×3, 5×5, and 7×7 to simultaneously capture fire features of different sizes. It then adaptively adjusts the features of each scale through a dynamic weight mechanism to obtain multi-scale fire feature data. It also includes a color perception structure, which extracts color information based on 1×1 convolution and obtains fire color feature data through Sigmoid activation processing. The CFB module performs feature fusion based on the first flame feature data, the fire multi-scale feature data and the fire color feature data to obtain a fire fusion feature.

5. The intelligent fire detection and location method based on visual recognition according to claim 1, characterized in that: Construct a flame depth recognition module, including a flame area recognition layer, a flame area segmentation layer, and a flame depth prediction layer; The flame area recognition layer identifies the fire fusion features through the target detection algorithm to obtain the flame warning area; The flame area segmentation layer uses a lightweight Mobile SAM model to identify the flame warning area and segments the flame warning area using a transfer learning method to obtain the flame distribution area and flame segmentation mask. The flame depth prediction layer identifies the flame distribution area and the flame segmentation mask through the DepthAnything depth estimation model, obtains the depth value corresponding to each pixel in the flame distribution area, and obtains flame depth data.

6. The intelligent fire detection and location method based on visual recognition according to claim 1, characterized in that: The process of obtaining the three-dimensional spatial position based on the flame depth map includes: Convert the pixel coordinates of the flame distribution area to camera coordinates according to the flame depth map: Z c =D(u,v) Among them, (X c ,Y c ,Z c ) represents the coordinates in the camera coordinate system; (u, v) represents the pixel coordinates in the flame depth map; D(u, v) is the depth value of the pixel; c x and c y is the main point coordinate; Then, the camera coordinates are converted to world coordinates based on the camera external parameters to obtain the three-dimensional spatial position of the flame: Among them, (X w, Y w ,Z w ) represents the coordinates in the world coordinate system; R, t are the camera extrinsic parameters, R represents the rotation matrix, and t represents the translation vector.

7. The intelligent fire detection and location method based on visual recognition according to claim 1, characterized in that: The fire warning coefficient identification process is as follows: determining the ignition point based on the three-dimensional spatial coordinates, and identifying the ignition point material and real-time environmental data; performing feature recognition on the time series data of the flame's three-dimensional spatial coordinates to obtain spatial density distribution, surface curvature changes, center of mass displacement trajectory, vertical height distribution and point cloud topological connectivity; and then combining the ignition point material and real-time environmental data to obtain the fire warning coefficient and issue a warning.

Images