Flame profile segmentation method and device in fire, medium and product

Abstract

The invention discloses a flame contour segmentation method and device in a fire, a medium and a product, and the method comprises the steps: extracting fire fact features of an original fire image based on a fact strategy module of a fire image segmentation network; an anti-fact strategy module based on the fire image segmentation network extracts fire anti-fact features of the anti-fact fire image; a hypothetical anti-fact strategy module based on the fire image segmentation network extracts hypothetical fire anti-fact features of the anti-fact negative example image; obtaining a target flame feature according to the fire fact feature and the hypothetical fire anti-fact feature; inputting the target flame features into a prediction module of a fire image segmentation network to obtain a target flame contour segmentation result; the target flame contour segmentation result is a mask image of the flame contour, so that flame static parameters are accurately obtained, and accurate segmentation of the flame contour is realized.

Description

Technical Field

[0001] The present application relates to the technical field of image semantic segmentation, and in particular to a method, device, medium and product for segmenting flame contours in a fire. Background Art

[0002] At present, deep learning methods based on encoder-decoder cascade architecture can use labeled data to train network models, thereby predicting the category of each pixel in the image and achieving end-to-end image segmentation. However, such methods usually perform well when dealing with targets with fixed shapes and clear edges, but face challenges when dealing with non-rigid objects such as smoke and flames.

[0003] Non-rigid objects such as smoke and flames have no fixed shape and size, and their edges are usually fuzzy, making it difficult to accurately segment the flame outline in fire scenes. Summary of the invention

[0004] The purpose of this application is to provide a method, device, medium and product for segmenting flame contours in a fire, so as to solve the problem that it is difficult to accurately segment flame contours in a fire scene.

[0005] To achieve the above objectives, this application provides the following solutions:

[0006] In a first aspect, the present application provides a method for segmenting a flame contour in a fire, comprising:

[0007] A fact strategy module based on a fire image segmentation network extracts fire fact features of an original fire image; the original fire image includes a flame area and an irrelevant background area; the fire fact features are features obtained by aggregating features of the flame area and features of the irrelevant background area;

[0008] A counterfactual strategy module based on a fire image segmentation network extracts fire counterfactual features of a counterfactual fire image; the counterfactual fire image is obtained by covering the flame in the original fire image with an artificial mask;

[0009] The hypothetical counterfactual strategy module based on the fire image segmentation network extracts hypothetical fire counterfactual features of the counterfactual negative image; the counterfactual negative image is an assumed background image without flames;

[0010] Obtaining target flame characteristics according to the fire fact characteristics and the hypothetical fire counterfactual characteristics;

[0011] The target flame feature is input into the prediction module of the fire image segmentation network to obtain the target flame contour segmentation result; the target flame contour segmentation result is a mask image of the flame contour; the flame static parameters corresponding to the mask image of the flame contour include flame height, flame width and flame area.

[0012] In a second aspect, the present application provides a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method for segmenting the flame contour in a fire described above.

[0013] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method for segmenting the outline of a flame in a fire as described above.

[0014] In a fourth aspect, the present application provides a computer program product, including a computer program, characterized in that when the computer program is executed by a processor, the steps of the method for segmenting the flame contour in a fire described above are implemented.

[0015] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0016] The present application provides a method, device, medium and product for segmenting flame contours in a fire. First, in a factual strategy module, the original fire image is input into the factual strategy module for feature extraction to obtain fire factual features. Then, in a counterfactual strategy module, the counterfactual fire image is input into the counterfactual strategy module to complete feature extraction and obtain fire counterfactual features. Then, the fire counterfactual features are simulated through counterfactual negative images and hypothetical counterfactual strategy modules to obtain hypothetical fire counterfactual features, and then accurate target flame features are obtained based on the fire factual features and hypothetical fire counterfactual features. Based on the accurate target flame features and the prediction module, the target flame contour segmentation result determined can accurately obtain the mask image of the flame contour, and then accurately obtain the flame static parameters, thereby achieving accurate segmentation of the flame contour. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 A schematic flow chart of a method for segmenting flame contours in a fire provided in an embodiment of the present application;

[0019] Figure 2 A fire situation monitoring framework diagram provided in an embodiment of the present application;

[0020] Figure 3A flow chart of an example of fire emergency management application in an emergency management center provided in an embodiment of the present application;

[0021] Figure 4 This is a multi-scale target flame contour segmentation result diagram of three groups of combustion image sequences by the SCF+GMMSeg model provided in the embodiment of the present application; wherein, Figure 4 (a) is the original image sequence diagram of the first combustion process of the GMMSeg model; Figure 4 (b) is the original image sequence diagram of the second combustion process of the GMMSeg model; Figure 4 (c) is the original image sequence diagram of the third combustion process of the GMMSeg model;

[0022] Figure 5 This is a multi-scale target flame contour segmentation result diagram of three groups of combustion image sequences by the SCF+SeaFormer_large model provided in the embodiment of the present application; wherein, Figure 5 (a) is the original image sequence of the first combustion process of the SeaFormer_large model; Figure 5 (b) is the original image sequence of the second combustion process of the SeaFormer_large model; Figure 5 (c) is the original image sequence of the third combustion process of the SeaFormer_large model;

[0023] Figure 6 It is a flame static parameter diagram of the five combustion processes inferred by SCF+GMMSeg provided in the embodiment of this application; wherein, Figure 6 (a) is the flame height diagram obtained by the GMMSeg model; Figure 6 (b) is the flame width diagram obtained by the GMMSeg model; Figure 6 (c) in the figure is the flame area diagram obtained by the GMMSeg model;

[0024] Figure 7 It is a flame static parameter diagram in five combustion processes inferred by the SCF+SeaFormer_large model provided in the embodiments of the present application; Figure 7 (a) is the flame height map obtained by the SeaFormer_large model; Figure 7 (b) is the flame width diagram obtained by the SeaFormer_large model; Figure 7 (c) is the flame area diagram obtained by the SeaFormer_large model;

[0025] Figure 8 It is a dynamic flame parameter diagram of the five combustion processes inferred by SCF+GMMSeg provided in the embodiments of this application; Figure 8 (a) is the flame height change rate diagram obtained by the GMMSeg model; Figure 8 (b) is the flame width change rate diagram obtained by the GMMSeg model; Figure 8 (c) is the flame area change rate diagram obtained by the GMMSeg model;

[0026] Fig. 9 It is a dynamic flame parameter diagram of the five combustion processes inferred by SCF+SeaFormer_large provided in the embodiments of this application; Fig. 9 (a) is the flame height change rate diagram obtained by the SeaFormer_large model; Fig. 9 (b) is the flame width change rate diagram obtained by the SeaFormer_large model; Fig. 9 (c) in the figure is the flame area change rate diagram obtained by the SeaFormer_large model. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0028] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0029] Existing fire detection methods based on visual information use CNN and Transformer encoder-decoder cascade architectures to segment fire images. This type of technology automatically learns image features through structures such as CNN, which can more comprehensively and efficiently extract flame semantic information in images and improve the performance of flame contour segmentation. The end-to-end training capability of deep learning methods makes them more flexible and generalizable, so they can often better adapt to various complex scenarios in flame segmentation tasks, thereby improving the accuracy and robustness of flame segmentation.

[0030] The FireViTNet model based on the encoder-decoder cascade architecture introduces a segmentation method that combines visual Transformers with convolutional neural networks, aiming to effectively process complex and dynamic fire images. FireViTNet uses the lightweight MobileViT as the basic architecture and enhances the spatial resolution of the model through CNN, thereby improving the segmentation accuracy of irregular targets such as flames and smoke. In order to further improve the segmentation performance, FireViTNet combines attention mechanisms and advanced segmentation modules. Among them, the CBAM attention mechanism is used to dynamically adjust the weights of feature maps so that the model can pay more attention to key areas and detail features; the DenseASPP module processes flame morphology of different scales through multi-scale feature fusion, enhancing the multi-scale perception ability of the model; SP pooling effectively improves the refinement and robustness of flame segmentation by selectively aggregating features from different regions.

[0031] Deep learning methods based on the encoder-decoder cascade architecture can use labeled data to train network models, thereby achieving predictions for each pixel category and realizing end-to-end image segmentation. Such methods usually perform well when dealing with targets with fixed shapes and clear edges, but face challenges when dealing with non-rigid objects such as smoke and flames. Since these objects have no fixed shape and size, and their edges are usually fuzzy, it is difficult to accurately segment the flame contour in fire scenes. In addition, the dynamic nature of flames and the fluidity of smoke make traditional segmentation methods prone to misclassification or omission when dealing with these complex scenes, thus affecting the accuracy and reliability of fire image analysis. Therefore, in fire monitoring and emergency response, more advanced segmentation models are needed that can handle these irregular and complex targets and improve the segmentation accuracy of flame areas in fire images.

[0032] like Figure 1 As shown, the present application provides a method for segmenting flame contours in a fire, comprising:

[0033] Step 101: extracting fire fact features of an original fire image based on a fact strategy module of a fire image segmentation network; the original fire image includes a flame area and an irrelevant background area; the fire fact features are features obtained by aggregating features of the flame area and features of the irrelevant background area.

[0034] In some embodiments, a fact strategy module based on a fire image segmentation network extracts fire fact features of an original fire image, specifically comprising: inputting the original fire image into the fire fact feature encoder to generate low-level fire fact basic features; and inputting the low-level fire fact basic features into the fire fact context sensor to generate the fire fact features.

[0035] Reference Figure 2 Specifically, the design of the fact strategy module is to simulate the phenomenon that the flame in the original fire image has background interference. In this path, first, the fire fact feature encoder extracts the basic features of the input original fire image. The specific calculation formula is:

[0036] F f =Fea f (I)

[0037] Among them, F f is the extracted low-level fire fact basic features; I is the original fire image; Fea f Fire fact feature encoder.

[0038] Among them, the fire fact feature encoder is preferably a module formed by removing some layers from CNN models such as ResNet and VggNet. Secondly, the fire fact context sensor is used to complete feature aggregation and generate features of the flame area and irrelevant background area, that is, the fire fact feature. The specific formula is:

[0039] G f =Con f (F f )

[0040] Among them, G f For fire fact characteristics, Con f Contextual sensor for fire facts.

[0041] Among them, the fire fact feature contains the mixed contextual features of global flame and background semantic information, that is, the mixed features of flame features and fire counterfactual features; the fire fact context sensor is a module based on the attention mechanism.

[0042] Step 102: extracting fire counterfactual features of a counterfactual fire image based on a counterfactual strategy module of a fire image segmentation network; the counterfactual fire image is obtained by covering the flame in the original fire image with an artificial mask.

[0043] Specifically, the present application designs a counterfactual strategy module consisting of a fire counterfactual feature encoder and a fire counterfactual context sensor. The counterfactual fire image is constructed as follows:

[0044] I cf =II f

[0045] Among them, I is the original fire image, I f is an image with only the flame area, I cfis a fire counterfactual image with only background information. Since there is no flame area in the counterfactual fire image, the initially generated feature map only contains background information. The process can be expressed as:

[0046] F cf =Fea cf (I m )

[0047] Among them, I m represents the counterfactual fire image; F cf Represents the low-level background features of the extracted counterfactual fire image. Secondly, the fire counterfactual context sensor is used to complete feature aggregation and generate deep semantic features of irrelevant background areas, namely, fire counterfactual features. The specific process can be expressed as:

[0048] G cf =Con cf (F cf )

[0049] Among them, G cf is a fire counterfactual feature that contains global background semantic information.

[0050] Step 103: extracting hypothetical fire counterfactual features of counterfactual negative images based on the hypothetical counterfactual strategy module of the fire image segmentation network; the counterfactual negative images are assumed to be images with no flames and background.

[0051] Specifically, the counterfactual strategy module is simulated by assuming the counterfactual strategy module to generate the fire counterfactual feature G cf The main component of the hypothetical counterfactual strategy module is the hypothetical fire counterfactual context sensor. f As input, the simulation features (i.e., the counterfactual features of the hypothetical fire) are generated. This process can be expressed as:

[0052] G pc =Con pc (F f )

[0053] Among them, G pc Counterfactual characteristics of hypothetical fire.

[0054] Step 104: Obtain target flame characteristics according to the fire fact characteristics and the hypothetical fire counterfactual characteristics.

[0055] In some embodiments, the difference between the fire fact feature of the factual strategy module and the hypothetical fire counterfactual feature of the hypothetical counterfactual strategy module is the target flame feature; the feature difference between the fire counterfactual feature of the counterfactual strategy module and the hypothetical fire counterfactual feature of the hypothetical counterfactual strategy module is less than a preset threshold.

[0056] Specifically, counterfactual analysis aims to separate the features of pure flame areas from mixed features. The fact strategy module and the counterfactual strategy module constitute an ideal subtractive fire counterfactual analysis. f Subtract the fire counterfactual feature G from cf , and obtain the unbiased flame characteristic G p , the calculation formula of unbiased flame characteristics is:

[0057] G p =G f -G cf

[0058] It is worth noting that since counterfactual data requires manual calibration, it can only exist in the training phase of the counterfactual strategy module, and only hypothetical fire counterfactual features can be used in the inference phase of the model (hypothetical fire counterfactual features are required to minimize the feature differences with the fire fact features during training, that is, the hypothetical fire counterfactual features are the approximate solution of the ideal fire counterfactual features).

[0059] It should be noted that the fire fact feature encoder and the fire counterfactual feature encoder share the same weights and will be frozen during the training of the fire image segmentation network. The reason for adopting this strategy is to make the context perception module only have the ability to distinguish between background and fire targets to avoid the backbone network from becoming interference.

[0060] In order to obtain the unbiased target flame feature, first, the fire fact feature and the hypothetical fire counterfactual feature are subtracted to calculate the target unbiased flame feature. The specific calculation formula is:

[0061]

[0062] Secondly, the convolutional layer is used to map the target unbiased flame features Mapping back to the original fire image size, we can obtain the target flame feature G with rich semantics. s , the specific process can be expressed as:

[0063]

[0064] Step 105: Input the target flame feature into the prediction module of the fire image segmentation network to obtain a target flame contour segmentation result; the target flame contour segmentation result is a mask image of the flame contour; the flame static parameters corresponding to the mask image of the flame contour include flame height, flame width and flame area.

[0065] Specifically, the prediction module of the fire image segmentation network realizes the rapid classification of each pixel in the original fire image based on the target flame features. The prediction module is actually a pixel-level classifier, which is generally composed of multiple convolutional layers and fully connected layers, and each convolutional layer is followed by a batch normalization layer.

[0066] Among them, based on the mask image of the flame outline in the original fire image, the flame height, width and flame area are obtained, and then the static state information of the flame is represented from the perspective of spatial characteristics.

[0067] Specifically, a fully connected layer is used to generate the category label of each pixel to obtain the true pixel category segmentation result. The calculation formula is:

[0068]

[0069] Among them, FC is a fully connected operation.

[0070] The flame contour segmentation result is represented as a mask format Mask∈{0, 1, 2} H×W or Mask∈{0,1} H×W , get the mask image of the flame outline, the specific process is as follows

[0071] First, based on the binary mask result Mask, obtain the coordinates of all non-zero elements in it, where the binary refers to {0,1}, which are background pixels and flame pixels respectively. Zero elements refer to background pixels, and non-zero elements refer to: 1 is flame pixels, 2 is smoke pixels, and 1 is the main research here. Because this method can be extended to smoke segmentation, non-zero elements are used here.

[0072] S flame ={(a i , b i )|Mask(a i , b i )≠0,0 <i≤n c}

[0073] Among them, (a i , b i ) represents the position of the i-th non-zero element in the mask image, 0≤a i ≤W and 0≤b i ≤H,n c Indicates the number of flame pixels.

[0074] Secondly, for all a i and b i To sort, first sort in ascending order according to the vertical axis. If the vertical axes are the same, sort in ascending order according to the horizontal axis.

[0075] S′ flame = sort(S flame )

[0076] Finally, the sorted flame pixel sequence S′ flame Perform contour extraction. Start with the first point (x1, y1) in the sorted sequence, take it as the first point on the contour, and delete it from the sequence. Find all points (x1, y1) where x=x1 or y=y1 from the remaining points. j ,y j ), where j is the jth point after sorting. Add it to the contour coordinate point sequence and delete it from the sequence. Then continue to search for the next contour point until the sequence is empty. The above process can be expressed as:

[0077] C flame =contour(S′ flame , (x1, y1))={(x1, y1), (x2, y2),..., (x n ,y n )}

[0078] Among them, contour represents the contour point sequence extraction method, n is the number of contour points, C flame That is, the flame initial contour coordinate points of the flame contour mask image. Based on the flame initial contour coordinate points, support can be provided for subsequent flame width, flame height, and flame area calculations.

[0079] The edge coordinate point C obtained by flame contour segmentation flame It does not fit the real contour well enough, so in order to obtain more accurate flame height, flame width and flame area, fusion and smoothing are used. The specific methods are:

[0080] First, perform Canny edge detection on the input original fire image I to obtain the edge point set E flame .

[0081] E flame =Canny(I)

[0082] For each pixel p in E flame In C, if pixel p is flame If the pixel p is near the pixel in C flame In the process, this process is repeated continuously, and finally the fused edge C′ is obtained. flame .

[0083] According to C′ flame Calculate the flame height H flame , Width W flame and area A flameThe height of the flame target can be defined as the difference between the maximum and minimum Y coordinates in the flame edge coordinate point sequence.

[0084] H flame =max(Y)-min(Y)

[0085] Among them, max means taking the maximum value. The width of the flame target can be defined as the difference between the maximum and minimum values ​​of the X coordinate in the flame edge coordinate point sequence, that is:

[0086] W flame =max(X)-min(X)

[0087] Among them, min means taking the minimum value. Finally, the flame area A is calculated using the polygon area method flame , flame area A flame The calculation formula is:

[0088]

[0089] After the above operations, more accurate flame height, width and area are obtained through fusion and smoothing.

[0090] It is worth noting that if the flame in the image is scattered, it will have multiple contour sequences. The flame height, width and area are calculated as follows:

[0091]

[0092] Among them, d is the number of times the flame target appears, that is, the number of flame contours.

[0093] In some embodiments, the construction process of the fire image segmentation network includes: inputting the historical original fire image into the fact strategy module to obtain the initial fire fact feature; inputting the historical counterfactual fire image into the counterfactual strategy module to obtain the initial fire counterfactual feature; inputting the historical counterfactual negative example image into the hypothetical counterfactual strategy module to obtain the initial hypothetical fire counterfactual feature; determining the first loss function of the irrelevant background area according to the initial fire counterfactual feature and the initial hypothetical fire counterfactual feature; obtaining the initial flame feature of the flame area according to the initial fire fact feature and the initial hypothetical fire counterfactual feature; determining the initial flame contour segmentation result according to the initial flame feature and the prediction module; determining the second loss function according to the initial flame contour segmentation result and the real flame contour segmentation result; wherein the real flame contour segmentation result is obtained based on label classification of each pixel in the historical original fire image; determining the fire image segmentation network according to the first loss function and the second loss function.

[0094] In some embodiments, a first loss function of the irrelevant background area is determined based on the initial fire counterfactual features and the initial hypothetical fire counterfactual features, specifically comprising: inputting a historical counterfactual fire image into the fire counterfactual feature encoder to generate low-level background features of the irrelevant background area; inputting the low-level background features of the irrelevant background area into the fire counterfactual context sensor to generate an initial fire counterfactual feature; inputting a historical counterfactual negative image into the hypothetical fire counterfactual context sensor to generate an initial hypothetical fire counterfactual feature; obtaining feature differences based on the initial hypothetical fire counterfactual feature and the initial fire counterfactual feature; and using a smoothing norm loss function to measure the feature differences to generate the first loss function.

[0095] In order to make the hypothetical fire counterfactual feature G pc and the fire counterfactual feature G cf As similar as possible, a smooth l1 norm loss function is used to measure the feature difference between them, and the first loss is generated. According to the feature difference between the initial fire counterfactual feature and the initial hypothetical fire counterfactual feature, the first loss L is determined. sim The specific formula is:

[0096] L sim =sl1(G pc , G cf )

[0097] By minimizing L sim , so that the hypothetical counterfactual context perception Con pc It is possible to explore the fire fact feature G f The ability to separate pure background features from the fire counterfactual feature G. cf The counterfactual characteristic G of the hypothetical fire pc Replace it to solve the problem of lack of ground-truth masks during inference of fire image segmentation network.

[0098] In some embodiments, determining the fire image segmentation network according to the first loss function and the second loss function specifically includes: comparing the target loss with the target preset threshold according to the first loss function and the second loss function, and when the target loss is less than or equal to the target preset threshold, determining that the currently trained fire image segmentation network is the fire image segmentation network.

[0099] Specifically, the factual strategy module, the hypothetical counterfactual strategy module, and the counterfactual strategy module are all activated during the training process. When segmenting fire pixels, the fire image segmentation network should minimize the feature difference between the features generated by the hypothetical fire counterfactual context sensor and the fire counterfactual context sensor. This feature difference is the first loss, i.e., L sim.

[0100] Wherein, each pixel in the original fire image is classified to obtain a true pixel category segmentation result; wherein the pixel category includes flame pixels and background pixels; the initial prediction module is a pixel-level classifier; the flame feature is input into the initial prediction module to obtain a predicted pixel category segmentation result; based on the predicted pixel category segmentation result and the true pixel category segmentation result, a second loss (cross entropy loss) is determined. Wherein, the predicted pixel category segmentation result is the initial flame contour segmentation result; the true pixel category segmentation result is the true flame contour segmentation result.

[0101] Segmentation results based on the initial flame contour And the real flame contour segmentation result y ij The difference is the second loss (cross entropy loss), using L ce Indicates that L ce It can be expressed as:

[0102]

[0103] Among them, y ij Represents the real pixel category segmentation result, that is, the real flame contour segmentation result; Represents the predicted pixel category segmentation result, that is, the initial flame contour segmentation result.

[0104] Therefore, a weighted combination of multiple loss terms is used to obtain the target loss L, which is used to train the fire image segmentation network.

[0105] L=λ ce L ce +λ sim L sim

[0106] Among them, L ce represents the cross entropy loss, λ ce and λ sim The parameters represent the weights of the two loss terms.

[0107] It is important to note that the truth mask is available during training but not during inference. The Hypothetical Counterfactual Policy module is built to replace the functionality of the fire counterfactual policy. Therefore, the fire counterfactual policy is removed during inference.

[0108] In some embodiments, after the target flame feature is input into the prediction module of the fire image segmentation network to obtain the target flame contour segmentation result, it also includes: based on the mask images of two frames of flame contours at adjacent moments, determining the change amount and time difference between two flame static parameters at adjacent moments; determining the change rate of the flame contour over time according to the change amount and the time difference; and monitoring and analyzing the change rate as the dynamic situation of the flame to obtain a fire situation detection result.

[0109] Among them, the fire situation detection result is to express the dynamic development trend of the fire from the perspective of time series characteristics, that is, Figure 3 Fire situation monitoring in.

[0110] In practical applications, suppose there is a series of continuous original fire images I seq ={I1, I2, ..., I num}, where num represents the length of the fire image sequence. For each image I i , the outline coordinate points of the corresponding flame can be obtained by the method mentioned above Here use Represents the set of contour coordinate points of the flame target in the i-th image at time t.

[0111] The rate of change of flame height, flame width and flame area can be approximately expressed by calculating the ratio of the change of flame height, width and area in two adjacent frames of images to the time interval. Assume that at time t, the flame height is H flame (t), flame width is W flame (t), the flame area is A flame (t). For consecutive time points t1 and t2, the changes in flame height, flame width and flame area can be expressed as:

[0112] ΔH=H flame (t2)-H flame (t1)

[0113] ΔW=W flame (t2)-W flame (t1)

[0114] ΔA=A flame (t2)-A flame (t1)

[0115] The change rate of flame height, flame width and flame area can be calculated by derivation. The specific calculation formula is:

[0116]

[0117] Among them, H′ flame (t), W′flame and A′ flame They represent the rates of change of flame height, flame width and flame area at time t respectively; It represents the limit where the time interval Δt approaches 0.

[0118] In practical applications, the fire emergency management of the emergency management center is used as a specific application example, such as Figure 3 shown.

[0119] Five sets of flame burning image sequences from the experimental scene were collected by a camera with a fixed viewing angle. The flame contour segmentation method in the fire was applied, that is, the flame pixels were classified based on the flame features after subtracting the fire fact features and the hypothetical fire counterfactual features. This method is the Subtractive CounterFactual (SCF) method.

[0120] Specifically, the SCF+GMMSeg model or the SCF+SeaFormer_large model can be used to infer the combustion images in the combustion image sequence. The segmentation effects of the two models are as follows: Figure 4 and Figure 5 As shown in the figure, GMMSeg is an image semantic segmentation model based on Gaussian mixture model and adaptive clustering, and its full name is Gaussian mixture based generative semantic segmentation models.

[0121] SegFormer is an image semantic segmentation model based on the Transformer structure, and its full name is Simple and efficient design for semantic segmentation with transformers.

[0122] The original image sequence of each combustion process includes two rows, the lower row is the target flame contour segmentation result corresponding to the upper row. The flame pixels are highlighted in green, while the background pixels are not colored and consistent with the original fire image.

[0123] There are weak flame targets in the initial stage of fire and the decay and extinguishing stage. There are fewer flame pixels in the flame area, but SCF+GMMSeg and SCF+SeaFormer_large are extremely sensitive to the subtle flame area. In the intense burning stage, the flame is obvious and large in scale. SCF+GMMSeg and SCF+SeaFormerlarge can obtain the mapping relationship between pixels and flame semantics and correctly identify the flame outline. In addition, when the flame is burning, light and shadow refracted by the flame will appear in the image background, which has a similar color level with the flame, but SCF+GMMSeg and SCF+SeaFormerlarge do not mistake the light and shadow for flame, which shows that it can automatically filter out interference features.

[0124] Based on the above segmentation of the five groups of combustion image sequences to obtain the five groups of flame contour sequences, the flame static parameter calculation method in step 105 is executed. The flame static parameters are as follows: Figure 6 and Figure 7 As shown in Figure 2, the flame static parameter inversion results obtained by the SCF+GMMSeg and SCF+SeaFormer_large models are basically consistent. Figure 6 or Figure 7 Among the three sub-images, the most obvious pattern is the flame height. Overall, from 0 to 100 seconds, the flames in the five groups of combustion experiments burned rapidly, reaching the peak values ​​of height, width and area at around 100 seconds. Subsequently, the various combustion state parameters remained stable or fluctuated slightly, indicating that the flame was in an intense combustion stage at this time. After 300 seconds, the first, third, fourth and fifth groups of combustion experiments entered the attenuation and extinction stage, and the fire gradually decreased. The second group of combustion experiments had a longer duration of intense combustion and entered the third stage of combustion at around 400 seconds. For single images at different combustion times, SCF+GMMSeg and SCF+SeaFormer_large can accurately calculate reasonable combustion parameters. As shown in Table 1, Table 1 is Figure 6 (a) and Figure 7 Table showing the correlation between the first combustion sequence in (a) and the flame combustion trend obtained by Mangs et al.

[0125] Table 1

[0126]

[0127] The flame height obtained by the SCF+GMMSeg and SCF+SeaFormer_large methods is consistent with the flame combustion trend obtained by Mangs et al. through thermocouple tracking, reaching 0.9838 and 0.9728 respectively, both close to 1, as shown in Table 1. This shows that the flame static parameters obtained by the proposed fire situation monitoring model are consistent with the change trend obtained by the physical device. This also fully proves that the flame combustion parameters calculated by the fire contour model improved by the subtraction counterfactual are reliable and effective.

[0128] Based on the flame static parameters calculated in step 105, dynamic flame parameters can be obtained. The dynamic flame parameter results are as follows: Figure 8 and Fig. 9 shown.

[0129] The dynamic flame parameter inversion results obtained by the SCF+GMMSeg and SCF+SeaFormer_large models are basically consistent. In the initial stage of the fire, the flame first spreads rapidly in the height and width directions, and the spread rate gradually decreases before reaching the intense combustion stage; after entering the intense combustion stage in the middle of the fire, the rate of change is 0 or gradually remains stable; in the late stage of the fire, the flame continues to decay, the various rates of change are negative, and the flame height, width and area continue to shrink. As shown in Table 2, Table 2 is Figure 8 (a) and Fig. 9 Table showing the correlation between combustion sequence 4 in (a) and the flame combustion trend obtained by Mangs et al.

[0130] Table 2

[0131]

[0132] The dynamic flame parameters obtained by the SCF+GMMSeg and SCF+SeaFormer_large models are more consistent than 0.95 with the flame peak evolution trend data obtained by Chen et al. using a physical projection system, which shows that the dynamic flame parameters obtained by the proposed fire situation monitoring model are reliable and reasonable.

[0133] The method proposed in this application can address the challenge of difficult fine-grained fire contour segmentation caused by blurred flame edges. The counterfactual analysis in causal reasoning is integrated into the fire pixel-level segmentation process. By removing the flames from the original fire image, a counterfactual sample containing only the background is imagined, and the difference between the "image facts" and "pure background counterfactual" features is maximized to eliminate the influence of mixed pixels. Ultimately, fine-grained pixel-level segmentation of flames and high-precision recognition of easily confused edge pixels are achieved, and based on the flame contour segmentation results, the flame static parameters and flame dynamic parameters are accurately calculated to provide time-series dynamic fire development trend information. This application can be combined with existing fire segmentation models to improve the performance of existing models. This fine-grained fire detection method has obvious advantages in the perception of details in fire scenes, and can accurately determine the boundaries and scope of fires, assisting fire departments in making rescue decisions during disasters and summarizing lessons learned after disasters.

[0134] The input of this application is an original fire image with flames. The flame contour is segmented for the input original fire image to obtain the flame contour segmentation result, and the flame static information and dynamic situation information are calculated based on the flame contour segmentation result. The method used in this application can also obtain a mask image of the flame contour, and the final output is the flame static information and dynamic situation information.

[0135] In an exemplary embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the above method is implemented when the processor executes the computer program.

[0136] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, and the computer program implements the above method when executed by a processor.

[0137] In an exemplary embodiment, a computer program product is provided, including a computer program, which implements the above method when executed by a processor.

[0138] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0139] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ReadOnlyMemory, ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0140] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0141] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0142] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0143] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A method for segmenting flame contours in a fire, characterized in that: include: The fact strategy module based on the fire image segmentation network extracts the fire fact features of the original fire image; The original fire image includes a flame area and an irrelevant background area; the fire fact feature is a feature obtained by aggregating the features of the flame area and the features of the irrelevant background area; A counterfactual strategy module based on a fire image segmentation network is used to extract fire counterfactual features of counterfactual fire images. The counterfactual fire image is obtained by covering the flame in the original fire image with an artificial mask; The hypothetical counterfactual strategy module based on the fire image segmentation network extracts hypothetical fire counterfactual features of counterfactual negative images; The counterfactual negative image is an assumed background image without flame; Obtaining target flame characteristics according to the fire fact characteristics and the hypothetical fire counterfactual characteristics; The target flame feature is input into the prediction module of the fire image segmentation network to obtain the target flame contour segmentation result; the target flame contour segmentation result is a mask image of the flame contour; the flame static parameters corresponding to the mask image of the flame contour include flame height, flame width and flame area.

2. The method for segmenting flame contours in a fire according to claim 1, characterized in that: The construction process of the fire image segmentation network includes: Inputting the historical original fire image into the fact strategy module to obtain the initial fire fact feature; Inputting historical counterfactual fire images into the counterfactual strategy module to obtain initial fire counterfactual features; Inputting the historical counterfactual negative example image into the hypothetical counterfactual strategy module to obtain the initial hypothetical fire counterfactual feature; Determining a first loss function for the irrelevant background area according to the initial fire counterfactual features and the initial hypothetical fire counterfactual features; Obtaining initial flame characteristics of the flame area according to the initial fire fact characteristics and the initial hypothetical fire counterfactual characteristics; Determining an initial flame contour segmentation result according to the initial flame feature and the prediction module; Determine a second loss function according to the initial flame contour segmentation result and the real flame contour segmentation result; wherein the real flame contour segmentation result is obtained based on label classification of each pixel in the historical original fire image; The fire image segmentation network is determined according to the first loss function and the second loss function.

3. The method for segmenting the flame contour in a fire according to claim 2, characterized in that: The factual strategy module includes a fire fact feature encoder and a fire fact context sensor connected in sequence; the counterfactual strategy module includes a fire counterfactual feature encoder and a fire counterfactual context sensor connected in sequence; the hypothetical counterfactual strategy module includes a hypothetical fire fact context sensor; the difference between the fire fact feature of the factual strategy module and the hypothetical fire counterfactual feature of the hypothetical counterfactual strategy module is the target flame feature; the feature difference between the fire counterfactual feature of the counterfactual strategy module and the hypothetical fire counterfactual feature of the hypothetical counterfactual strategy module is less than a preset threshold.

4. The method for segmenting the flame contour in a fire according to claim 3, characterized in that: Determining a first loss function of the irrelevant background area according to the initial fire counterfactual features and the initial hypothetical fire counterfactual features specifically includes: Inputting the historical counterfactual fire image into the fire counterfactual feature encoder to generate low-level background features of the irrelevant background area; Inputting the low-level background features of the irrelevant background area into the fire counterfactual context sensor to generate initial fire counterfactual features; Inputting the historical counterfactual negative example image into the hypothetical fire counterfactual context sensor to generate an initial hypothetical fire counterfactual feature; Obtaining a feature difference based on the initial hypothetical fire counterfactual feature and the initial fire counterfactual feature; The feature difference is measured using a smoothing norm loss function to generate the first loss function.

5. The method for segmenting the flame contour in a fire according to claim 3, characterized in that: The fact strategy module based on the fire image segmentation network extracts the fire fact features of the original fire image, including: Inputting the original fire image into the fire fact feature encoder to generate low-level fire fact basic features; The low-level fire fact basic features are input into the fire fact context sensor to generate the fire fact features.

6. The method for segmenting the flame contour in a fire according to claim 2, characterized in that: Determining the fire image segmentation network according to the first loss function and the second loss function specifically includes: According to the first loss function and the second loss function, the target loss is compared with a target preset threshold, and when the target loss is less than or equal to the target preset threshold, the fire image segmentation network currently being trained is determined to be the fire image segmentation network.

7. The method for segmenting flame contours in a fire according to claim 1, characterized in that: After inputting the target flame feature into the prediction module of the fire image segmentation network to obtain the target flame contour segmentation result, the method further includes: Based on the mask images of two frames of flame contours at adjacent moments, the change amount and time difference between two flame static parameters at adjacent moments are determined; Determining the rate of change of the flame profile over time according to the change amount and the time difference; The change rate is used as the flame dynamic situation for monitoring and analysis to obtain a fire situation detection result.

8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method for segmenting the contour of a flame in a fire according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for segmenting the contour of a flame in a fire according to any one of claims 1 to 7 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method for segmenting the contour of a flame in a fire according to any one of claims 1 to 7 are implemented.

Images