A forest fire size discrimination method based on a multi-module fusion deep neural network

Abstract

The application discloses a forest fire size discrimination method based on a multi-module fusion deep neural network, and relates to the technical field of intelligent monitoring of forest fire images. The application first detects the sky and the tree trunk in the RGB image of the forest fire scene, determines the size of the forest scene and forms a scene coding vector; performs flame semantic segmentation on the RGB image to obtain a flame mask binary image and a flame area proportion; generates a mask binary image based on the tree trunk constraint according to the size of the forest scene, extracts flame visual features in combination with the Sobel gradient features; generates a feature scaling coefficient and a feature offset amount respectively by using the scene coding vector and the flame area proportion, modulates the flame visual features and splices to obtain fusion features, and determines the fire size category according to the fusion features. The application makes the scene size, the flame area proportion and the spatial structure jointly participate in discrimination, and improves the category discrimination ability.

Description

Technical Field

[0001] This invention relates to the field of intelligent monitoring technology for forest fire images, and in particular to a method for determining the size of a forest fire based on a multi-module fusion deep neural network. Background Technology

[0002] Forest fires are among the most destructive natural disasters globally, causing severe ecological damage and enormous economic losses, and threatening human lives. In recent years, influenced by global climate change, the probability, scope, and destructive power of forest fires have continued to rise.

[0003] Currently, forest fire monitoring based on drones equipped with visual devices has become the mainstream method. However, existing forest fire monitoring research mostly focuses on flame target detection (i.e., determining whether a fire has occurred or the location of the flames), while systematic exploration of fire scale and severity assessment is still lacking. In terms of actual forest fire emergency response, accurately understanding the size of the fire is crucial for subsequent firefighting force deployment and rescue plan formulation.

[0004] Existing forest fire monitoring technologies have the following shortcomings: 1) Lack of fire size discrimination capability: Existing methods mostly only output a binary classification result of whether a fire has occurred or the location of the flames, and cannot provide fine-grained classification information on fire size, which makes it difficult to meet the actual needs of emergency rescue decision-making. 2) Lack of an effective fusion mechanism for multi-source features: Existing methods fail to effectively fuse scene semantic information, flame spatial distribution features and flame visual features, which limits the accuracy of fire size determination; 3) Lack of dedicated dataset support: Real forest fire image data is extremely scarce. Existing public datasets are mostly geared towards flame detection rather than fire size discrimination tasks, and lack labeled data for fine-grained classification of fire size. Summary of the Invention

[0005] The purpose of this invention is to provide a forest fire size discrimination method based on a multi-module fusion deep neural network, which can solve the problem of insufficient information for distinguishing fire size categories due to insufficient fusion of scene semantic information, flame spatial distribution characteristics and flame visual features in existing forest fire monitoring, and improve the ability to distinguish different fire size categories.

[0006] To achieve the above objectives, the present invention provides the following solution: A method for determining the magnitude of forest fires based on a multi-module fusion deep neural network includes: The sky and tree trunks in the RGB image of a forest fire scene are detected. The size of the forest scene is determined based on the detection results. If tree trunks are detected, it is determined to be a small scene; if only the sky is detected, it is determined to be a large scene; if neither the sky nor tree trunks are detected, it is determined to be a medium scene. The size of the forest scene is then encoded to obtain a scene encoding vector. Flame semantic segmentation is performed on the RGB image to obtain a binary image of the flame mask and the flame area ratio; Generate a binary mask image based on tree trunk constraints according to the size of the forest scene. In small scenes, the flame binary mask image is cropped with the minimum Y value of the tree trunk detection box corresponding to the detected tree trunk as the threshold, and the flame area above the tree trunk is retained. In medium or large scenes, the flame binary mask image is used as a binary mask image based on tree trunk constraints. Sobel gradient features are calculated on the binary mask image based on trunk constraints. The obtained Sobel gradient features are then concatenated with the binary mask image of the flame and the binary mask image based on trunk constraints to extract the visual features of the flame. Feature scaling factors and feature offsets are generated based on the scene encoding vector. The flame visual features are modulated using the feature scaling factors and feature offsets to obtain modulated features based on the scene encoding vector. Based on the flame area ratio, a feature scaling factor and a feature offset are generated. The flame visual features are modulated using the feature scaling factor and the feature offset to obtain modulated features based on the flame area ratio. By splicing the modulation features based on the scene encoding vector and the modulation features based on the flame area ratio, a fused feature is obtained, and the fire size category is determined based on the fused feature.

[0007] Preferably, the sky and tree trunks in the RGB image of the forest fire scene are detected, the size of the forest scene is determined based on the detection results, and the size of the forest scene is encoded to obtain a scene encoding vector, including: The YOLOv11 object detection network was used to detect the sky and tree trunks in RGB images of a forest fire scene. The size of the forest scene was determined based on the detection results of the sky and tree trunks; One-hot vector encoding is performed on the size of the forest scene, encoding the small scene as [1 0 0], the medium scene as [01 0], and the large scene as [0 0 1], to obtain the scene encoding vector.

[0008] Preferably, the RGB image is subjected to flame semantic segmentation to obtain a flame mask binary image and flame area proportion, including: We used an improved U-Net to perform flame semantic segmentation on RGB images of forest fire scenes, set the downsampling number to 3 times, and extracted flame features step by step using 32, 64, 128, and 256 feature channels; In the decoding stage, transposed convolution is used for feature upsampling, and shallow spatial features and deep semantic features of the corresponding scale are fused through skip connections. The fused features are mapped to a single-channel flame prediction map through 1×1 convolution, and pixel-level segmentation results are obtained using the Sigmoid function. The pixel-level segmentation results are then converted into a binary flame mask map based on the segmentation threshold. The number of flame pixels in the binary image of the flame mask is counted, and the ratio of the number of flame pixels to the total number of pixels in the RGB image of the forest fire scene is determined as the flame area ratio.

[0009] Preferably, generating a mask binary map based on tree trunk constraints according to the size of the forest scene includes: When the forest scene is small, the minimum Y value of the tree trunk detection box corresponding to the detected tree trunk is used as the threshold to crop the flame mask binary map, retaining the flame area above the tree trunk, and obtaining a mask binary map based on tree trunk constraints. When the forest scene is of medium or large size, the flame mask binary image is used as a mask binary image based on the tree trunk constraint.

[0010] Preferably, Sobel gradient features are calculated on the binary mask image based on trunk constraints. The obtained Sobel gradient features are then concatenated with the flame binary mask image and the binary mask image based on trunk constraints to extract the flame visual features, including: Sobel gradient feature calculation is performed on the binary mask image based on the trunk constraint to obtain the horizontal gradient map, the vertical gradient map, and the gradient magnitude map; The horizontal gradient map, vertical gradient map, gradient magnitude map, flame mask binary map, and mask binary map based on trunk constraints are stitched together to obtain a five-channel image. The five-channel image was input into a lightweight convolutional neural network for feature extraction, resulting in 64-dimensional visual features of the flame.

[0011] Preferably, feature scaling factors and feature offsets are generated based on the scene encoding vector, and the flame visual features are modulated using the feature scaling factors and feature offsets to obtain modulated features based on the scene encoding vector, including: The scene encoding vector is input into a multi-layer fully connected neural network for feature mapping. The network structure of the multi-layer fully connected neural network is 3→16→32→64→128, resulting in a 128-dimensional scene feature vector. The 128-dimensional scene feature vector is split into two 64-dimensional vectors along the feature dimension, which are used as the feature scaling factor and the feature offset, respectively. The feature scaling factor is multiplied element-wise with the flame visual feature, and the feature obtained by multiplication is added element-wise with the feature offset to obtain the modulation feature based on the scene coding vector.

[0012] Preferably, a feature scaling factor and a feature offset are generated based on the flame area ratio, and the flame visual features are modulated using the feature scaling factor and the feature offset to obtain modulated features based on the flame area ratio, including: The flame area ratio is input into the feature modulation parameter generation neural network for feature mapping. The network structure of the feature modulation parameter generation neural network is 1→16→32→64→128, resulting in a 128-dimensional feature vector. The 128-dimensional feature vector is split into two 64-dimensional vectors along the feature dimension, which are used as the feature scaling factor and the feature offset, respectively. The feature scaling factor is multiplied element-wise with the flame visual feature, and the feature obtained from the multiplication is added element-wise with the feature offset to obtain the modulation feature based on the flame area ratio.

[0013] Preferably, the fused feature is a 128-dimensional feature vector formed by splicing modulation features based on scene coding vector and modulation features based on flame area ratio in the feature dimension.

[0014] Preferably, the fire size category is determined based on fusion characteristics, including: The fused features are input into a shallow fully connected neural network to classify the size of the fire. The network structure of the shallow fully connected neural network is 128→64→32→11, resulting in an 11-dimensional classification vector. The size category of the fire is determined based on the 11-dimensional classification vector.

[0015] Preferably, the fire size categories include small scene with no fire, small scene with small fire, small scene with medium fire, small scene with large fire, medium scene with no fire, medium scene with small fire, medium scene with medium fire, medium scene with large fire, large scene with no fire, large scene with medium fire, and large scene with large fire.

[0016] The present invention discloses the following beneficial effects: 1) This invention detects the sky and tree trunks in RGB images of forest fire scenes, determines small, medium or large scenes based on the detection results, and forms scene encoding vectors. This allows the size of the forest scene to be used as input information for judging the size of the fire in feature processing, reducing the impact of differences in the forest scene on the judgment of the size of the fire when the judgment is based solely on the image features of the flame itself.

[0017] 2) This invention generates a binary mask image based on tree trunk constraints according to the size of the forest scene. In small scenes, the flame area above the tree trunk is retained based on the position of the tree trunk detection box. In medium and large scenes, the flame mask image is retained. Then, Sobel gradient feature calculation is performed on the binary mask image based on tree trunk constraints. The flame visual features are extracted by combining the flame mask image and the binary mask image based on tree trunk constraints. This allows the position of the flame relative to the tree trunk, the distribution of the flame area, and the edge structure of the flame to participate in the determination of the fire size, thereby improving the ability to distinguish between different flame spatial forms.

[0018] 3) This invention generates feature scaling coefficients and feature offsets based on the scene encoding vector and the flame area ratio, and modulates the flame visual features using the corresponding feature scaling coefficients and feature offsets, so that the forest scene size and the flame area ratio participate in the adjustment of the flame visual features, thereby combining scene information, flame area information and flame visual features, and improving the problem of insufficient representation of fire size when multiple types of information are used independently.

[0019] 4) This invention splices the modulation features based on the scene encoding vector and the modulation features based on the flame area ratio, and determines the fire size category based on the obtained fused features. This allows the fire size discrimination to simultaneously utilize the forest scene size, the flame area ratio, and the flame visual features formed after tree trunk constraints and Sobel gradient calculation, thereby increasing the feature differences between different fire size categories and improving the ability to distinguish between fire size categories. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 A flowchart of the method provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the overall structure of a multi-module fusion deep neural network provided in an embodiment of the present invention; Figure 3 This is an example diagram of a small scene where only the tree trunk is detected, provided as an embodiment of the present invention. Figure 4 Example diagram of a small scene where the sky and tree trunks are detected simultaneously, provided for embodiments of the present invention; Figure 5 This is an example diagram of a large scene provided for an embodiment of the present invention; Figure 6 This is a scene example diagram provided in an embodiment of the present invention; Figure 7 This is a schematic diagram of the overall structure of the improved U-Net network provided in an embodiment of the present invention; Figure 8 A schematic diagram of the convolutional neural network structure for flame structure feature extraction provided in an embodiment of the present invention; Figure 9 This is a schematic diagram of the neural network structure for modulating forest scene size features provided in an embodiment of the present invention; Figure 10 A schematic diagram of the flame area ratio feature modulation neural network structure provided in an embodiment of the present invention; Figure 11 This is a schematic diagram of the neural network structure for classifying fire size according to an embodiment of the present invention; Figure 12 This is a small-scale fireless example diagram provided in an embodiment of the present invention; Figure 13 This is an example diagram of a small scene with a small fire provided in an embodiment of the present invention; Figure 14 This is an example diagram of fire in a small scene provided in an embodiment of the present invention; Figure 15 This is an example diagram of a small-scale fire provided in an embodiment of the present invention; Figure 16 This is an example diagram of a fireless scene provided in an embodiment of the present invention; Figure 17 This is an example diagram of a small fire scene provided in an embodiment of the present invention; Figure 18 This is an example diagram of fire in a scene provided in an embodiment of the present invention; Figure 19 This is an example diagram of a large fire scene provided in an embodiment of the present invention; Figure 20 This is an example diagram of a large-scale fireless scene provided in an embodiment of the present invention; Figure 21 This is an example diagram of a large-scale fire scene provided in an embodiment of the present invention; Figure 22 This is an example diagram of a large-scale fire scene provided in an embodiment of the present invention; Figure 23 The accuracy curve of the fire size discrimination network provided in the embodiments of the present invention; Figure 24 This is a network loss curve for fire size discrimination provided in an embodiment of the present invention; Figure 25 The Grad-CAM analysis provided in this embodiment of the invention uses original forest fire images; Figure 26 This is a visualization diagram of the Grad-CAM multi-module fusion deep neural network provided in an embodiment of the present invention. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0024] Figure 1 The method flowchart provided in the embodiments of the present invention is as follows: Figure 1 As shown, this embodiment provides a method for determining the magnitude of a forest fire based on a multi-module fusion deep neural network, including: Step 100: Detect the sky and tree trunks in the RGB image of the forest fire scene, determine the size of the forest scene based on the detection results, determine the small scene when tree trunks are detected, determine the large scene when only the sky is detected, and determine the medium scene when neither the sky nor tree trunks are detected. Encode the size of the forest scene to obtain the scene encoding vector. Step 200: Perform flame semantic segmentation on the RGB image to obtain a binary image of the flame mask and the flame area ratio; Step 300: Generate a binary mask image based on tree trunk constraints according to the size of the forest scene. In small scenes, the flame binary mask image is cropped with the minimum Y value of the tree trunk detection box corresponding to the detected tree trunk as the threshold, and the flame area above the tree trunk is retained. In medium or large scenes, the flame binary mask image is used as a binary mask image based on tree trunk constraints. Step 400: Calculate the Sobel gradient features of the binary mask based on the trunk constraint, and extract the visual features of the flame by concatenating the obtained Sobel gradient features with the flame binary mask and the binary mask based on the trunk constraint. Step 500: Generate feature scaling coefficients and feature offsets based on the scene encoding vector, and use the feature scaling coefficients and feature offsets to modulate the flame visual features to obtain the modulated features based on the scene encoding vector; Step 600: Generate feature scaling factor and feature offset based on flame area ratio, and use feature scaling factor and feature offset to modulate flame visual features to obtain modulated features based on flame area ratio. Step 700: Concatenate the modulation features based on the scene coding vector and the modulation features based on the flame area ratio to obtain the fused features, and determine the fire size category based on the fused features.

[0025] Figure 2 This is a schematic diagram of the overall structure of a multi-module fusion deep neural network provided in an embodiment of the present invention. Figure 2 In this process, RGB images of a forest fire scene are processed into a flame segmentation section and a forest scene size determination section. The flame segmentation section generates a flame mask binary image (Mask) and a flame area ratio (R). The forest scene size determination section uses a YOLOv11 network to detect the sky and tree trunks, and generates a scene encoding vector (Scene Vector). The flame mask binary image (Mask) is further processed into a tree trunk-constrained mask binary image (Mask_trunk), a horizontal gradient image (Sobelx), a vertical gradient image (Sobely), and a gradient magnitude image (Sobelmag). The Mask, Mask_trunk, Sobelx, Sobely, and Sobelmag are then concatenated along the channel dimension to form a five-channel image.

[0026] Figure 2 The flame structure feature extraction convolutional neural network shown is formed sequentially. Each feature channel has a corresponding spatial size of [number] channels. The final-level feature size is After being flattened, it forms 3D features, then mapped sequentially to The final 64-dimensional flame visual feature (visual_feat) is formed. The flame area ratio R is fed into the flame area ratio feature modulation neural network, and then... After feature mapping, the data is split into 64-dimensional A2 and 64-dimensional B2; the scene encoding vector enters the forest scene size feature modulation neural network, and then... After feature mapping, it is split into 64-dimensional A1 and 64-dimensional B1.

[0027] Figure 2 The feature fusion section shown includes two independent feature modulation branches. A1 and B1 act on the 64-dimensional flame visual feature visual_feat to form a 64-dimensional modulation feature F1, and A2 and B2 act on the same flame visual feature visual_feat to form a 64-dimensional modulation feature F2. F1 and F2 are concatenated along the feature dimensions to form a 128-dimensional fusion feature. The 128-dimensional fusion feature is then passed through 64-dimensional, 32-dimensional, and 11-dimensional fire size classification neural networks to form an 11-dimensional classification vector. Figure 2 In this context, ConvolutionalBlock represents a convolutional block, One-dimensional Vector represents a one-dimensional vector, Concatenation represents feature concatenation, Element-wise Multiplication represents element-wise multiplication, and Element-wise Addition represents element-wise addition.

[0028] This embodiment uses a visual sensor mounted on a drone or fixed monitoring equipment to acquire RGB images of forest fire scenes, with an image resolution of [missing information]. The image is set to pixels, and the RGB image of the forest fire scene is used as the common input for forest scene size discrimination and flame semantic segmentation.

[0029] For step 100, the YOLOv11 object detection network was used to detect the sky and tree trunks in the RGB images of the forest fire scene. The dataset used for forest scene size determination contained a total of 7581 images, with 6221 images in the training set and 1360 images in the validation set. The training set contained 3568 sky object annotations and 10304 tree trunk object annotations; the validation set contained 994 sky object annotations and 2135 tree trunk object annotations. The forest scene size data sources included 1122 forest images obtained from the network, 556 forest images from the publicly available YOLO dataset, and 5903 self-taken forest scene images.

[0030] The size of the forest scene is determined based on the detection results of the sky and tree trunks. If tree trunks are detected, it is determined to be a small scene, including two cases: only tree trunks are detected and both sky and tree trunks are detected. If only sky is detected and no tree trunks are detected, it is determined to be a large scene. If neither sky nor tree trunks are detected, it is determined to be a medium scene.

[0031] Figure 3 This is an example image of a small scene where only tree trunks are detected, provided as an embodiment of the present invention. The small scene corresponds to the observation situation when the drone is flying at low altitude and the camera is facing the direction of the drone's movement. The image mainly includes close-up forest structures, and tree trunk areas with obvious vertical textures and rough surface features can be observed. Figure 3 Tree trunks were detected but the sky was not, so the forest scene size was determined to be a small scene.

[0032] Figure 4 This is an example image of a small scene where the sky and tree trunks are detected simultaneously, provided as an embodiment of the present invention. Figure 4 The forest fire scene in the image includes both the sky area and tree trunk areas with distinct structural features. The presence of tree trunks in the detection results indicates the forest scene is a small-scale scene.

[0033] Figure 5 This is an example image of a large-scale scene provided in an embodiment of the present invention. A large-scale scene corresponds to the observation situation where a drone flies over the forest canopy with the camera facing the direction of the drone's movement. The image mainly includes a large area of ​​sky and distant forest landscape, with less close-up forest structure. When the observation distance is long, vegetation detail information is relatively reduced. When the detection result only includes the sky and not the tree trunks, the forest scene size is determined to be a large-scale scene.

[0034] Figure 6This is an example image of a mid-scene provided in an embodiment of the present invention. A mid-scene corresponds to an observation situation where the drone flies at a height close to the forest canopy and the camera is facing the ground. The image mainly includes the forest canopy area and the ground surface area. Due to the influence of vegetation shading and the shooting direction, there are usually no obvious tree trunk areas and sky areas in the image; it mainly presents vegetation distribution characteristics and canopy texture information. When neither the sky nor tree trunks are detected, the forest scene size is determined to be a mid-scene.

[0035] After determining the size of the forest scene, one-hot vector encoding is performed. The scene encoding vector corresponding to the small scene is: The scene encoding vector corresponding to the scene in the middle is: The scene encoding vector corresponding to a large scene is: All three encodings are 3D scene encoding vectors, which are used as feature data corresponding to the forest scene size and are then processed as forest scene size feature modulation.

[0036] For step 200, flame semantic segmentation is performed on the RGB image of the forest fire scene. This embodiment uses... Figure 7 The improved U-Net network is shown. The improved U-Net network sets the downsampling number to 3 times, which is one less continuous downsampling compared to the original U-Net with 4 downsampling, in order to reduce the loss of small target and edge information caused by continuous downsampling; in the encoding stage, 32, 64, 128 and 256 feature channels are used to extract flame features step by step.

[0037] Figure 7 This is a schematic diagram of the overall structure of the improved U-Net network provided in this embodiment of the invention. An RGB image of a forest fire scene, after being resized, is used as the three-channel input. To maintain the image size relationship before and after in this embodiment, Figure 7 Input dimensions in Setup. The encoding section includes, in sequence, a Conv_Block with 3 input channels and 32 output channels, a Conv_Block with 32 input channels and 64 output channels, a Conv_Block with 64 input channels and 128 output channels, and a Conv_Block with 128 input channels and 256 output channels. Downsampling is applied between adjacent Conv_Blocks.

[0038] Figure 7 The Downsample function uses Maxpool2d, and the kernel size is [size missing]. Step size is Conv_Block internally includes two levels of Conv2d processing, with each Conv2d convolution kernel size being [size missing]. Step size is The fill parameter is Each Conv2d level is followed by BatchNorm2d and LeakyReLU in sequence.

[0039] Figure 7 The decoding part uses ConvTranspose2d for feature upsampling, and the number of channels of ConvTranspose2d is set to the number of input channels. The kernel size is Step size is The features generated by upsampling are concatenated with the shallow spatial features at the corresponding scale in the encoding stage, and then processed by Conv_Block. In the decoding stage, Conv_Blocks with 256 input channels and 128 output channels, 128 input channels and 64 output channels, and 64 input channels and 32 output channels are generated sequentially. Finally, a single-channel flame prediction map is formed using Conv2d with 32 input channels, 1 output channel, and a kernel size of 1, and pixel-level segmentation results are obtained using the Sigmoid function.

[0040] In this embodiment, the segmentation threshold is set to... The pixel-level segmentation result is greater than... When the corresponding pixel is identified as a flame pixel, it is assigned a value. Pixel-level segmentation results are no larger than When the corresponding pixel is identified as background, it is assigned a value. This forms a binary image of the flame mask.

[0041] Count the number of flame pixels in the binary image of the flame mask, and calculate the flame area ratio R according to the following formula: (1) In the formula, Indicates the percentage of flame area. This indicates the number of predicted flame pixels. This indicates the total number of pixels in the image.

[0042] Flame semantic segmentation is optimized using the Binary Cross-Entropy (BCE) loss function, calculated as follows: (2) In the formula, Represents the binary cross-entropy loss. Indicates the first in the real mask The pixel value of each pixel. The model predicts the first Each flame pixel value This represents the total number of pixels. The binary cross-entropy loss is used to constrain the prediction results for flame pixels and background pixels.

[0043] For step 300, a binary mask image based on tree trunk constraints is generated according to the size of the forest scene. When the forest scene size is small, tree trunk detection boxes are obtained based on the tree trunk positions detected by the YOLOv11 object detection network. Using the minimum Y value of the tree trunk detection box as a threshold, the regions in the flame binary mask image Mask below the corresponding Y value are set to 0, while the flame regions located above the upper boundary of the tree trunk detection box are retained, forming a binary mask image Mask_trunk based on tree trunk constraints.

[0044] When the forest scene size is medium or large, and no tree trunks are detected in the corresponding forest scene, the flame mask binary map Mask is directly used as the mask binary map Mask_trunk based on the tree trunk constraint, so that all three forest scenes form Mask_trunks for Sobel gradient feature calculation.

[0045] For step 400, Sobel gradient features are calculated on the mask binary image Mask_trunk based on the trunk constraint, resulting in the horizontal gradient map Sobelx, the vertical gradient map Sobely, and the gradient magnitude map Sobelmag. Sobelx is used to represent the pixel changes in the horizontal direction, Sobely is used to represent the pixel changes in the vertical direction, and Sobelmag is used to represent the overall gradient magnitude. The three gradient maps together reflect the changes in flame shape and flame boundary structure.

[0046] The Sobelx, Sobely, Sobelmag, flame mask binary image Mask, and trunk-constrained mask binary image Mask_trunk are concatenated along the channel dimension to form a shape of size [size missing]. The five-channel image. The five-channel image enters... Figure 8 The flame structure features shown are extracted using a convolutional neural network.

[0047] Figure 8 A schematic diagram of the convolutional neural network structure for flame structure feature extraction provided in an embodiment of the present invention. Figure 8 The input labels in the diagram correspond to a five-channel image; C represents the number of channels, K represents the kernel size, S represents the stride, and P represents the padding parameter.

[0048] The parameters of the first convolutional layer are: .

[0049] The parameters of the second convolutional layer are: .

[0050] The parameters of the third convolutional layer are: .

[0051] The parameters of the fourth convolutional layer are .

[0052] The parameters of the fifth convolutional layer are .

[0053] The spatial dimensions corresponding to the five levels of convolution are as follows: Compress to The size of the last-level convolutional feature is After being flattened, it forms Dimensional features; 3D feature mapping to Dimension, activated by ReLU and After processing, map to Dimension, then activated by ReLU and Processing, ultimately forming 3D flame visual characteristics, Figure 8 The value is labeled Visual_feat, and in this embodiment, it is denoted as visual_feat. The above spatial size compression is completed by five Conv2D operations with a step size of 2, without adding any other processing that changes the spatial size.

[0054] For step 500, input the 3D scene encoding vector. Figure 9 The forest scene shown is a size feature modulation neural network.

[0055] Figure 9 This is a schematic diagram of a neural network structure for modulating forest scene size features, provided in an embodiment of the present invention. Figure 9 The input dimension is The 3D scene encoding vector is sequentially mapped to Dimensionality. 16-dimensional, 32-dimensional, and 64-dimensional features were activated by ReLU and... Processing; the 128-dimensional features are sequentially processed using ReLU, Tanh, and... deal with.

[0056] Formed through the above processing The 128-dimensional scene feature vector is split into two parts along the feature dimension from the middle. dimensional vectors, denoted as follows: and . This represents the feature scaling factor formed based on the scene encoding vector. This represents the feature offset formed based on the scene encoding vector.

[0057] The 64-dimensional flame visual feature (visual_feat) is modulated using A1 and B1, and the calculation formula is as follows: (3) In the formula, This represents the modulation features based on the scene coding vector. Indicates the feature scaling factor. Indicates the feature offset. Represents the 64-dimensional visual features of a flame. This indicates element-wise multiplication. , and All data are 64-dimensional.

[0058] For step 600, the flame area ratio R is input in a one-dimensional form. Figure 10 The flame area ratio feature modulated neural network is shown.

[0059] Figure 10 A schematic diagram of the flame area ratio feature modulation neural network structure provided in an embodiment of the present invention. Figure 10 The input dimension is The flame area percentage is mapped sequentially to Dimensionality. 16-dimensional, 32-dimensional, and 64-dimensional features were activated by ReLU and... Processing; the 128-dimensional features are sequentially processed using ReLU, Tanh, and... deal with.

[0060] Formed through the above processing A 128-dimensional feature vector is split into two parts along the feature dimension from the middle. dimensional vectors, denoted as follows: and . This represents the feature scaling factor based on the proportion of flame area. This represents the feature offset based on the proportion of flame area.

[0061] The 64-dimensional flame visual feature (visual_feat) is modulated using A2 and B2, and the calculation formula is as follows: (4) In the formula, This indicates modulation characteristics based on the proportion of flame area. Indicates the feature scaling factor. Indicates the feature offset. This represents the 64-dimensional visual features of a flame. , and All data are 64-dimensional.

[0062] For step 700, the 64-dimensional modulation features and 64-dimensional modulation features By splicing along the feature dimension, a sequence is formed. 128-dimensional fusion features. Entering 128-dimensional fusion features. Figure 11 The fire size classification neural network shown.

[0063] Figure 11 This is a schematic diagram of a neural network structure for classifying fire size according to an embodiment of the present invention. Figure 11 The input dimension is The 128-dimensional fusion features are first mapped to... Dimension, after ReLU activation and Processing; 64-dimensional feature mapping to Dimension, then after ReLU activation and Processing; 32-dimensional features are ultimately formed through linear mapping. An 11-dimensional classification vector is used to determine the fire size category.

[0064] This embodiment sets 11 categories of fire size, including small scene with no fire, small scene with small fire, small scene with medium fire, small scene with large fire, medium scene with no fire, medium scene with small fire, medium scene with medium fire, medium scene with large fire, large scene with no fire, large scene with medium fire, and large scene with large fire.

[0065] The fire size label is set based on the size of the forest scene and the spatial distribution characteristics of the flames. In small scenes, tree trunks can be detected, and the fire size is mainly distinguished based on the position of the flames relative to the tree trunks and the spatial distribution of the flames. Figure 12 This is a small-scale, fireless example image provided in an embodiment of the present invention; there is no flame area in the image. Figure 13 The small-scene fire example diagram provided in the embodiment of the present invention shows that the flame area is located below the upper boundary of the detected tree trunk detection box, and the burning area is mainly located in the lower layer of the forest. Figure 14 The example diagram of fire in a small scene provided in the embodiment of the present invention shows that part of the flame area extends above the upper boundary of the tree trunk detection frame and includes multiple scattered flame areas. Figure 15 The image provided in this embodiment of the invention is an example of a small-scale fire. The image includes one or more large continuous flame areas, corresponding to a relatively concentrated and intense combustion state.

[0066] In medium-sized scenes, it is usually impossible to detect obvious sky or tree trunk areas; the size of the fire is mainly distinguished based on the range of flame coverage and spatial distribution. Figure 16 This is an example image of a fireless scene provided in an embodiment of the present invention; there is no flame area in the image. Figure 17 This is an example image of a small fire in a mid-scene provided in an embodiment of the present invention. The image includes sparsely distributed small-area flame regions. Figure 18 The example diagram of medium fire in a medium scene provided in the embodiment of the present invention shows that, compared with small fire in a medium scene, the number of flame areas is increased and the area of ​​a single flame area is increased. Figure 19 This is an example image of a large fire in a medium-scale scene provided in an embodiment of the present invention. The image shows a continuous area of ​​flames covering a large area of ​​forest canopy.

[0067] Large-scale scenes correspond to long-distance observation situations, where the flames that can be observed are usually already up to the tree canopy height. This embodiment sets three categories for large-scale scenes: no fire, medium fire, and large fire. Figure 20 This is a large-scale fireless example image provided by an embodiment of the present invention; there is no flame area in the image. Figure 21 This is an example image of a large-scale fire scene provided in an embodiment of the present invention. The image includes multiple flame areas that are separated from each other and have limited spatial coverage. Figure 22 The image provided in this embodiment of the invention shows a large-scale fire scene, which is a continuous area of ​​flame with a high spatial occupancy.

[0068] The 11 types of labels mentioned above are used for category labeling of the fire size classification dataset. The actual fire size is determined by two-way modulation features formed by the scene encoding vector, the proportion of flame area, and the visual features of the flame, as well as the fused features. The manual label descriptions are used for organizing the training sample categories.

[0069] The fire size discrimination network is optimized using the Cross Entropy (CE) loss function, calculated as follows: (5) In the formula, Represents cross-entropy loss, This represents the total number of categories, in this embodiment. ; Indicates the true distribution at the th The probability of a class is determined by one-hot encoding of the true class. The model represents the first time. The predicted probability of a class.

[0070] In the fire size classification dataset, each category contains 500 training images, 125 validation images, and 125 test images. The training set for the 11 categories contains a total of 5500 images, the validation set contains 1375 images, and the test set contains 1375 images.

[0071] In the training parameters of the fire size discrimination network, the epoch is set to... The batch size is set to The learning rate is set to Dropout is set to It uses the Adam optimizer.

[0072] Figure 23 The accuracy curve of the fire size discrimination network provided in the embodiments of the present invention. Figure 23 The x-axis represents Epoch, and the curves represent Training Acc and Validation Acc, respectively. Accuracy increases rapidly in the early stages of training, gradually stabilizing with each training epoch, and reaching a stable state after approximately 40 epochs.

[0073] Figure 24 The loss curve of the fire size discrimination network provided in the embodiment of the present invention. Figure 24 The x-axis represents Epoch, and the curves represent Training Loss and Val Loss, respectively. The loss is higher in the early stages of training, gradually decreases as training progresses, and generally stabilizes after about 40 epochs.

[0074] Under the above training parameters, the training set loss (Train Loss) is: The validation set loss Val Loss is The training set accuracy, Train Acc, is... The validation set accuracy Val Acc is 1. The validation set accuracy, converted to a percentage, is approximately .

[0075] Under the experimental conditions given in this embodiment, the precision of the improved lightweight U-Net flame semantic segmentation network is: IoU is The above values ​​are used to record the experimental results of the corresponding network under the data and training parameters of this embodiment.

[0076] To observe the model's focus on the flame region during flame structure feature extraction, Grad-CAM was used for visualization analysis. Flame structure feature extraction utilized a flame mask binary map Mask generated by U-Net, a mask binary map Mask_trunk based on trunk constraints, and Sobelx, Sobely, and Sobelmag calculated based on Mask_trunk.

[0077] Figure 25 The Grad-CAM analysis provided in this embodiment of the invention uses original forest fire images. Figure 26 This is a visualization diagram of the Grad-CAM multi-module fusion deep neural network provided in an embodiment of the present invention. Figure 26The thermal distribution in the image is used to display the region of interest of the model in the RGB image of the forest fire scene, and to observe the model response after Mask, Mask_trunk and three Sobel gradient features are jointly involved in the extraction of flame structure features.

[0078] In another embodiment, a forest fire size discrimination system based on a multi-module fusion deep neural network, corresponding to the above method, is provided. The forest fire size discrimination system includes an image acquisition module, a forest scene size discrimination module, a flame semantic segmentation module, a flame structure feature extraction module, a scene size feature modulation parameter generation module, a flame area proportion feature modulation parameter generation module, a feature fusion and fire size classification module, and a result display and early warning module.

[0079] The image acquisition module includes a visible light RGB image sensor mounted on a drone or fixed monitoring equipment, used for real-time acquisition of images with a resolution of [resolution missing]. The system generates pixel-perfect images of forest fire scenes and transmits the image data to a data processing unit via wireless or wired communication.

[0080] The forest scene size determination module has a built-in microprocessor or GPU, configured with a YOLOv11 target detection network model and scene classification logic. It receives image data generated by the image acquisition module, detects two types of environmental targets: sky and tree trunks, determines the size of the forest scene as small, medium, or large based on the detection results, and forms a three-dimensional scene encoding vector.

[0081] The flame semantic segmentation module establishes a data connection with the image acquisition module, configures an improved lightweight U-Net semantic segmentation network, performs pixel-level flame region segmentation on forest fire scene images, forms a flame mask binary map, and calculates the flame area ratio.

[0082] The flame structure feature extraction module establishes data connections with the flame semantic segmentation module and the forest scene size discrimination module. It incorporates tree trunk constraint mask generation logic, Sobel gradient calculation logic, and a lightweight convolutional neural network. It receives the binary image of the flame mask and the coordinates of the tree trunk detection box to form 64-dimensional flame visual features.

[0083] The scene size feature modulation parameter generation module establishes a data connection with the forest scene size discrimination module, and incorporates a multi-layer fully connected neural network with the following network structure: The 3D scene encoding vector is mapped to a 128-dimensional scene feature vector, and the 128-dimensional scene feature vector is split into a 64-dimensional feature scaling factor and a 64-dimensional feature offset.

[0084] The flame area proportion feature modulation parameter generation module establishes a data connection with the flame semantic segmentation module, and incorporates a feature modulation parameter generation neural network with the following structure: The one-dimensional flame area ratio is mapped to a 128-dimensional feature vector, and the 128-dimensional feature vector is split into a 64-dimensional feature scaling factor and a 64-dimensional feature offset.

[0085] The feature fusion and fire size classification module establishes data connections with the two feature modulation parameter generation modules and the flame structure feature extraction module, respectively. Two 64-dimensional modulation features are formed according to equations (3) and (4). These two 64-dimensional modulation features are then concatenated along the feature dimensions to form a 128-dimensional fusion feature. Finally, the network structure is used to... A fire size classification neural network determines the fire size category.

[0086] The results show that the early warning module, feature fusion module, and fire size classification module establish data connection, send the determined fire size category to the ground station or monitoring center, and provide visualization and early warning prompts.

[0087] The image acquisition module constitutes the front-end image acquisition part. The forest scene size discrimination module and the flame semantic segmentation module process forest fire scene images respectively. The flame structure feature extraction module, the scene size feature modulation parameter generation module, and the flame area ratio feature modulation parameter generation module complete the corresponding feature processing. The feature fusion and fire size classification module completes two-way feature modulation, feature stitching, and fire size classification. The result display and early warning module completes the terminal display and early warning prompts for the fire size category.

[0088] This embodiment uses forest scene size discrimination, flame semantic segmentation, tree trunk constraint flame structure feature extraction, and two-way feature modulation based on scene encoding vector and flame area ratio to enable forest scene size, flame area ratio, and flame visual features to jointly participate in the determination of fire size category. Tree trunk constraint mask binary map and Sobel gradient features are used to increase the spatial structure information of the flame area above the upper boundary of the tree trunk detection box. The two-way modulated features are spliced ​​together to form a 128-dimensional fusion feature, which is used to distinguish the fire size category under different forest scenes.

[0089] The beneficial effects of this embodiment are as follows: 1) This embodiment determines the size of a forest scene by detecting the sky and tree trunks in the RGB image of a forest fire scene, and encodes the forest scene size into a scene encoding vector, so that information corresponding to different forest scenes can be used to determine the size of the fire. Compared with the processing method that only classifies based on the image features of the flame itself, the forest scene size information participates in the modulation of the visual features of the flame, reducing the impact of differences in the appearance of the flame caused by different observation scenes on the determination of the size of the fire.

[0090] 2) This embodiment uses an improved U-Net to perform flame semantic segmentation on RGB images of forest fire scenes. The downsampling times are set to 3 times, using 32, 64, 128, and 256 feature channels. Skip connections are used to fuse shallow spatial features and deep semantic features at corresponding scales to preserve spatial information of small flame areas and flame edges. Under the experimental conditions of this embodiment, the flame semantic segmentation network achieves a precision of 88.86% and an IoU of 69.50%, indicating that the above network structure can effectively segment flame areas in forest fire images.

[0091] 3) This embodiment further generates a mask binary map based on tree trunk constraints according to the size of the forest scene, and extracts flame visual features by combining the horizontal gradient map, vertical gradient map, and gradient magnitude map, so that the position of the flame relative to the tree trunk, the distribution of the flame area, and the structure of the flame edge jointly participate in the feature representation. On this basis, the scene encoding vector and the flame area ratio respectively generate feature scaling coefficients and feature offsets, and modulate the same flame visual feature. The two modulated features are spliced ​​along the feature dimension to form a 128-dimensional fusion feature, so that the forest scene size, the flame area ratio, and the flame visual features jointly participate in the determination of the fire size category, increasing the feature differentiation information between different fire size categories.

[0092] 4) This embodiment employs a flame semantic segmentation network with 3-fold downsampling, 64-dimensional flame visual features, and a fire size classification neural network with a progression from 128 to 64 to 32 to 11, maintaining a relatively compact network structure. Under the specified forest fire size classification dataset and training parameters, the training set accuracy is 0.98964, the validation set accuracy is 0.99564, and the validation set accuracy is approximately 99.56%, indicating that the technical solution comprised of forest scene size discrimination, flame semantic segmentation, tree trunk-constrained flame structure feature extraction, and two-way feature modulation can distinguish between 11 fire size categories.

[0093] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0094] Specific examples have been used to illustrate the principles and implementation methods of this invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this invention. Furthermore, those skilled in the art will recognize that, based on the ideas of this invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this invention.

Claims

1. A method for determining the magnitude of forest fires based on a multi-module fusion deep neural network, characterized in that, include: The sky and tree trunks in the RGB image of a forest fire scene are detected. The size of the forest scene is determined based on the detection results. If tree trunks are detected, it is determined to be a small scene. If only the sky is detected, it is determined to be a large scene. If neither the sky nor tree trunks are detected, it is determined to be a medium scene. The size of the forest scene is then encoded to obtain a scene encoding vector. Perform flame semantic segmentation on the RGB image to obtain a flame mask binary image and flame area ratio; A mask binary map based on tree trunk constraints is generated according to the size of the forest scene. In the small scene, the flame mask binary map is cropped with the minimum Y value of the tree trunk detection box corresponding to the detected tree trunk as a threshold, and the flame area above the tree trunk is retained. In the medium or large scene, the flame mask binary image is used as the mask binary image based on the trunk constraint; Sobel gradient features are calculated on the binary mask image based on the trunk constraint. The obtained Sobel gradient features are then concatenated with the binary mask image of the flame and the binary mask image based on the trunk constraint to extract the visual features of the flame. Based on the scene encoding vector, feature scaling coefficients and feature offsets are generated. The flame visual features are modulated using the feature scaling coefficients and feature offsets to obtain modulated features based on the scene encoding vector. Based on the flame area ratio, a feature scaling factor and a feature offset are generated. The flame visual features are modulated using the feature scaling factor and the feature offset to obtain modulated features based on the flame area ratio. By splicing the modulation features based on the scene encoding vector and the modulation features based on the flame area ratio, a fused feature is obtained, and the fire size category is determined based on the fused feature.

2. The forest fire intensity determination method based on a multi-module fusion deep neural network according to claim 1, characterized in that, The sky and tree trunks in the RGB image of a forest fire scene are detected. Based on the detection results, the size of the forest scene is determined, and the forest scene size is encoded to obtain a scene encoding vector, including: The YOLOv11 object detection network was used to detect the sky and tree trunks in the RGB image of the forest fire scene. The size of the forest scene was determined based on the detection results of the sky and tree trunks; The forest scene size is one-hot vector encoded, the small scene is encoded as [1 0 0], the medium scene is encoded as [0 1 0], and the large scene is encoded as [0 0 1], to obtain the scene encoding vector.

3. The forest fire intensity determination method based on a multi-module fusion deep neural network according to claim 1, characterized in that, Perform flame semantic segmentation on the RGB image to obtain a flame mask binary image and flame area proportion, including: The improved U-Net was used to perform flame semantic segmentation on the RGB images of the forest fire scene. The downsampling time was set to 3 times, and flame features were extracted step by step using 32, 64, 128 and 256 feature channels. In the decoding stage, transposed convolution is used for feature upsampling, and shallow spatial features and deep semantic features of the corresponding scale are fused through skip connections. The fused features are mapped to a single-channel flame prediction map by 1×1 convolution, and pixel-level segmentation results are obtained by using the Sigmoid function. The pixel-level segmentation results are then converted into the flame mask binary map according to the segmentation threshold. The number of flame pixels in the binary image of the flame mask is counted, and the ratio of the number of flame pixels to the total number of pixels in the RGB image of the forest fire scene is determined as the flame area ratio.

4. The forest fire intensity determination method based on a multi-module fusion deep neural network according to claim 1, characterized in that, Generate a mask binary map based on tree trunk constraints according to the size of the forest scene, including: When the size of the forest scene is the small scene, the minimum Y value of the tree trunk detection box corresponding to the detected tree trunk is used as the threshold to crop the flame mask binary image, retaining the flame area above the tree trunk, and thus obtaining the mask binary image based on tree trunk constraints. When the size of the forest scene is the medium scene or the large scene, the flame mask binary image is used as the mask binary image based on the tree trunk constraint.

5. The forest fire intensity determination method based on a multi-module fusion deep neural network according to claim 1, characterized in that, Sobel gradient features are calculated on the trunk-constrained binary mask image. The obtained Sobel gradient features are then concatenated with the flame binary mask image and the trunk-constrained binary mask image to extract flame visual features, including: Sobel gradient feature calculation is performed on the binary mask image based on the trunk constraint to obtain the horizontal gradient map, the vertical gradient map, and the gradient magnitude map; The horizontal gradient map, the vertical gradient map, the gradient magnitude map, the flame mask binary map, and the mask binary map based on the trunk constraint are stitched together to obtain a five-channel image; The five-channel image is input into a lightweight convolutional neural network for feature extraction, resulting in 64-dimensional visual features of the flame.

6. The forest fire intensity determination method based on a multi-module fusion deep neural network according to claim 1, characterized in that, Based on the scene encoding vector, feature scaling factors and feature offsets are generated. The flame visual features are then modulated using the feature scaling factors and feature offsets to obtain modulated features based on the scene encoding vector, including: The scene encoding vector is input into a multi-layer fully connected neural network for feature mapping. The network structure of the multi-layer fully connected neural network is 3→16→32→64→128, resulting in a 128-dimensional scene feature vector. The 128-dimensional scene feature vector is split into two 64-dimensional vectors along the feature dimension, which are used as the feature scaling factor and the feature offset, respectively. The feature scaling factor is multiplied element-wise with the flame visual feature, and the feature obtained from the multiplication is added element-wise with the feature offset to obtain the modulation feature based on the scene coding vector.

7. The forest fire intensity determination method based on a multi-module fusion deep neural network according to claim 1, characterized in that, Based on the flame area ratio, a feature scaling factor and a feature offset are generated. The flame visual features are then modulated using the feature scaling factor and the feature offset to obtain modulated features based on the flame area ratio, including: The flame area ratio is input into the feature modulation parameter generation neural network for feature mapping. The network structure of the feature modulation parameter generation neural network is 1→16→32→64→128, resulting in a 128-dimensional feature vector. The 128-dimensional feature vector is split into two 64-dimensional vectors along the feature dimension, which are used as the feature scaling factor and the feature offset, respectively. The feature scaling factor is multiplied element-wise with the flame visual feature, and the feature obtained by multiplication is added element-wise with the feature offset to obtain the modulation feature based on the flame area ratio.

8. The forest fire intensity determination method based on a multi-module fusion deep neural network according to claim 1, characterized in that, The fusion feature is a 128-dimensional feature vector formed by splicing the modulation feature based on the scene encoding vector and the modulation feature based on the flame area ratio in the feature dimension.

9. The forest fire intensity determination method based on a multi-module fusion deep neural network according to claim 1, characterized in that, The fire size category is determined based on the fusion features, including: The fused features are input into a shallow fully connected neural network for fire size classification. The network structure of the shallow fully connected neural network is 128→64→32→11, resulting in an 11-dimensional classification vector. The fire size category is determined based on the 11-dimensional classification vector.

10. The method for determining the magnitude of a forest fire based on a multi-module fusion deep neural network according to claim 1, characterized in that, The fire size categories include small scene with no fire, small scene with a small fire, small scene with a medium fire, small scene with a large fire, medium scene with no fire, medium scene with a small fire, medium scene with a medium fire, medium scene with a large fire, large scene with no fire, large scene with a medium fire, and large scene with a large fire.

Images