Flame smoke detection method and system based on improved YOLOv8s

Abstract

The invention discloses a flame smoke detection method and system based on improved YOLOv8s, and relates to the technical field of fire recognition, and the method comprises the steps: constructing an up-sampling update layer based on common convolution, dynamic convolution and cavity convolution corresponding to different receptive fields; constructing an aspect ratio consistency penalty according to the aspect ratio difference item based on nonlinear transformation; replacing an up-sampling initial layer in the YOLOv8s network architecture with the up-sampling updating layer, reconstructing a loss function in the YOLOv8s network architecture through aspect ratio consistency penalty and linear interval mapping, and constructing a YOLOv8s basic model; training a YOLOv8s basic model based on the historical fire image data to obtain a flame smoke detection model; and acquiring real-time image data, and acquiring a flame smoke detection result according to the real-time image data and the flame smoke detection model. The method has the beneficial effects that the recognition accuracy and comprehensiveness of extreme forms such as flame and smoke are improved.

Description

Technical Field

[0001] This application relates to the field of fire detection technology, and in particular to a flame and smoke detection method and system based on an improved YOLOv8s. Background Technology

[0002] Initially, fire detection relied on manual patrols. However, this method suffers from significant drawbacks, including low efficiency and poor reliability. Traditional fire alarm systems depend on various contact sensors (such as smoke, temperature, particle, and radiation sensors) to monitor key fire-related parameters in the environment in real time, thereby identifying fire types, locating fire sources, and assessing the damage. However, these sensors are only triggered when preset thresholds are reached, and their coverage, installation location, and environment all present limitations and challenges for their implementation in outdoor spaces. Consequently, fires are often only discovered after they have spread extensively, making fire control and suppression extremely difficult.

[0003] Traditional image processing-based fire detection methods utilize computer vision technology to analyze images or videos, extracting visual features of flames or smoke to determine whether a fire has occurred. This method significantly reduces the time required to detect a fire; however, these methods are not robust because different combustibles produce different flames and smoke. Furthermore, they struggle to achieve high accuracy and robustness when faced with interfering factors such as changes in lighting, obstruction, or background shifts.

[0004] Deep learning-based methods have demonstrated exceptional capabilities in automatically extracting and enhancing learned feature representations. The YOLO series of algorithms is a popular target detection algorithm, renowned for its speed and accuracy. Models such as YOLOv5 and YOLOv8 have introduced numerous proven advanced techniques, achieving significant progress in practical engineering applications. However, flames and smoke are influenced by the overall environment, exhibiting different shapes, sizes, and colors. Fire scenes often contain flames and smoke targets of multiple scales simultaneously. Furthermore, small flames and smoke targets, due to their low pixel ratio and weak feature representation capabilities, are easily confused with complex background environments.

[0005] The patent "A Method for Recognizing Fire Smoke and Flame Images," publication number CN115761627A, published on March 7, 2023, specifically discloses a method for creating a dataset of fire smoke and flame images; improving the YOLOv5s model to obtain a fire recognition model for recognizing fire smoke and flame images; performing data augmentation on the fire smoke and flame images; inputting the data-augmented fire smoke and flame images into the fire recognition model for training to obtain a trained fire recognition model; and inputting the fire smoke and flame images into the trained fire recognition model to obtain the recognition results of the fire smoke and flame images. However, it also suffers from low accuracy in detecting small-sized flames and smoke targets. Summary of the Invention

[0006] This application addresses the problem in existing flame and smoke detection technologies that fail to balance comprehensive large-scale recognition with detailed small-scale recognition. It provides a flame and smoke detection method and system based on an improved YOLOv8s. By fusing ordinary convolution, dynamic convolution, and dilated convolution with different receptive fields, the upsampling process can simultaneously preserve features at different scales, thus maintaining both comprehensive large-scale recognition and detailed small-scale recognition. Furthermore, nonlinear transformation is used to strengthen aspect ratio consistency penalty, enhancing geometric sensitivity and improving the accuracy and comprehensiveness of recognizing extreme forms such as flames and smoke.

[0007] To achieve the aforementioned technical objectives, this application provides a technical solution: a flame and smoke detection method based on an improved YOLOv8s, comprising the following steps: constructing an upsampling update layer based on ordinary convolution, dynamic convolution, and dilated convolution corresponding to different receptive fields; constructing an aspect ratio consistency penalty based on a nonlinear transformation according to the aspect ratio difference term; replacing the upsampling initial layer in the YOLOv8s network architecture with the upsampling update layer, reconstructing the loss function in the YOLOv8s network architecture with the aspect ratio consistency penalty and linear interval mapping, and constructing a YOLOv8s basic model; training the YOLOv8s basic model based on historical fire image data to obtain a flame and smoke detection model; acquiring real-time image data, and obtaining flame and smoke detection results based on the real-time image data and the flame and smoke detection model.

[0008] Furthermore, the construction of the upsampling update layer based on ordinary convolution, dynamic convolution, and dilated convolution corresponding to different receptive fields includes: the ordinary convolution corresponds to the first receptive field, the dynamic convolution corresponds to the second receptive field, and the dilated convolution corresponds to the third receptive field, wherein the first receptive field is smaller than the second receptive field, and the second receptive field is smaller than the third receptive field.

[0009] Furthermore, the construction of the upsampling update layer based on ordinary convolution, dynamic convolution, and dilated convolution corresponding to different receptive fields also includes: constructing a feature extraction fusion layer based on ordinary convolution, dynamic convolution, and dilated convolution corresponding to different receptive fields; constructing a point sampling generator based on a dynamic range factor and / or a static range factor; and constructing an upsampling update layer based on the feature extraction fusion layer and the point sampling generator.

[0010] Furthermore, the construction of aspect ratio consistency penalty based on aspect ratio difference based on nonlinear transformation includes: constructing aspect ratio consistency penalty based on aspect ratio difference using hyperbolic tangent function or logarithmic scale difference.

[0011] Furthermore, the step of training a YOLOv8s base model based on historical fire image data to obtain a flame and smoke detection model includes: performing data preprocessing on the historical fire image data to obtain a flame and smoke training dataset; performing feature extraction at different levels on the flame and smoke training dataset through a backbone network to obtain feature maps at different levels; performing multi-scale extraction of features in the feature maps and performing feature fusion at different levels through a neck network with an upsampling update layer to generate multi-scale feature maps; performing prediction processing on the multi-scale feature maps through a detection head to obtain predicted bounding boxes; and dynamically adjusting the model weights through a loss function to obtain the optimal predicted bounding boxes, thereby completing the model training and obtaining the flame and smoke detection model.

[0012] Furthermore, the process of extracting features from the feature map at multiple scales and fusing features at different levels through the neck network of the upsampling update layer to generate a multi-scale feature map also includes: performing multi-scale extraction and cross-scale splicing of the feature map based on top-down fusion through the upsampling update layer, feature extraction layer, and feature splicing layer to generate an upsampled fused feature map; and performing feature enhancement of the upsampled fused feature map through the feature extraction layer, feature splicing layer, and attention enhancement layer through bottom-up enhancement to generate a multi-scale feature map.

[0013] Furthermore, the attention enhancement layer performs the following: preprocessing of the input feature map based on layer normalization and grouping parameters to obtain a reshaped feature map; parallel output of attention-enhanced feature maps based on spatial feature enhancement and cross-spatial interaction; and fusion of the parallel output attention-enhanced feature maps to generate a multi-scale feature map.

[0014] Furthermore, the step of parallel outputting attention-enhanced feature maps based on spatial feature enhancement and cross-spatial interaction from reshaped feature maps includes: performing parallel decomposition of the reshaped feature maps along the horizontal and vertical directions using global average pooling to obtain horizontal and vertical feature maps; adjusting the dimensional order of the horizontal feature maps to make their shapes consistent with the vertical feature maps, concatenating the horizontal and vertical feature maps based on the height direction, and performing a 1×1 convolution operation to obtain a spatial fusion feature map; cutting the spatial fusion feature map into two tensors, processing the two tensors respectively using the SiLU activation function to obtain a first tensor feature map and a second tensor feature map; performing a reweighting operation and grouping normalization based on the first tensor feature map, the second tensor feature map, and the reshaped feature map to generate a first attention-enhanced feature map; and performing a 3×3 convolution operation on the reshaped feature map to generate a second attention-enhanced feature map.

[0015] Furthermore, the multi-scale feature map generation based on parallel output attention-enhanced feature map fusion includes: obtaining a first attention weight matrix and a second attention weight matrix corresponding to a first attention-enhanced feature map and a second attention-enhanced feature map based on two-dimensional average pooling and reshaping in parallel; obtaining a first multi-scale feature map based on the first attention weight matrix and the second attention weight matrix corresponding to the first attention-enhanced feature map and the second attention-enhanced feature map; obtaining a second multi-scale feature map based on the first multi-scale feature map and the reshaping feature map; obtaining a third multi-scale feature map based on the second multi-scale feature map based on skip connections and depthwise separable convolution; obtaining a fourth multi-scale feature map based on the third multi-scale feature map based on pointwise convolution; and obtaining a multi-scale feature map based on the fourth multi-scale feature map and the input feature map based on residual connections.

[0016] Another technical solution provided in this application is a flame and smoke detection system based on an improved YOLOv8s, used to implement the method described above, comprising: a data acquisition unit for acquiring historical fire image data and real-time image data; a model building unit for constructing an upsampling update layer and a loss function, and training a flame and smoke detection model based on the upsampling update layer, the loss function, and the YOLOv8s network architecture according to the historical fire image data; and a detection feedback unit for obtaining flame and smoke detection results based on the real-time image data and the flame and smoke detection model.

[0017] The beneficial effects of this application are as follows: 1. By fusing ordinary convolution, dynamic convolution, and dilated convolution with different receptive fields, features at different scales can be preserved simultaneously during the upsampling process, improving the comprehensiveness of capturing multi-scale features of flames and smoke. At the same time, dynamic convolution and dilated convolution reduce the computational load in the multi-scale feature extraction process, lowering computational complexity and achieving parallel and lightweight multi-convolution fusion. Furthermore, by strengthening the aspect ratio consistency penalty through nonlinear transformation, geometric sensitivity is enhanced, improving the recognition accuracy of extreme forms such as flames and smoke. Combined with linear interval mapping, the prediction error is mapped to an interval that better matches the characteristics of flames and smoke, optimizing the gradient distribution of the loss value, accelerating model convergence, and improving model training efficiency, making it suitable for applications in emergency and complex scenarios.

[0018] 2. By constructing a parallel dual-path structure through spatial attention and cross-spatial interaction, feature enhancement fusion of local spatial attention and global context modeling is achieved, improving the efficiency of feature representation learning and model training. In practical applications, the model can update weights more promptly based on the latest data, improving the efficiency and accuracy of flame or smoke recognition in fire scenarios. Attached Figure Description

[0019] Figure 1 This is a schematic flowchart of a flame and smoke detection method based on an improved YOLOv8s according to this application. Figure 2 This is a schematic diagram of the upsampling update layer architecture of a flame and smoke detection method based on an improved YOLOv8s according to this application; Figure 3 This is a schematic diagram of the point sampling generator architecture for a flame and smoke detection method based on an improved YOLOv8s, as described in this application. Figure 1 ; Figure 4 This is a schematic diagram of the point sampling generator architecture for a flame and smoke detection method based on an improved YOLOv8s, as described in this application. Figure 2 ; Figure 5 This is a schematic diagram of the attention enhancement layer architecture of a flame and smoke detection method based on an improved YOLOv8s according to this application. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description of this application is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely one preferred embodiment of this application and are only used to explain this application. They do not limit the scope of protection of this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0021] like Figure 1As shown in the first embodiment of this application, a flame and smoke detection method based on improved YOLOv8s includes the following steps: An upsampling update layer is constructed based on ordinary convolution, dynamic convolution, and dilated convolution corresponding to different receptive fields; A width-to-height consistency penalty is constructed based on the difference in width-to-height ratio using nonlinear transformation; The upsampling initial layer in the YOLOv8s network architecture is replaced by the upsampling update layer. The loss function in the YOLOv8s network architecture is reconstructed with aspect ratio consistency penalty and linear interval mapping to build the YOLOv8s basic model. A YOLOv8s base model was trained based on historical fire image data to obtain a flame and smoke detection model. Acquire real-time image data, and obtain flame and smoke detection results based on the real-time image data and the flame and smoke detection model.

[0022] In this embodiment, by fusing ordinary convolution, dynamic convolution, and dilated convolution with different receptive fields, features at different scales can be preserved simultaneously during upsampling, improving the comprehensiveness of multi-scale feature capture of flames and smoke. Simultaneously, dynamic and dilated convolutions reduce the computational load during multi-scale feature extraction, lowering computational complexity and achieving parallel and lightweight multi-convolution fusion. Furthermore, nonlinear transformations are used to strengthen aspect ratio consistency penalties, enhancing geometric sensitivity and improving the accuracy of identifying extreme forms such as flames and smoke. Combined with linear interval mapping, the prediction error is mapped to an interval that better reflects the characteristics of flames and smoke, optimizing the gradient distribution of the loss value, accelerating model convergence, and improving model training efficiency, making it suitable for applications in emergency and complex scenarios.

[0023] Specifically, ordinary convolution corresponds to the first receptive field, dynamic convolution corresponds to the second receptive field, and dilated convolution corresponds to the third receptive field. The first receptive field is smaller than the second receptive field, and the second receptive field is smaller than the third receptive field.

[0024] Ordinary convolutions extract features from small receptive fields, dynamic convolutions extract features from medium receptive fields, and dilated convolutions extract features from large receptive fields. By leveraging the complementary characteristics of different convolution methods, ordinary convolutions achieve lightweight and efficient local detail extraction with fixed parameters; dynamic convolutions dynamically match key structures of medium-sized features with adaptive parameters; and dilated convolutions expand the receptive field with dilation, improving global extraction efficiency and reducing the computational cost of large-scale extraction. This ensures that features of different scales remain physically isolated during the extraction stage, reducing cross-scale interference. Thus, the synergy of multiple convolution combinations achieves a balance between globality and isolation in multi-scale feature extraction, handling features from details to the whole while avoiding scale confusion during global convolution, thereby improving the accuracy of feature extraction.

[0025] like Figure 2 As shown, the upsampling update layer is constructed based on ordinary convolution, dynamic convolution, and dilated convolution corresponding to different receptive fields, including: An upsampling update layer is constructed based on 3x3 ordinary convolution, 5x5 dynamic convolution, and 7x7 dilated convolution.

[0026] Traditional multi-scale fusion, using fixed 3×3 or 5×5 convolutions, suffers from computational redundancy, uneven weight distribution, and the inability to adapt to the differences in different regions or samples due to all inputs sharing the same convolutional kernel parameters. In the upsampling update layer (MS-DySample module), the 5×5 dynamic convolution provides a moderate receptive field, capturing local-to-global transition features. Furthermore, the kernel weights are dynamically generated based on the input features, avoiding the limitations of small convolutional kernels and the higher benefits of dynamic adjustment associated with large convolutional kernels due to their numerous parameters and wide range of receptive field variations. Dilated convolutions can significantly expand the receptive field while maintaining high-resolution feature maps; a 7×7 dilated convolution requires only 49 parameters with a dilation rate of 2, far more efficient than directly using a 15×15 convolution. Therefore, a combination of 3x3 ordinary convolutions, 5x5 dynamic convolutions, and 7x7 dilated convolutions achieves an optimal balance between computational efficiency, receptive field flexibility, and contextual coverage in multi-scale feature extraction.

[0027] The construction of upsampling update layers based on ordinary convolutions, dynamic convolutions, and dilated convolutions corresponding to different receptive fields also includes: A feature extraction and fusion layer is constructed based on ordinary convolution, dynamic convolution, and dilated convolution corresponding to different receptive fields; Construct a point sampling generator based on dynamic range factors and / or static range factors; An upsampling update layer is constructed based on a feature extraction fusion layer and a point sampling generator.

[0028] When the content and structure of the input feature map vary greatly, it is necessary to adjust the offset to suit different regions. Therefore, the offset calculation is constructed based on the dynamic range factor, and the offset range of the sampling position is adaptively adjusted according to the features to improve the detection accuracy.

[0029] When the content and structure of the input feature map are relatively stable, the static range factor can be used to avoid overlapping sampling positions and confusion of sampling boundaries, thereby improving sampling efficiency and stability.

[0030] In this embodiment, a point sampling generator is constructed based on dynamic range factor and static range factor. The dynamic range factor adjusts the offset range according to the complexity of local features to adapt to the spatial changes of complex regions, while the static range factor limits the maximum threshold of the offset to avoid the dynamic range factor from over-adapting to local features, reduce the probability of offset error, ensure the stability of the overall sampling process, and improve the accuracy of multi-scale feature extraction.

[0031] Based on nonlinear transformation, a width-to-height aspect ratio consistency penalty is constructed according to the aspect ratio difference, including: Aspect ratio consistency penalty is constructed based on the hyperbolic tangent function or logarithmic scale difference according to the aspect ratio difference.

[0032] By introducing nonlinear transformations, the model's sensitivity to extreme proportions of differences is enhanced, thereby adapting to the recognition of irregularly shaped flames and smoke.

[0033] Furthermore, the upsampling initial layer in the YOLOv8s network architecture is replaced by the upsampling update layer. The loss function in the YOLOv8s network architecture is reconstructed with aspect ratio consistency penalty and linear interval mapping to build the YOLOv8s basic model.

[0034] In this embodiment, the YOLOv8s basic model includes a backbone network, a neck network based on an upsampling update layer, a detection head, and a loss function.

[0035] The backbone network includes a C2f module, a convolutional module, and an SPPF module. The neck network based on the upsampling update layer includes an upsampling update layer, a C2f module, a convolutional module, and a multi-scale attention module.

[0036] Furthermore, a YOLOv8s base model is trained based on historical fire image data to obtain a flame and smoke detection model, including: Perform data preprocessing on historical fire image data to obtain a flame and smoke training dataset; By using a backbone network to perform feature extraction at different levels on the flame and smoke training dataset, feature maps at different levels are obtained. The neck network of the upsampling update layer extracts features from the feature map at multiple scales and fuses features at different levels to generate a multi-scale feature map. The detection head performs prediction processing on multi-scale feature maps to obtain predicted bounding boxes; By dynamically adjusting the model weights using a loss function, the optimal predicted bounding box is obtained, thus completing the model training and acquiring a flame and smoke detection model.

[0037] Specifically, data preprocessing is performed on historical fire image data to obtain a flame and smoke training dataset, including: Historical fire image data were filtered based on similarity thresholds and occlusion ratio thresholds to obtain a training dataset of flames and smoke.

[0038] In this embodiment, more than 300 fire video clips are acquired. These clips include different geographical distributions, scene distributions, lighting conditions, backgrounds, combustible materials, combustion processes and states, flame-like objects, and both partially visible and partially obscured elements, to cover all possible fire scenarios. Each fire video clip is sampled at 20-frame intervals to ensure sufficient image data, and the resulting historical fire image data is saved in JPG format. It should be noted that in practical applications, the number of fire video clips acquired and the frame sampling interval can be set according to actual needs.

[0039] Filtering historical fire image data based on similarity thresholds and occlusion ratio thresholds includes: If the similarity between any two historical fire images is greater than the similarity threshold, then one of the historical fire images will be removed. If the occlusion ratio of any historical fire image data exceeds the occlusion ratio threshold, then that historical fire image data will be filtered out.

[0040] Duplicate Cleaner 5 can be used to filter historical fire image data based on a similarity threshold, eliminating highly similar redundant images as much as possible, thereby improving the diversity of the training dataset and the generalization ability of the model training. In this embodiment, the similarity threshold is 90%.

[0041] It should be noted that during the screening of historical fire image data, images that do not contain flames and smoke also need to be removed to avoid interference.

[0042] Learning the features of occluded images can enhance the model's generalization ability, but excessive occlusion can lead to the loss of key target information and the introduction of interfering information, thereby weakening the model's robustness. Therefore, an occlusion ratio threshold is set to filter and discard some severely occluded data and retain moderately occluded samples. In this embodiment, the occlusion ratio threshold is 60%.

[0043] By using a backbone network to perform feature extraction at different levels on the flame and smoke training dataset, feature maps corresponding to different resolutions are obtained.

[0044] At this point, the neck network of the upsampling update layer extracts features from the feature map at multiple scales and performs feature fusion at different levels to generate a multi-scale feature map, which also includes: Based on top-down fusion, multi-scale extraction and cross-scale stitching of feature maps are performed through an upsampling update layer, a feature extraction layer, and a feature stitching layer to generate an upsampled fused feature map. By performing feature enhancement through a bottom-up approach, upsampling and fusion feature maps are performed through a feature extraction layer, a feature concatenation layer, and an attention enhancement layer to generate multi-scale feature maps.

[0045] In this embodiment, the backbone network includes a first convolutional module, a second convolutional module, a first feature extraction module, a third convolutional module, a fourth convolutional module, a fifth convolutional module, a fourth feature extraction module, and a pyramid pooling module, all connected sequentially. The first feature extraction module outputs a first feature map, the second feature extraction module outputs a second feature map, the third feature extraction module outputs a third feature map, and the pyramid pooling module outputs a fourth feature map. The resolution of the first, second, third, and fourth feature maps decreases sequentially.

[0046] In the top-down fusion process, the upsampling update layer outputs a first upsampled feature map based on the fourth feature map, and the feature concatenation layer generates a first upsampled fused feature map based on the first upsampled feature map and the third feature map. The feature extraction layer extracts features from the first upsampled fused feature map, and the extracted fifth feature map is input to the upsampling update layer to generate a second upsampled feature map. The feature concatenation layer generates a second upsampled fused feature map based on the second upsampled feature map and the second feature map. The feature extraction layer extracts features from the second upsampled fused feature map, and the extracted sixth feature map is input to the upsampling update layer to generate a third upsampled feature map. The feature concatenation layer generates a third upsampled fused feature map based on the third upsampled feature map and the first feature map.

[0047] In the bottom-up enhancement process, the feature extraction layer generates the seventh feature map based on the third upsampled fused feature map, and the attention enhancement layer outputs the first multi-scale feature map based on the seventh feature map; the feature stitching map fuses the seventh feature map and the sixth feature map, and extracts the feature map through the feature extraction layer to generate the eighth feature map, and the attention enhancement layer outputs the second multi-scale feature map based on the eighth feature map; the feature stitching map fuses the eighth feature map and the fifth feature map, and extracts the feature map through the feature extraction layer to generate the ninth feature map, and the attention enhancement layer outputs the third multi-scale feature map based on the ninth feature map; the feature stitching map fuses the ninth feature map and the fourth feature map, and extracts the feature map through the feature extraction layer to generate the tenth feature map, and the attention enhancement layer outputs the fourth multi-scale feature map based on the tenth feature map.

[0048] It should be noted that the above process is only one feasible implementation. In other embodiments, upsampling update layers, feature concatenation layers, feature extraction layers, and attention enhancement layers are set according to the resolution level of the backbone network. Specifically, the upsampling update layer is an MS-DySample module, the feature extraction layer and feature extraction module are C2f modules, the attention enhancement layer is an MSFSA module, and the pyramid pooling module is an SPPF module. During the bottom-up enhancement process, the feature map output by the feature extraction layer also needs to undergo channel adjustment by a convolution module to facilitate feature concatenation.

[0049] Compared to traditional bilinear interpolation or transposed convolution, the upsampling update layer dynamically generates sampling kernels to adapt to the input feature content, avoiding grid-like artifacts ("checkerboard effect") in the feature map after upsampling. Simultaneously, its lightweight design reduces computational complexity and improves computational efficiency. The dynamic upsampling update layer performs feature extraction with different receptive fields and combines it with bidirectional fusion of enhanced semantics. This preserves details of small targets while avoiding the impact of weak semantics of small target details on model training, thus improving the accuracy of scene detail detection.

[0050] Specifically, the following is executed in the upsampling update layer: By performing feature extraction on different receptive fields of the feature map through ordinary convolution, dynamic convolution and dilated convolution of the upsampling update layer, a fused feature map is obtained. A point sampling generator is used to obtain a sampling set based on the fused feature map; Resampling is performed based on the sample set and bilinear interpolation to obtain an upsampled feature map.

[0051] In this embodiment, the assumed bilinear interpolation X is resampled using the grid_sample function and the position of the sampling set S to obtain an upsampled feature map.

[0052] like Figure 3 As shown, in the point sampling generator constructed based on the static range factor, the point sampling generator obtains the sampling set based on the fused feature map, including: Two linear layers are constructed based on the number of channels in the fused feature map and the upsampling scaling factor; The initial offset is output based on the two linear layers and the fused feature map; The static offset is obtained based on the initial offset and the preset static range factor; Spatial offset is obtained by performing spatial reshaping on the static offset through pixel shuffling; The sampling set is obtained using the spatial offset and the original sampling grid.

[0053] If we set the upsampling scaling factor 's' and the number of channels in the fused feature map to 'C', then the number of input channels in the linear layer is 'C', and the number of output channels in the linear layer is 'C'. The fused feature map is processed through a linear layer to obtain an initial offset O. This initial offset O is multiplied by a preset static range factor to obtain a static offset, which locally constrains the wander range of the sampling position. Spatial reshaping is then performed on the static offset through pixel shuffling, converting the channel-dimensional offset into spatial coordinates to obtain the spatial offset. At this point, the sampling set coordinates are the sum of the spatial offset and the original sampling grid coordinates. In this embodiment, the preset static range factor is 0.25.

[0054] like Figure 4 As shown, in the point sampling generator constructed based on the dynamic range factor, the point sampling generator obtains the sampling set based on the fused feature map, including: A dynamic range factor is generated based on a preset static range factor and a Sigmoid function. Two linear layers are constructed based on the number of channels in the fused feature map and the upsampling scaling factor; Dynamic offsets are generated based on the fused feature map using two linear layers and a dynamic range factor. Spatial reshaping is performed on the dynamic offset by pixel shuffling to obtain the spatial offset; The sampling set is obtained using the spatial offset and the original sampling grid.

[0055] At this point, the preset static range factor is 0.5, and the dynamic range factor is generated in the range [0,0.5] with 0.25 as the center by the Sigmoid function.

[0056] In other embodiments, a point sampling generator corresponding to the static range factor and a point sampling generator corresponding to the dynamic range factor can be constructed simultaneously. Thus, feature extraction of different receptive fields of the feature map is performed through ordinary convolution, dynamic convolution, and dilated convolution in the upsampling update layer, respectively, and fusion contribution weights are obtained. The range factor of point sampling is obtained based on these fusion contribution weights. Specifically, a static range factor corresponding to ordinary convolution, a dynamic range factor corresponding to dynamic convolution, and a balanced range factor corresponding to dilated convolution are set. The static range factor, dynamic range factor, and balanced range factor are weighted according to the fusion contribution weights corresponding to each convolution process to obtain the range factor of point sampling. The static range factor is 0.15, the dynamic range factor is dynamically adjusted within a range of 0.3 according to the Sigmoid function, and the balanced range factor is dynamically adjusted within a range of 0.1 based on the static range factor of 0.2 according to the Sigmoid function.

[0057] like Figure 5 As shown, the following is performed in the attention enhancement layer: The input feature map is preprocessed based on layer normalization and grouping parameters to obtain the reconstructed feature map; Based on spatial feature enhancement and cross-spatial interaction, attention-enhanced feature maps are output in parallel according to the reshaped feature maps. Attention-enhanced feature map fusion based on parallel output generates multi-scale feature maps.

[0058] In this embodiment, a parallel dual-path structure is constructed through spatial attention and cross-spatial interaction to achieve feature enhancement fusion of local spatial attention and global context modeling, thereby improving the efficiency of feature representation learning and model training. In practical applications, the model can update weights more promptly based on the latest data, improving the efficiency and accuracy of flame or smoke recognition in fire scenarios.

[0059] Specifically, the preprocessing of the input feature map based on layer normalization and grouping parameters to obtain the reshaped feature map includes: Perform layer normalization on the input feature map; Based on the feature score parameters, the channel reshaping of the layer-normalized feature map is performed to obtain the reshaped feature map.

[0060] The feature map's channel dimension C is split into G groups using the feature score parameter G (i.e., C = G × (C / G)), and the grouped dimensions are then reshaped into the batch dimension. This allows different groups to learn differentiated semantic features. By isolating the learning process of different channels through grouping, redundant information interference between channels is avoided, enhancing the model's ability to capture diverse features.

[0061] Based on spatial feature enhancement and cross-spatial interaction, attention-enhanced feature maps are output in parallel according to the reshaped feature maps, including: Spatial feature enhancement of the reshaped feature map is performed based on parallel spatial decomposition in the horizontal and vertical directions to generate a first attention-enhanced feature map; Cross-spatial interaction of reconstructed feature maps is performed based on convolutional operations to generate a second attention-enhanced feature map.

[0062] Specifically, in the process of spatial feature enhancement: Global average pooling is used to decompose the reconstructed feature map in parallel along the horizontal and vertical directions to obtain horizontal and vertical feature maps, thus separating and highlighting the global trend of features in the horizontal and vertical directions. The dimensional order of the horizontal feature map is adjusted to make the shape of the horizontal feature map consistent with that of the vertical feature map. The horizontal and vertical feature maps are concatenated based on the height direction, and a 1×1 convolution operation is used to obtain the spatial fusion feature map, thereby realizing cross-directional information fusion. The spatial fusion feature map is divided into two tensors, and the two tensors are processed separately using the SiLU activation function to obtain the first tensor feature map and the second tensor feature map. Nonlinearity is introduced by the activation function to enhance the expressive power of the weights. Based on the first tensor feature map, the second tensor feature map, and the reshaped feature map, perform reweighting and grouping normalization to generate the first attention-enhanced feature map.

[0063] During cross-space interaction: Using 3×3 convolutions to perform convolution operations on the reshaped feature map generates a second attention-enhanced feature map. The convolution operation captures local cross-channel interactions and multi-scale information, thereby expanding the feature space.

[0064] Furthermore, the generation of multi-scale feature maps by fusing attention-enhanced feature maps based on parallel output includes: Based on two-dimensional average pooling and reshaping parallel acquisition of the first attention-enhanced feature map and the second attention-enhanced feature map, the first attention weight matrix and the second attention weight matrix are obtained respectively; The first multi-scale feature map is obtained based on the first attention-enhanced feature map and the first attention weight matrix and the second attention weight matrix corresponding to the second attention-enhanced feature map; A second multi-scale feature map is obtained based on the first multi-scale feature map and the reconstructed feature map; Based on skip connections and depthwise separable convolutions, a third multi-scale feature map is obtained from the second multi-scale feature map. The fourth multi-scale feature map is obtained from the third multi-scale feature map based on pointwise convolution; Based on residual connections, multi-scale feature maps are obtained from the fourth multi-scale feature map and the input feature map.

[0065] By using two-dimensional average pooling and reshaping to obtain dependencies in different directions and then fusing them, the results of cross-space interaction between two parallel sub-networks are obtained, achieving richer feature aggregation. While capturing pixel-level pairwise relationships, the global background of all pixels is highlighted, thereby avoiding interference from the scene background and improving the accuracy of flame and smoke recognition.

[0066] Specifically, the first attention weight matrix and the second attention weight matrix, obtained in parallel based on two-dimensional average pooling and reshaping, for the first attention-enhanced feature map and the second attention-enhanced feature map, include: After the first attention enhancement feature map is subjected to two-dimensional average pooling, the order of its two reciprocal dimensions is swapped, and then the Softmax activation function is used to obtain the first attention weight matrix corresponding to the first attention enhancement feature map. After the second attention-enhanced feature map is subjected to two-dimensional average pooling, its two reciprocal dimensions are concatenated to obtain the first attention weight matrix corresponding to the second attention-enhanced feature map. After the second attention-enhanced feature map is subjected to two-dimensional average pooling, the order of its two inverse dimensions is swapped, and then the Softmax activation function is used to obtain the second attention weight matrix corresponding to the first attention-enhanced feature map. After the first attention-enhanced feature map is processed by two-dimensional average pooling, the second attention weight matrix corresponding to the first attention-enhanced feature map is obtained.

[0067] The first multi-scale feature map is obtained based on the first attention-enhanced feature map and the corresponding first attention weight matrix and second attention weight matrix of the second attention-enhanced feature map, including: Based on the Matmul operation, a first weight feature map corresponding to the first attention weight matrix and a second weight feature map corresponding to the second attention weight matrix are constructed according to the first attention weight matrix and the second attention weight matrix, respectively, based on the first attention enhanced feature map and the second attention enhanced feature map. The first multi-scale feature map is obtained by summing the first weighted feature map and the second weighted feature map.

[0068] Obtaining the second multi-scale feature map based on the first multi-scale feature map and the reconstructed feature map includes: The first multi-scale feature map is processed using the Sigmoid activation function. The processed first multi-scale feature map is then reweighted with the reconstructed feature map to obtain the second multi-scale feature map.

[0069] After processing the first multi-scale feature map with the Sigmoid activation function, it is reweighted with the reshaped feature map, and finally reshaped into a batch of channel recovery to obtain the second multi-scale feature map.

[0070] Furthermore, a 3x3 depthwise separable convolution (DW-Conv) with skip connections, a BatchNorm2d function, and a SiLU activation function are stacked to process the second multi-scale feature map and obtain the third multi-scale feature map. This avoids information loss, promotes gradient flow, and preserves the effective information in the second multi-scale feature map.

[0071] The third multi-scale feature map is processed using a 1x1 pointwise convolution (Pw-Conv) to obtain the fourth multi-scale feature map. The pointwise convolution enables depthwise separable convolution with skip connections, which effectively controls model complexity when emphasizing both local and global features of the image.

[0072] Therefore, the fourth multi-scale feature map is residually concatenated with the original input feature map to output a multi-scale feature map.

[0073] The attention enhancement layer learns effective channel descriptions using shallow residual structures and balances model cost and accuracy. Through dynamic weight allocation, it can guide the model to focus on key regions or salient features while effectively suppressing irrelevant background noise, thereby significantly improving the accuracy and robustness of the model in complex fire scenarios.

[0074] In this embodiment, the loss function reconstructed in the YOLOv8s network architecture using aspect ratio consistency penalty and linear interval mapping includes: In constructing a width-to-height consistency penalty based on the logarithmic scale difference according to the aspect ratio difference: ; ; ; in, Represents the reconstructed loss function. Indicates intersection, union, and ratio. Represents Euclidean distance. Indicates the predicted bounding box. Represents the true bounding box. This represents the diagonal length of the smallest closure region that encloses both the predicted and ground truth bounding boxes. Represents dynamic geometric weights. This parameter measures the consistency between the aspect ratio of the predicted bounding box and the true bounding box. This indicates the width of the predicted bounding box. Indicates the height of the predicted bounding box. This represents the width of the actual bounding box. This represents the height of the actual bounding box. Represents the penalty function. To express summation, This represents the hyperparameters used to control hard samples.

[0075] In constructing a width-to-height aspect ratio consistency penalty based on the hyperbolic tangent function according to aspect ratio differences: ; ; .

[0076] In this embodiment, a nonlinear transformation aspect ratio consistency penalty is introduced into the loss function, making the loss function more sensitive to extreme ratio differences. This allows for accurate capture of the features of irregularly shaped flames and smoke, and provides stronger shape sensitivity to targets with extreme shapes.

[0077] Furthermore, based on real-time image data, the trained flame and smoke detection model is used to perform flame and smoke detection, achieving comprehensive feature extraction at large, medium, and small scales, thus improving the comprehensiveness of flame and smoke recognition. The loss function is used to improve the recognition accuracy of flames and smoke with extreme sizes.

[0078] As a second embodiment of this application, a flame and smoke detection system based on an improved YOLOv8s includes: The data acquisition unit is used to acquire historical fire image data and real-time image data; The model building unit is used to build the upsampling update layer and the loss function, and to train the flame and smoke detection model based on historical fire image data using the upsampling update layer, the loss function, and the YOLOv8s network architecture. The detection feedback unit is used to obtain the flame and smoke detection results based on real-time image data and the flame and smoke detection model.

[0079] The data acquisition unit is connected to both the model building unit and the detection feedback unit. By extracting features from different receptive fields in the upsampling update layer, it simultaneously achieves local detail recognition and global feature recognition for flame and smoke detection. Furthermore, it enhances the recognition of irregular objects through nonlinear transformation of the loss function, thereby improving the integrated local and global recognition, as well as the recognition of extreme shape proportions in flame and smoke detection. This makes it suitable for fire detection scenarios where flame and smoke shapes are easily variable and susceptible to environmental interference, thus improving the accuracy of fire detection.

[0080] In this embodiment, a deep learning platform can be built based on the Ubuntu 22.04 operating system, and PyTorch is selected as the training and testing framework for the deep learning model. GPU parallel computing is achieved in conjunction with the CUDA 11.8 and cuDNN 8.6.0 acceleration libraries. In terms of hardware, the training and testing processes of the model both utilize two NVIDIA GeForce RTX4090 GPUs with 24GB of dedicated video memory each, one Intel Xeon Gold 6132 CPU, and 32GB of DDR4 memory.

[0081] The training period was set to 300 epochs. Training was automatically terminated when the validation set performance did not improve for 50 consecutive epochs, preserving the optimal model parameters. The training used pre-trained model weights on the COCO dataset as initial parameters. During training, the input image resolution was fixed at 640×640 pixels, the batch size was set to 32, the optimizer was SGD, the initial learning rate was 0.001, and cosine annealing was used to adjust the learning rate. Other data augmentation strategies were set to default.

[0082] The specific embodiments described above are preferred embodiments of a flame and smoke detection method and system based on improved YOLOv8s according to this application, and are not intended to limit the specific scope of this application. The scope of this application includes but is not limited to the specific embodiments described above. All equivalent changes made in accordance with the shape and structure of this application are within the protection scope of this application.

Claims

1. A flame and smoke detection method based on an improved YOLOv8s, characterized in that: Includes the following steps: An upsampling update layer is constructed based on ordinary convolution, dynamic convolution, and dilated convolution corresponding to different receptive fields; A width-to-height consistency penalty is constructed based on the aspect ratio difference term using nonlinear transformation; The upsampling initial layer in the YOLOv8s network architecture is replaced by the upsampling update layer. The loss function in the YOLOv8s network architecture is reconstructed with aspect ratio consistency penalty and linear interval mapping to build the YOLOv8s basic model. A YOLOv8s base model was trained based on historical fire image data to obtain a flame and smoke detection model. Acquire real-time image data, and obtain flame and smoke detection results based on the real-time image data and the flame and smoke detection model.

2. The flame and smoke detection method based on improved YOLOv8s as described in claim 1, characterized in that: The construction of the upsampling update layer based on ordinary convolution, dynamic convolution, and dilated convolution corresponding to different receptive fields includes: The ordinary convolution corresponds to the first receptive field, the dynamic convolution corresponds to the second receptive field, and the dilated convolution corresponds to the third receptive field. The first receptive field is smaller than the second receptive field, and the second receptive field is smaller than the third receptive field.

3. The flame and smoke detection method based on improved YOLOv8s as described in claim 1, characterized in that: The construction of the upsampling update layer based on ordinary convolution, dynamic convolution, and dilated convolution corresponding to different receptive fields also includes: A feature extraction and fusion layer is constructed based on ordinary convolution, dynamic convolution, and dilated convolution corresponding to different receptive fields; Construct a point sampling generator based on dynamic range factors and / or static range factors; An upsampling update layer is constructed based on a feature extraction fusion layer and a point sampling generator.

4. The flame and smoke detection method based on improved YOLOv8s as described in claim 1, characterized in that: The aspect ratio consistency penalty based on aspect ratio differences constructed by nonlinear transformation includes: Aspect ratio consistency penalty is constructed based on the hyperbolic tangent function or logarithmic scale difference according to the aspect ratio difference.

5. The flame and smoke detection method based on improved YOLOv8s as described in claim 1, characterized in that: The process of training a YOLOv8s base model based on historical fire image data to obtain a flame and smoke detection model includes: Perform data preprocessing on historical fire image data to obtain a flame and smoke training dataset; By using a backbone network to perform feature extraction at different levels on the flame and smoke training dataset, feature maps at different levels are obtained. The neck network of the upsampling update layer extracts features from the feature map at multiple scales and fuses features at different levels to generate a multi-scale feature map. The detection head performs prediction processing on multi-scale feature maps to obtain predicted bounding boxes; By dynamically adjusting the model weights using a loss function, the optimal predicted bounding box is obtained, thus completing the model training and acquiring a flame and smoke detection model.

6. The flame and smoke detection method based on improved YOLOv8s as described in claim 5, characterized in that: The process of extracting features from the feature map at multiple scales and fusing features at different levels through the neck network of the upsampling update layer to generate a multi-scale feature map further includes: Based on top-down fusion, multi-scale extraction and cross-scale stitching of feature maps are performed through an upsampling update layer, a feature extraction layer, and a feature stitching layer to generate an upsampled fused feature map. By performing feature enhancement through a bottom-up approach, upsampling and fusion feature maps are performed through a feature extraction layer, a feature concatenation layer, and an attention enhancement layer to generate multi-scale feature maps.

7. The flame and smoke detection method based on improved YOLOv8s as described in claim 6, characterized in that: Perform the following in the attention enhancement layer: The input feature map is preprocessed based on layer normalization and grouping parameters to obtain the reconstructed feature map; Based on spatial feature enhancement and cross-spatial interaction, attention-enhanced feature maps are output in parallel according to the reshaped feature maps. Attention-enhanced feature map fusion based on parallel output generates multi-scale feature maps.

8. The flame and smoke detection method based on improved YOLOv8s as described in claim 7, characterized in that: The parallel output of attention-enhanced feature maps based on spatial feature enhancement and cross-spatial interaction, according to the reshaped feature maps, includes: Global average pooling is used to decompose the reconstructed feature map in parallel along the horizontal and vertical directions to obtain the horizontal and vertical feature maps. Adjust the dimensional order of the horizontal feature map to make the shape of the horizontal feature map consistent with that of the vertical feature map. Perform concatenation of the horizontal and vertical feature maps based on the height direction and use a 1×1 convolution operation to obtain the spatial fusion feature map. The spatial fusion feature map is cut into two tensors, and the two tensors are processed by the SiLU activation function to obtain the first tensor feature map and the second tensor feature map. Based on the first feature map, the second feature map, and the reshaped feature map, perform reweighting and grouping normalization to generate the first attention-enhanced feature map; A second attention-enhanced feature map is generated by performing a 3×3 convolution operation on the reshaped feature map.

9. The flame and smoke detection method based on improved YOLOv8s as described in claim 7, characterized in that: The attention-enhanced feature map fusion based on parallel output to generate multi-scale feature maps includes: Based on two-dimensional average pooling and reshaping parallel acquisition of the first attention-enhanced feature map and the second attention-enhanced feature map, the first attention weight matrix and the second attention weight matrix are obtained respectively; The first multi-scale feature map is obtained based on the first attention-enhanced feature map and the first attention weight matrix and the second attention weight matrix corresponding to the second attention-enhanced feature map; A second multi-scale feature map is obtained based on the first multi-scale feature map and the reconstructed feature map; Based on skip connections and depthwise separable convolutions, a third multi-scale feature map is obtained from the second multi-scale feature map. The fourth multi-scale feature map is obtained from the third multi-scale feature map based on pointwise convolution; Based on residual connections, multi-scale feature maps are obtained from the fourth multi-scale feature map and the input feature map.

10. A flame and smoke detection system based on an improved YOLOv8s, used to implement the method as described in any one of claims 1 to 9, characterized in that: include: The data acquisition unit is used to acquire historical fire image data and real-time image data; The model building unit is used to build the upsampling update layer and the loss function, and to train the flame and smoke detection model based on historical fire image data using the upsampling update layer, the loss function, and the YOLOv8s network architecture. The detection feedback unit is used to obtain the flame and smoke detection results based on real-time image data and the flame and smoke detection model.

Images