Method and apparatus for verifying deployment of a flame smoke detection network, device and medium
By constructing the YOLO-DSDFE detection network and combining a lightweight backbone network with a deep separable feature enhancement module, the complexity and computational overhead of flame and smoke detection on resource-constrained platforms are solved, achieving efficient and accurate flame and smoke detection.
Patent Information
- Application Number
- CN202611120122.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-27
- Publication Date
- 2026-11-10
AI Technical Summary
Existing forest fire detection technologies struggle to achieve efficient and accurate flame and smoke detection on resource-constrained FPGA platforms. Traditional methods increase network complexity, parameter count, and computational overhead, leading to risks of false detections and missed detections.
The YOLO-DSDFE detection network is adopted, combined with the YOLOv4-tiny dual-scale detection framework and the MobileNetV2 lightweight backbone network. A depth-separable deep feature enhancement module is introduced at the deep feature level. Through batch normalization fusion and INT16 fixed-point quantization, the deployment and verification on the ZYNQ7020 FPGA board are realized.
While reducing the number of parameters and computational load, it maintains the ability to detect flame morphology, weak smoke texture, and shallow edge information, achieving efficient deployment and accurate detection on resource-constrained platforms.
Smart Images

Figure CN123071867A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of target detection technology, and in particular to a method, apparatus, equipment and medium for deploying and verifying a flame and smoke detection network. Background Technology
[0002] Forest fire detection is a crucial research topic in forest resource and ecological environment security. Image-based flame and smoke recognition, by extracting the edge, shape, color, texture, and spatial location features of flames and smoke, transforms the early fire assessment problem into a computer vision target detection problem, which can serve applications such as forest fire prevention, drone inspection, and smart agriculture. Forest fire images are typically acquired in open scenes and are easily affected by factors such as target scale, changes in lighting, cloud cover, and tree backgrounds. In the early stages of a fire, the flame area is small, and the smoke target boundary is blurred and the texture is weak, making it easy to be visually confused with clouds, shadows, and complex backgrounds, thus leading to the risk of false positives and false negatives.
[0003] Traditional forest fire monitoring relies primarily on manual inspections and sensor detection, but these methods are easily limited by coverage, response latency, and environmental interference in open forest areas. The YOLO series models employ a single-stage end-to-end detection approach, balancing detection accuracy and inference efficiency. Existing fire detection research often improves accuracy through enhanced feature fusion and the introduction of attention mechanisms, but these methods typically increase network complexity, parameter count, and computational overhead, making them unsuitable for deployment on resource-constrained FPGA platforms such as the ZYNQ7020. Summary of the Invention
[0004] This invention provides a method, apparatus, device, and medium for deploying and verifying a flame and smoke detection network. It can maintain the ability to detect and express flame morphology, weak smoke texture, shallow edge, and local contour information while reducing the number of parameters, computational load, and hardware mapping complexity. It also enables BN fusion, INT16 fixed-point quantization, and deployment and verification on the ZYNQ7020 FPGA board.
[0005] In a first aspect, embodiments of the present invention provide a deployment and verification method for a flame and smoke detection network, comprising: A YOLO-DSDFE detection network for flame and smoke detection is constructed. The YOLO-DSDFE detection network adopts the YOLOv4-tiny dual-scale detection framework, uses the MobileNetV2 lightweight backbone network as the backbone network, and connects a depth-separable deep feature enhancement module at the deep features output by the backbone network. The YOLO-DSDFE detection network was trained using a flame and smoke dataset, and the weight model with the best performance on the validation set was selected. Batch normalization fusion is performed on the trained weight model to fuse the parameters of the batch normalization layer into the weights and biases of the convolutional layer, resulting in fused floating-point parameters. The fused floating-point parameters are converted into INT16 fixed-point parameters according to a preset scaling factor and exported as a binary file that can be read by the FPGA board. Inference is executed on the FPGA side through PS / PL collaborative scheduling. The PS side is responsible for configuring the parameters of the convolution IP core layer by layer, managing the feature map cache address and controlling the data flow. The PL side loads the binary file and performs convolution calculation using a custom configurable convolution IP core. The PC reads the raw data from the dual-scale detection head output from the FPGA, decodes and post-processes it according to the preset anchor frame, confidence threshold and non-maximum suppression threshold, and completes the evaluation and verification of the detection results.
[0006] In some embodiments, the YOLO-DSDFE detection network includes: The YOLOv4-tiny dual-scale detection framework includes a high-resolution detection branch and a low-resolution detection branch. The MobileNetV2 lightweight backbone network is used to replace the original YOLOv4-tiny backbone network. The MobileNetV2 lightweight backbone network outputs shallow features and deep features. The shallow features are 26×26 in size, and are adjusted to 256 channels by 1×1 pointwise convolution before being connected to the high-resolution detection branch. The deep features are 13×13 in size, and are adjusted to 512 channels by 1×1 pointwise convolution before being connected to the low-resolution detection branch. A depth-separable deep feature enhancement module is connected to the deep feature output of the MobileNetV2 lightweight backbone network and is used to perform lightweight enhancement of deep semantic features. The depth-separable deep feature enhancement module includes a first branch and a second branch set in parallel. The first branch is connected in series with a first 3×3 deep convolutional layer, a ReLU6 activation function layer and a second 3×3 deep convolutional layer. The second branch is connected in series with a third 3×3 deep convolutional layer. The outputs of the first branch and the second branch are connected to an element-wise addition unit. The element-wise addition unit is used to add the outputs of the first branch and the second branch after aligning them in the channel dimension and spatial dimension, and generate the enhanced deep features to be output to the low-resolution detection branch.
[0007] In some embodiments, both the high-resolution detection branch and the low-resolution detection branch are constructed using rule operators, which include 1×1 pointwise convolution, 3×3 depthwise separable convolution, ReLU6 activation function, upsampling layer and feature concatenation layer, and do not contain attention mechanism or dynamic branch structure.
[0008] In some embodiments, the convolutional IP core supports three computation modes: standard convolution, pointwise convolution, and depthwise separable convolution. The convolutional IP kernel integrates an input buffer, a weight buffer, a bias buffer, a parallel multiply-accumulate array, a bias and activation unit, and an output buffer. The convolutional IP core interacts with the PS terminal and DDR memory via the AXI bus interface. The PS terminal configures the convolutional kernel type, stride, padding method, and feature map size via the AXI bus. The PL terminal transfers feature maps and parameter data via the AXI host read / write interface.
[0009] In some embodiments, the execution of inference via PS / PL co-scheduling on the FPGA side includes: The PS end reads the input image data and INT16 fixed-point parameters from the DDR memory and loads the parameters into the corresponding cache of the convolution IP core at the PL end; The PS terminal sends start commands to the PL terminal layer by layer, configuring the input feature map address, output feature map address, convolution type, and stride parameters of the current layer. After receiving the instruction, the PL terminal starts the convolution calculation. After the calculation is completed, it notifies the PS terminal through an interrupt signal and writes the output feature map back to the specified address of the DDR memory. The PS terminal controls the flow of feature maps between high-resolution detection branches, low-resolution detection branches, and feature splicing layers based on the network topology, until the calculation of all network layers is completed.
[0010] In some embodiments, the step of reading the raw data of the dual-scale detection head output from the FPGA on the PC, decoding and post-processing it according to preset anchor frames, confidence thresholds and non-maximum suppression thresholds, and completing the evaluation and verification of the detection results includes: On the PC, a post-processing protocol that is completely consistent with the PyTorch training environment is built, including the same anchor box size, class order, confidence filtering threshold, and non-maximum suppression cross-union ratio threshold. Compare the fixed-point inference results output from the FPGA with the floating-point inference results output from the PyTorch, and calculate the deviations between the two in terms of bounding box coordinates, confidence level, and class probability. If the deviation is within the preset tolerance range, the software and hardware outputs are considered to be consistent, and the model adaptation is successful.
[0011] In some embodiments, the method further includes: When the feature map size exceeds the cache capacity of the FPGA's on-chip block random access memory (BRAM), the PS terminal controls the PL terminal to process the feature map in blocks, sequentially moving the image blocks to the on-chip cache for calculation, and writing the calculation results back to the corresponding location in the DDR memory. The memory access delay is masked by overlapping data moving and calculation processes.
[0012] Secondly, embodiments of the present invention also provide a deployment verification device for a flame and smoke detection network, the device comprising: The construction module is used to construct a YOLO-DSDFE detection network for flame and smoke detection. The YOLO-DSDFE detection network adopts the YOLOv4-tiny dual-scale detection framework, uses the MobileNetV2 lightweight backbone network as the backbone network, and connects a depth-separable deep feature enhancement module at the deep features output by the backbone network. The training module is used to train the YOLO-DSDFE detection network using the flame and smoke dataset and select the weight model with the best performance on the validation set. The fusion module is used to perform batch normalization fusion on the trained weight model, fusing the parameters of the batch normalization layer into the weights and biases of the convolutional layer to obtain the fused floating-point parameters. The conversion module is used to convert the fused floating-point parameters into INT16 fixed-point parameters according to a preset scaling factor, and export them as a binary file that can be read by the FPGA board. The inference module is used to perform inference on the FPGA side through PS / PL collaborative scheduling. The PS side is responsible for configuring the parameters of the convolution IP core layer by layer, managing the feature map cache address and controlling the data flow. The PL side loads the binary file and performs convolution calculation using a custom configurable convolution IP core. The verification module is used to read the raw data of the dual-scale detection head output from the FPGA on the PC, decode and post-process it according to the preset anchor frame, confidence threshold and non-maximum suppression threshold, and complete the evaluation and verification of the detection results.
[0013] Thirdly, embodiments of the present invention also provide an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, when the processor executes the computer program, it implements the deployment and verification method of the flame and smoke detection network as described in the first aspect.
[0014] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing computer-executable instructions for performing the deployment verification method of the flame and smoke detection network as described in the first aspect.
[0015] According to embodiments of the present invention, a method, apparatus, device, and medium for deploying and validating a flame and smoke detection network are provided. The method includes: constructing a YOLO-DSDFE detection network for flame and smoke detection, wherein the YOLO-DSDFE detection network adopts the YOLOv4-tiny dual-scale detection framework, uses a lightweight MobileNetV2 backbone network as the backbone network, and connects depth-separable deep feature enhancement modules at the deep features output by the backbone network; training the YOLO-DSDFE detection network using a flame and smoke dataset, selecting the weight model with the best performance on the validation set; and batch normalizing and fusing the trained weight models. The parameters of the batch normalization layer are fused into the weights and biases of the convolutional layer to obtain fused floating-point parameters. These parameters are then converted to INT16 fixed-point parameters according to a preset scaling factor and exported as a binary file readable by the FPGA board. Inference is executed on the FPGA via PS / PL collaborative scheduling. The PS is responsible for configuring the parameters of the convolutional IP cores layer by layer, managing the feature map cache address, and controlling the data flow. The PL loads the binary file and performs convolution calculations using a custom configurable convolutional IP core. The PC reads the raw data from the dual-scale detection head output from the FPGA, decodes and post-processes it according to preset anchor boxes, confidence thresholds, and non-maximum suppression thresholds, and completes the evaluation and verification of the detection results. Based on this, this embodiment of the invention can maintain the ability to detect and express flame morphology, weak smoke texture, shallow edges, and local contour information while reducing the number of parameters, computational load, and hardware mapping complexity. It also achieves BN fusion, INT16 fixed-point quantization, and deployment verification on the ZYNQ7020 FPGA board. Attached Figure Description
[0016] Figure 1A This is a flowchart of a deployment and verification method for a flame and smoke detection network provided in one embodiment of the present invention; Figure 1B This is a structural block diagram of the YOLO-DSDFE detection network provided in one embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of a depth-separable deep feature enhancement module provided in one embodiment of the present invention; Figure 3 This is a flowchart illustrating the overall deployment of a flame and smoke detection network according to an embodiment of the present invention. Figure 4 This is a schematic diagram of the deployment verification device for a flame and smoke detection network provided in one embodiment of the present invention; Figure 5 This is a schematic diagram of an electronic device provided in one embodiment of the present invention. Detailed Implementation
[0017] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.
[0018] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the following drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.
[0019] In this embodiment of the invention, the terms "furthermore," "exemplarily," or "optionally" are used as examples, illustrations, or descriptions and should not be construed as being more preferred or advantageous than other embodiments or designs. The use of the terms "furthermore," "exemplarily," or "optionally" is intended to present the relevant concepts in a specific manner.
[0020] First, let's analyze some of the terms used in this invention: YOLO-DSDFE: A flame and smoke detection network that includes a depth-separable deep feature enhancement module.
[0021] DSDFE: Depth Separable Deep Feature Enhancement Module.
[0022] YOLOv4-tiny: A lightweight object detection network for YOLO version 4.
[0023] MobileNetV2: A lightweight convolutional neural network with second-generation depthwise separable convolutions.
[0024] FPGA: Field Programmable Gate Array.
[0025] ZYNQ7020: Resource-constrained field-programmable gate array system-on-a-chip platform.
[0026] P4 detection branch: High-resolution detection branch.
[0027] P5 detection branch: Low-resolution detection branch.
[0028] feat1: Shallow features.
[0029] feat2: Deep features.
[0030] BN fusion: The batch normalization layer parameters are fused into the convolution weights and biases, and the BN layer is no longer executed separately during the inference phase.
[0031] INT16 fixed-point quantization: Exports the fused floating-point parameters as 16-bit fixed-point integer parameters; this invention uses a fixed scaling factor and is consistent with the HLS-side ap_fixed<16,7> data format.
[0032] HLS / IP: HLS stands for High-Level Synthesis; in this invention, basic convolution operators are implemented using Vivado HLS and exported as custom IP cores that can be called by Vivado.
[0033] PS / PL: Processing System and Programmable Logic in the ZYNQ platform; in this invention, PS is responsible for layer-by-layer scheduling and data organization, and PL executes configurable HLS convolutional IPs.
[0034] NMS: Non-maximum suppression, used to retain the final detection result from the candidate detection box.
[0035] AXI: Advanced Scalable Interface Bus, used for data transmission and control configuration between the PS and PL ends.
[0036] DDR: Double Data Rate External Memory, used to store input images, feature maps, weights, and bias data.
[0037] DSP: Digital Signal Processing resource, used for multiply-accumulate calculations in FPGAs.
[0038] BRAM: Block Random Access Memory, used for on-chip caching in FPGAs.
[0039] DWConv: Depthwise convolution, which is a convolution calculation performed on a channel-by-channel basis.
[0040] PWConv: Pointwise convolution, i.e., 1×1 convolution, used for channel transformation.
[0041] StdConv: Standard convolution.
[0042] Upsample: Upsampling is used to enlarge the size of the feature map.
[0043] Tiling: Block-based computation, used to reduce on-chip cache pressure.
[0044] Pipeline: Streamlined processing used to improve computational throughput.
[0045] Ping-pong buffer: Used for overlapping data handling and computation processes.
[0046] Anchor: Anchor box, used for bounding box decoding in the YOLO detection head.
[0047] To facilitate a more convenient description of the working principle of the embodiments of the present invention, the following introduction of relevant technical scenarios is given first.
[0048] Forest fire detection is a crucial research topic in forest resource and ecological environment security. Image-based flame and smoke recognition, by extracting the edge, shape, color, texture, and spatial location features of flames and smoke, transforms the early fire assessment problem into a computer vision target detection problem, which can serve applications such as forest fire prevention, drone inspection, and smart agriculture. Forest fire images are typically acquired in open scenes and are easily affected by factors such as target scale, changes in lighting, cloud cover, and tree backgrounds. In the early stages of a fire, the flame area is small, and the smoke target boundary is blurred and the texture is weak, making it easy to be visually confused with clouds, shadows, and complex backgrounds, thus leading to the risk of false positives and false negatives.
[0049] Traditional forest fire monitoring relies primarily on manual inspections and sensor detection, but these methods are easily limited by coverage, response latency, and environmental interference in open forest areas. The YOLO series models employ a single-stage end-to-end detection approach, balancing detection accuracy and inference efficiency. Existing fire detection research often improves accuracy through enhanced feature fusion and the introduction of attention mechanisms, but these methods typically increase network complexity, parameter count, and computational overhead, making them unsuitable for deployment on resource-constrained FPGA platforms such as the ZYNQ7020.
[0050] Based on this, the present invention provides a method, apparatus, device, and medium for deploying and validating a flame and smoke detection network. The method for deploying and validating the flame and smoke detection network includes: constructing a YOLO-DSDFE detection network for flame and smoke detection, wherein the YOLO-DSDFE detection network adopts the YOLOv4-tiny dual-scale detection framework, uses a lightweight MobileNetV2 backbone network as the backbone network, and connects depthwise separable deep feature enhancement modules at the deep features output by the backbone network; training the YOLO-DSDFE detection network using a flame and smoke dataset, and selecting the weight model with the best performance on the validation set; performing batch normalization fusion on the trained weight model, fusing the parameters of the batch normalization layer into the weights of the convolutional layer. During the biasing process, the fused floating-point parameters are obtained. These parameters are then converted to INT16 fixed-point parameters according to a preset scaling factor and exported as a binary file readable by the FPGA board. Inference is executed on the FPGA via PS / PL collaborative scheduling. The PS is responsible for configuring the parameters of the convolutional IP cores layer by layer, managing the feature map cache address, and controlling the data flow. The PL loads the binary file and performs convolution calculations using a custom configurable convolutional IP core. On the PC, the original dual-scale detection head data output from the FPGA is read, decoded, and post-processed according to preset anchor frames, confidence thresholds, and non-maximum suppression thresholds to complete the evaluation and verification of the detection results. Based on this, this embodiment of the invention can maintain the ability to detect and express flame morphology, weak smoke texture, shallow edges, and local contour information while reducing the number of parameters, computational load, and hardware mapping complexity. It also achieves BN fusion, INT16 fixed-point quantization, and deployment verification on the ZYNQ7020 FPGA board.
[0051] The embodiments of the present invention will be further described below with reference to the accompanying drawings.
[0052] like Figure 1A As shown, Figure 1A This is a flowchart of a deployment and verification method for a flame and smoke detection network provided in an embodiment of the present invention. The deployment and verification method for the flame and smoke detection network may include, but is not limited to, steps S101 to S106.
[0053] Step S101: Construct a YOLO-DSDFE detection network for flame and smoke detection. The YOLO-DSDFE detection network adopts the YOLOv4-tiny dual-scale detection framework, uses the MobileNetV2 lightweight backbone network as the backbone network, and connects a depth-separable deep feature enhancement module at the deep features output by the backbone network. Step S102: Train the YOLO-DSDFE detection network using the flame and smoke dataset, and select the weight model with the best performance on the validation set. Step S103: Perform batch normalization fusion on the trained weight model, and fuse the parameters of the batch normalization layer into the weights and biases of the convolutional layer to obtain the fused floating-point parameters. Step S104: Convert the fused floating-point parameters into INT16 fixed-point parameters according to the preset scaling factor, and export them as a binary file that can be read by the FPGA board. Step S105: Inference is executed on the FPGA side through PS / PL collaborative scheduling. The PS side is responsible for configuring the parameters of the convolution IP core layer by layer, managing the feature map cache address and controlling the data flow. The PL side loads the binary file and performs convolution calculation using a custom configurable convolution IP core. Step S106: Read the raw data of the dual-scale detection head output from the FPGA on the PC, and decode and post-process it according to the preset anchor frame, confidence threshold and non-maximum suppression threshold to complete the evaluation and verification of the detection results.
[0054] It is understandable that, such as Figure 1B As shown, the detection network for flame and smoke detection constructed in this invention is a lightweight detection network based on YOLO-DSDFE. It retains the dual-scale detection framework of YOLOv4-tiny and adopts the MobileNetV2 backbone to reduce the number of parameters and computational cost. A depth-separable deep feature enhancement module is introduced at deep feature levels to enhance the semantic expression of flame morphology and weak smoke texture. After training, the model is further validated through Batch Normalization (BN) fusion, INT16 fixed-point parameter export, and ZYNQ7020 FPGA board-side inference to analyze the consistency between software and hardware outputs.
[0055] It is understood that the YOLO-DSDFE detection network in this embodiment of the invention, while retaining the YOLOv4-tiny dual-scale detection framework, reduces the number of model parameters and computational load through a lightweight MobileNetV2 backbone, making the network structure more suitable for deployment on resource-constrained FPGA edge platforms. A depthwise separable deep feature enhancement module compensates for potential deficiencies in deep semantic expression caused by the lightweight backbone, improving the model's ability to discriminate differences in flame morphology, weak smoke texture, and complex backgrounds. Simultaneously, this embodiment employs BN fusion, INT16 fixed-point quantization, a custom HLS convolutional IP kernel, and PS / PL collaborative layer-by-layer scheduling to transform the software-trained model into a fixed-point inference process executable on the FPGA board, reducing computational and parameter reading overhead during the inference phase and improving the model's adaptability to low-resource hardware platforms such as the ZYNQ7020. This embodiment can achieve board-side inference verification while maintaining flame and smoke detection capabilities, and provides a deployable technical solution for low-cost, low-power forest fire monitoring nodes, edge intelligent fire early warning devices, and drone-based fire identification modules.
[0056] Understandably, YOLOv4-tiny consists of a backbone feature extraction network, a feature fusion structure, and a dual-scale detection head. The dual-scale detection head includes a P4 high-resolution detection branch and a P5 low-resolution detection branch. The P4 high-resolution detection branch is more advantageous for small-scale target detection, while the P5 low-resolution detection branch has stronger semantic representation capabilities and is suitable for identifying targets with more obvious semantic information.
[0057] This invention introduces MobileNetV2 into the YOLOv4-tiny backbone network. MobileNetV2 reduces the amount of multiply-accumulate operations through inverse residual structures and depthwise separable convolutions, maintaining a certain feature extraction capability while reducing the number of parameters and computational cost. Furthermore, two key feature layers are selected from the backbone network for dual-scale detection: shallow feature feat1 has high spatial resolution and is used to preserve edge, texture, and local color information; deep feature feat2 has stronger semantic expressive power and is used for classifying flame and smoke targets.
[0058] In the backbone network structure of this invention, the input image size is 416×416; after feature extraction by the MobileNetV2 network, two detection feature layers are obtained: the shallow layer feat1 has a size of 26×26, which is adjusted to 256 channels by 1×1 convolution and used for the P4 high-resolution detection branch; the deep layer feat2 has a size of 13×13, which is adjusted to 512 channels by 1×1 convolution and used for the low-resolution P5 detection branch.
[0059] It should be noted that, where hardware resources permit or the model structure requires further adjustment, some convolutional layers in the P4 high-resolution detection branch can be replaced with depthwise separable convolutions. This alternative can still improve the detection performance of small-scale flames, thin smoke, and weakly boundary targets by utilizing deep semantic enhancement and shallow detail compensation, and can further reduce the number of parameters and computational cost of the P4 high-resolution detection branch, making the model easier to deploy on resource-constrained devices, but may lead to a decrease in detection accuracy.
[0060] Understandably, the lightweight structure introduced into MobileNetV2 with the YOLOv4-tiny backbone may weaken the ability to express deep semantics. To address the issues of indistinct textures, large morphological variations, blurred boundaries, and susceptibility to background interference in small-scale flames, thin smoke, and weak-boundary smoke, this invention designes a depth-separable deep feature enhancement module at the deep feature feat2 output of the backbone, such as... Figure 2 As shown.
[0061] The depthwise separable deep feature enhancement module is located at the deep feature feat2 output of the backbone network and is used to perform lightweight enhancement of deep semantic features. It mainly consists of multiple 3×3 depthwise separable convolutional branches, ReLU6 activation function, and element-wise addition structure. Without significantly increasing the number of parameters and computation, it enhances the ability of deep features to express differences in flame morphology, weak smoke texture, and complex background.
[0062] The structure of the depthwise separable deep feature enhancement module is as follows: After inputting the deep feature feat2, the feature enters two depthwise separable convolution branches. The first branch sequentially passes through a 3×3 depthwise separable convolution, a ReLU6 activation function, and another 3×3 depthwise separable convolution to enhance the spatial representation and semantic representation capabilities of the deep features. The second branch passes through a 3×3 depthwise separable convolution to perform lightweight transformation and feature compensation on the input features. The outputs of the two branches are element-wise summed under the condition that the number of channels and the feature map size are consistent, resulting in the enhanced deep feature output. This module mainly consists of depthwise separable convolution, ReLU6 activation, and element-wise summation. The operator structure is regular, the computational overhead is low, and it is easy to implement INT16 fixed-point conversion and FPGA mapping.
[0063] It is understood that the embodiments of this invention construct a deployment and verification process from PyTorch model training, Batch Normalization (BN) fusion, INT16 fixed-point parameter derivation, to ZYNQ7020 board-side inference and PC-side result evaluation. Considering the limited logic resources, DSP resources, and on-chip storage on the FPGA side, this invention prioritizes the use of regular operators such as 1×1 pointwise convolution, 3×3 depthwise separable convolution, ReLU6, upsampling, and feature concatenation in the model design, avoiding the introduction of irregular structures such as complex attention and dynamic branches, in order to improve the fixed-point normalization and hardware mapping friendliness.
[0064] The overall process consists of two stages: The first stage is the software-side parameter conversion and fixed-point export, which performs Batch Normalization (BN) fusion on the trained PyTorch weights and exports the fused weights and biases as INT16 binary files according to the board-side fixed-point format to reduce the overhead of independent BN calculation and parameter reading on the board; The second stage is the board-side execution process, which implements the basic convolution operator in Vivado HLS, completes IP core integration and bitstream generation in Vivado, and completes the whole network layer-by-layer inference on the Vitis side through PS / PL collaboration.
[0065] Understandably, after the model training is complete, the optimal weights in PyTorch are fused using Batch Normalization (BN). After BN fusion, the inference stage no longer needs to execute a separate BN layer; the equivalent computation can be completed directly using the fused convolutional weights and biases, thereby simplifying the network structure and reducing the computation and memory access overhead of intermediate layers.
[0066] Subsequently, the fused floating-point parameters are exported as INT16 fixed-point parameters. This invention uses a fixed scaling factor S=512=2^9 to maintain consistency with the HLS-side ap_fixed<16,7> data format; during fixed-point export, rounding is performed and the quantization results are limited to the INT16 value range. This process converts PyTorch-side floating-point parameters into FPGA-side loadable fixed-point parameters, providing a parameter basis for board-side inference and output consistency verification.
[0067] Understandably, during the hardware implementation phase, Vivado HLS uses C / C++ to describe the underlying computational logic of basic operators such as standard convolution, pointwise convolution, and depthwise separable convolution, and completes the design of data bit width, input / output interfaces, address transmission, and on-chip cache. After C simulation, synthesis, and interface verification, the operator module is exported as a Vivado-callable IP core. Subsequently, in Vivado, the connection configuration of the ZYNQ PS end, DDR memory, AXI bus, and custom convolution IP is completed, and a bitstream file is generated. The configurable HLS convolution acceleration IP includes input cache, weight cache, bias cache, cache interface, parallel multiply-accumulate array, bias and activation units, and output cache set. The PS end configures the IP through the AXI bus, and feature maps and parameters are transmitted through the AXI host read / write interface.
[0068] On the Vitis side, an application project is built based on the generated hardware platform, and the PS-side control program is written. During board-side inference, the PS-side is responsible for overall network scheduling, including input data reading, weight file loading, feature map cache address configuration, convolutional layer parameter passing, branch structure synchronization, and output file saving; the PL-side mainly performs computationally intensive convolution operations.
[0069] During inference, the PS (Power Supply) layer passes parameters such as input feature map address, output feature map address, feature map size, number of channels, convolution kernel type, stride, and weight address to the IP (Input / Output Layer) core layer by layer. After each layer's calculation, the output feature map is written back to a designated buffer area and used as input for subsequent layers. After inference, the FPGA board generates the raw output file of the dual-scale detection head. The PC reads this output file and performs bounding box decoding, non-maximum suppression, and index calculation according to the same anchor, class order, confidence threshold, NMS threshold, and evaluation protocol as the PyTorch end.
[0070] It is understood that the present invention provides a deployment and verification method for a flame and smoke detection network. First, a flame and smoke target detection dataset is prepared, and the images are uniformly adjusted to a 416×416 input size, retaining only flame and smoke targets. Then, based on the YOLOv4-tiny dual-scale detection framework, MobileNetV2 is introduced as a lightweight backbone network to extract shallow features feat1 and deep features feat2. The shallow features are used to preserve edge, texture, and local contour information, while the deep features are used to enhance the semantic discrimination ability of flame and smoke targets.
[0071] In the network structure, a depthwise separable deep feature enhancement module is set at the deep feature feat2. The features of this module are fed into two depthwise separable convolution branches. The first branch sequentially passes through a 3×3 depthwise separable convolution, a ReLU6 activation function, and another 3×3 depthwise separable convolution to enhance the spatial representation and semantic representation capabilities of the deep features. The second branch passes through a 3×3 depthwise separable convolution to perform lightweight transformation and feature compensation on the input features. The outputs of the two branches are added element-wise under the condition that the number of channels and the feature map size are consistent, to obtain the enhanced deep feature output. This module mainly consists of depthwise separable convolution, ReLU6 activation, and element-wise addition. The operator structure is regular, the computational cost is low, and it is easy to perform INT16 fixed-point conversion and FPGA mapping.
[0072] like Figure 3 As shown, after model training, the weights with the best performance on the validation set are selected for batch normalization (BN) fusion, and the batch normalized layer parameters are fused into the convolutional weights and biases to reduce the independent computational overhead during the inference stage. Subsequently, the fused floating-point parameters are exported as INT16 fixed-point parameters using a fixed scaling factor, and a binary file of weights and biases readable by the FPGA board is generated. In the hardware implementation stage, basic operators such as standard convolution, pointwise convolution, and depthwise separable convolution are implemented using Vivado HLS, and a custom convolutional IP core is exported. The connection between the PS end, DDR storage, AXI bus, and custom convolutional IP core of the ZYNQ7020 platform is completed in Vivado. A PS end control program is written on the Vivado side to implement input data reading, weight loading, feature cache address configuration, layer-by-layer scheduling, branch synchronization, and output saving. After the board-side inference is completed, a dual-scale detector head raw output file is generated, and then the PC side completes decoding and index calculation according to the same class order, anchor box settings, confidence threshold, non-maximum suppression threshold, and evaluation protocol as the PyTorch side.
[0073] Understandably, compared to the original YOLOv4-tiny and MobileNetV2-YOLOv4-tiny baseline models, YOLO-DSDFE improves flame and smoke detection performance while maintaining a lightweight design. This invention does not improve accuracy by increasing model size, but rather through the collaborative design of a lightweight backbone network and deeply separable deep feature enhancement modules, reducing model complexity while enhancing the ability to identify small-scale flames, thin smoke, and weakly boundaryed targets. In terms of deployment verification, after BN fusion, INT16 fixed-pointing, and ZYNQ7020 FPGA board-side inference, YOLO-DSDFE still maintains good consistency between software and hardware output, indicating the feasibility of deploying this model on resource-constrained FPGA platforms.
[0074] In addition, such as Figure 4 As shown, one embodiment of the present invention also discloses a deployment verification device for a flame and smoke detection network, the device comprising: Module 110 is used to build a YOLO-DSDFE detection network for flame and smoke detection. The YOLO-DSDFE detection network adopts the YOLOv4-tiny dual-scale detection framework, uses the MobileNetV2 lightweight backbone network as the backbone network, and connects a depth-separable deep feature enhancement module at the deep features output by the backbone network. Training module 120 is used to train the YOLO-DSDFE detection network using the flame and smoke dataset and select the weight model with the best performance on the validation set. The fusion module 130 is used to perform batch normalization fusion on the trained weight model, fusing the parameters of the batch normalization layer into the weights and biases of the convolutional layer to obtain the fused floating-point parameters. The conversion module 140 is used to convert the fused floating-point parameters into INT16 fixed-point parameters according to a preset scaling factor, and export them as a binary file that can be read by the FPGA board. The inference module 150 is used to perform inference on the FPGA side through PS / PL collaborative scheduling. The PS side is responsible for configuring the parameters of the convolution IP core layer by layer, managing the feature map cache address and controlling the data flow. The PL side loads the binary file and performs convolution calculation using a custom configurable convolution IP core. The verification module 160 is used to read the raw data of the dual-scale detection head output from the FPGA on the PC, decode and post-process it according to the preset anchor frame, confidence threshold and non-maximum suppression threshold, and complete the evaluation and verification of the detection results.
[0075] The deployment verification device for the flame and smoke detection network in this embodiment of the invention is used to execute the deployment verification method for the flame and smoke detection network in the above embodiment. Its specific processing procedure is the same as that of the deployment verification method for the flame and smoke detection network in the above embodiment, and will not be described in detail here.
[0076] In addition, such as Figure 5 As shown, one embodiment of the present invention also discloses an electronic device, including: at least one processor 210; at least one memory 220 for storing at least one program; and when the at least one program is executed by the at least one processor 210, implementing the deployment verification method of the flame and smoke detection network as in any of the preceding embodiments.
[0077] In addition, one embodiment of the present invention discloses a computer-readable storage medium storing computer-executable instructions for performing the deployment verification method of the flame and smoke detection network as described in any of the preceding embodiments.
[0078] The system architecture and application scenarios described in the embodiments of this invention are for the purpose of more clearly illustrating the technical solutions of the embodiments of this invention, and do not constitute a limitation on the technical solutions provided by the embodiments of this invention. As those skilled in the art will know, with the evolution of system architecture and the emergence of new application scenarios, the technical solutions provided by the embodiments of this invention are also applicable to similar technical problems.
[0079] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.
[0080] In hardware implementations, the division between functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.
[0081] The terms “component,” “module,” “system,” etc., used in this specification are used to refer to computer-related entities, hardware, firmware, combinations of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program, or a computer. As illustrated, applications running on computing devices and computing devices can both be components. One or more components may reside in a process or execution thread, and components may be located on a single computer or distributed among two or more computers. Furthermore, these components can be executed from various computer-readable media on which various data structures are stored. Components can communicate, for example, via local or remote processes based on signals having one or more data packets (e.g., data from two components interacting with another component between a local system, a distributed system, or a network, such as the Internet interacting with other systems via signals).
Claims
1. A method for deploying and validating a flame and smoke detection network, comprising: A YOLO-DSDFE detection network for flame and smoke detection is constructed. The YOLO-DSDFE detection network adopts the YOLOv4-tiny dual-scale detection framework, uses the MobileNetV2 lightweight backbone network as the backbone network, and connects a depth-separable deep feature enhancement module at the deep features output by the backbone network. The YOLO-DSDFE detection network was trained using a flame and smoke dataset, and the weight model with the best performance on the validation set was selected. Batch normalization fusion is performed on the trained weight model to fuse the parameters of the batch normalization layer into the weights and biases of the convolutional layer, resulting in fused floating-point parameters. The fused floating-point parameters are converted into INT16 fixed-point parameters according to a preset scaling factor and exported as a binary file that can be read by the FPGA board. Inference is executed on the FPGA side through PS / PL collaborative scheduling. The PS side is responsible for configuring the parameters of the convolution IP core layer by layer, managing the feature map cache address and controlling the data flow. The PL side loads the binary file and performs convolution calculation using a custom configurable convolution IP core. The PC reads the raw data from the dual-scale detection head output from the FPGA, decodes and post-processes it according to the preset anchor frame, confidence threshold and non-maximum suppression threshold, and completes the evaluation and verification of the detection results.
2. The method according to claim 1, characterized in that, The YOLO-DSDFE detection network includes: The YOLOv4-tiny dual-scale detection framework includes a high-resolution detection branch and a low-resolution detection branch. The MobileNetV2 lightweight backbone network is used to replace the original YOLOv4-tiny backbone network. The MobileNetV2 lightweight backbone network outputs shallow features and deep features. The shallow features are 26×26 in size, and are adjusted to 256 channels by 1×1 pointwise convolution before being connected to the high-resolution detection branch. The deep features are 13×13 in size, and are adjusted to 512 channels by 1×1 pointwise convolution before being connected to the low-resolution detection branch. A depth-separable deep feature enhancement module is connected to the deep feature output of the MobileNetV2 lightweight backbone network and is used to perform lightweight enhancement of deep semantic features. The depth-separable deep feature enhancement module includes a first branch and a second branch set in parallel. The first branch is connected in series with a first 3×3 deep convolutional layer, a ReLU6 activation function layer and a second 3×3 deep convolutional layer. The second branch is connected in series with a third 3×3 deep convolutional layer. The outputs of the first branch and the second branch are connected to an element-wise addition unit. The element-wise addition unit is used to add the outputs of the first branch and the second branch after aligning them in the channel dimension and spatial dimension, and generate the enhanced deep features to be output to the low-resolution detection branch.
3. The method according to claim 2, characterized in that, Both the high-resolution detection branch and the low-resolution detection branch are constructed using regular operators, which include 1×1 pointwise convolution, 3×3 depthwise separable convolution, ReLU6 activation function, upsampling layer and feature concatenation layer, and do not contain attention mechanism or dynamic branch structure.
4. The method according to claim 1, characterized in that, The convolutional IP kernel supports three computation modes: standard convolution, pointwise convolution, and depthwise separable convolution. The convolutional IP kernel integrates an input buffer, a weight buffer, a bias buffer, a parallel multiply-accumulate array, a bias and activation unit, and an output buffer. The convolutional IP core interacts with the PS terminal and DDR memory via the AXI bus interface. The PS terminal configures the convolutional kernel type, stride, padding method, and feature map size via the AXI bus. The PL terminal transfers feature maps and parameter data via the AXI host read / write interface.
5. The method according to claim 1, characterized in that, The execution of inference via PS / PL collaborative scheduling on the FPGA side includes: The PS end reads the input image data and INT16 fixed-point parameters from the DDR memory and loads the parameters into the corresponding cache of the convolution IP core at the PL end; The PS terminal sends start commands to the PL terminal layer by layer, configuring the input feature map address, output feature map address, convolution type, and stride parameters of the current layer. After receiving the instruction, the PL terminal starts the convolution calculation. After the calculation is completed, it notifies the PS terminal through an interrupt signal and writes the output feature map back to the specified address of the DDR memory. The PS terminal controls the flow of feature maps between high-resolution detection branches, low-resolution detection branches, and feature splicing layers based on the network topology, until the calculation of all network layers is completed.
6. The method according to claim 1, characterized in that, The process of reading the raw data from the dual-scale detection head output from the FPGA on the PC, decoding and post-processing it according to preset anchor frames, confidence thresholds, and non-maximum suppression thresholds, and completing the evaluation and verification of the detection results includes: On the PC, a post-processing protocol that is completely consistent with the PyTorch training environment is built, including the same anchor box size, class order, confidence filtering threshold, and non-maximum suppression cross-union ratio threshold. Compare the fixed-point inference results output from the FPGA with the floating-point inference results output from the PyTorch, and calculate the deviations between the two in terms of bounding box coordinates, confidence level, and class probability. If the deviation is within the preset tolerance range, the software and hardware outputs are considered to be consistent, and the model adaptation is successful.
7. The method according to claim 1, characterized in that, The method further includes: When the feature map size exceeds the cache capacity of the FPGA's on-chip block random access memory (BRAM), the PS terminal controls the PL terminal to process the feature map in blocks, sequentially moving the image blocks to the on-chip cache for calculation, and writing the calculation results back to the corresponding location in the DDR memory. The memory access delay is masked by overlapping data moving and calculation processes.
8. A deployment verification device for a flame and smoke detection network, characterized in that, The device includes: The construction module is used to construct a YOLO-DSDFE detection network for flame and smoke detection. The YOLO-DSDFE detection network adopts the YOLOv4-tiny dual-scale detection framework, uses the MobileNetV2 lightweight backbone network as the backbone network, and connects a depth-separable deep feature enhancement module at the deep features output by the backbone network. The training module is used to train the YOLO-DSDFE detection network using the flame and smoke dataset and select the weight model with the best performance on the validation set. The fusion module is used to perform batch normalization fusion on the trained weight model, fusing the parameters of the batch normalization layer into the weights and biases of the convolutional layer to obtain the fused floating-point parameters. The conversion module is used to convert the fused floating-point parameters into INT16 fixed-point parameters according to a preset scaling factor, and export them as a binary file that can be read by the FPGA board. The inference module is used to perform inference on the FPGA side through PS / PL collaborative scheduling. The PS side is responsible for configuring the parameters of the convolution IP core layer by layer, managing the feature map cache address and controlling the data flow. The PL side loads the binary file and performs convolution calculation using a custom configurable convolution IP core. The verification module is used to read the raw data of the dual-scale detection head output from the FPGA on the PC, decode and post-process it according to the preset anchor frame, confidence threshold and non-maximum suppression threshold, and complete the evaluation and verification of the detection results.
9. An electronic device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, when the processor executes the computer program, it implements the deployment and verification method for a flame and smoke detection network as described in any one of claims 1 to 7.
10. A computer-readable storage medium storing computer-executable instructions for performing the deployment verification method of a flame and smoke detection network as described in any one of claims 1 to 7.
