Unmanned aerial vehicle pyrotechnic small target detection method, device and system

Abstract

The application discloses a kind of unmanned aerial vehicle pyrotechnic small target detection method, device and system.The method comprises: obtaining the bimodal image to be detected of fire monitoring area, and the bimodal image to be detected includes near-infrared image and visible light image;Through the improved YOLOv11s network of pre-trained extraction bimodal image in pyrotechnic suspected area of bimodal image to be detected, wherein the convolution block attention module is embedded in the main network and feature pyramid network of improved YOLOv11s network;Through the FireSmokeCNN double-branch convolution network of pre-trained extraction near-infrared feature and visible light feature in bimodal image of pyrotechnic suspected area, and near-infrared feature and visible light feature are high-dimensional feature fusion, to obtain the fusion feature vector of pyrotechnic suspected area, and according to the fusion feature vector pyrotechnic suspected area is carried out pyrotechnic small target detection.Thereby, the real-time performance and accuracy of pyrotechnic small target detection are improved.

Description

Technical Field

[0001] This application relates to the field of unmanned aerial vehicle (UAV) fire detection technology, and in particular to a method, device and system for detecting small smoke and fire targets carried by an UAV. Background Technology

[0002] With their advantages of high mobility, wide inspection range, and ability to reach complex spaces, drones have been widely used for early fire inspections in forests, industrial parks, industrial plants, and tall buildings. Early fire targets are mostly small objects such as weak flames and thin smoke, characterized by indistinct features and a small pixel ratio in the image. Furthermore, drone inspections are easily affected by factors such as lighting, background interference, and flight vibration, making the detection of small fire targets difficult.

[0003] Currently, most drone smoke and fire detection technologies use a single YOLO series model for target detection, which has the following technical shortcomings: First, the model is not optimized for small target features, resulting in weak feature capture capabilities for small smoke and fire targets and high rates of missed detections and false detections. Second, most detection methods use single-modal images, such as visible light or near-infrared images. Visible light image detection results are greatly affected by lighting conditions, and the detection performance is poor at night or in strong light environments. While near-infrared images can overcome the influence of lighting conditions, they provide limited feature information when used alone. Some solutions attempt to combine visible light and near-infrared dual-modal images for detection, but these often only perform simple image stitching and do not achieve deep cross-modal feature fusion, failing to fully utilize the complementary features of near-infrared and visible light. Third, the detection algorithms are not specifically optimized for the flight characteristics of drones and the limitations of on-device computing resources, resulting in poor compatibility with drone on-device hardware. Complex models are difficult to implement inference in real time on the drone, making it difficult to meet the real-time and accuracy requirements for drone smoke and fire target detection. Summary of the Invention

[0004] This application aims to at least partially address one of the technical problems in the related art. To this end, one objective of this application is to propose an unmanned aerial vehicle (UAV)-borne method, apparatus, and system for detecting small targets involving smoke and fire, thereby improving the real-time performance and accuracy of small target detection while maintaining a lightweight detection model.

[0005] In a first aspect, embodiments of this application propose an unmanned aerial vehicle (UAV) method for detecting small smoke targets. The method includes: acquiring a dual-modal image to be detected of a fire monitoring area, the dual-modal image to be detected including a near-infrared image and a visible light image; extracting a dual-modal image of a suspected smoke area from the dual-modal image to be detected using a pre-trained improved YOLOv11s network, wherein the backbone network and feature pyramid network of the improved YOLOv11s network are embedded with convolutional block attention modules, the convolutional block attention modules including channel attention branches and spatial attention branches, which are used to enhance the semantic features and pixel-level spatial features of the small smoke targets, respectively; extracting near-infrared features and visible light features from the dual-modal image of the suspected smoke area using a pre-trained FireSmokeCNN dual-branch convolutional network, and fusing the near-infrared features and the visible light features in a high-dimensional feature fusion to obtain a fused feature vector of the suspected smoke area; and detecting small smoke targets in the suspected smoke area based on the fused feature vector.

[0006] In some embodiments, the step of extracting the bimodal image of the suspected fireworks region in the bimodal image to be detected using a pre-trained improved YOLOv11s network includes: inputting the bimodal image to be detected into the pre-trained improved YOLOv11s detection model to obtain the detection box coordinates and a first confidence score of the suspected fireworks region; when the first confidence score is greater than a first preset confidence score, extracting the suspected fireworks region from the bimodal image to be detected based on the detection box coordinates to obtain the bimodal image of the suspected fireworks region.

[0007] In some embodiments, the FireSmokeCNN dual-branch convolutional network includes: a near-infrared feature extraction branch and a visible light feature extraction branch, each comprising four convolutional blocks connected in sequence and a convolutional block attention module. Each convolutional block includes a convolutional layer, a batch normalization (BN) layer, a ReLU activation function, and a MaxPool pooling layer connected in sequence; a cross-modal fusion layer for performing the high-dimensional feature fusion; and a classification output layer comprising a fully connected layer, a Dropout layer, and a Softmax activation function connected in sequence.

[0008] In some embodiments, the step of extracting near-infrared and visible light features from the bimodal image of the suspected fireworks region using a pre-trained FireSmokeCNN dual-branch convolutional network, fusing the near-infrared and visible light features in a high-dimensional feature fusion to obtain a fused feature vector for the suspected fireworks region, and performing small fire target detection in the suspected fireworks region based on the fused feature vector, includes: extracting near-infrared features from the near-infrared image of the suspected fireworks region using the near-infrared feature extraction branch of the FireSmokeCNN dual-branch convolutional network, and extracting visible light features from the visible light image of the suspected fireworks region using the visible light feature extraction branch of the FireSmokeCNN dual-branch convolutional network; fusing the results of the near-infrared and visible light feature extraction in a high-dimensional feature fusion layer of the FireSmokeCNN dual-branch convolutional network to obtain a fused feature vector for the suspected fireworks region; and performing small fire target detection in the suspected fireworks region based on the fused feature vector using the classification output layer of the FireSmokeCNN dual-branch convolutional network.

[0009] In some embodiments, the step of detecting small fire targets in the suspected fire area based on the fused feature vector through the classification output layer of the FireSmokeCNN dual-branch convolutional network includes: obtaining the fire type and second confidence level of the suspected fire area corresponding to the fused feature vector through the classification output layer of the FireSmokeCNN dual-branch convolutional network, wherein the fire type includes fire and smoke coexistence, pure smoke, pure flame, and no fire; when the fire type of the suspected fire area corresponding to the fused feature vector is fire and smoke coexistence, pure smoke, or pure flame, and the second confidence level is greater than a second preset confidence level, it is determined that there is a small fire target in the suspected fire area corresponding to the fused feature vector.

[0010] In some embodiments, the method further includes: constructing a dual-modal image dataset of small fireworks targets, the dataset including pairs of near-infrared images and visible light images, and labeling detection boxes for small fireworks targets in each image, the dataset also including background images without small fireworks targets as interference images; embedding convolutional block attention modules into the backbone network and feature pyramid network of the YOLOv11s network to obtain an improved YOLOv11s network; initializing the training parameters of the improved YOLOv11s network, using the CIoU loss function, and iteratively training the improved YOLOv11s network using the dual-modal image dataset of small fireworks targets until the model converges, to obtain the pre-trained improved YOLOv11s network.

[0011] In some embodiments, the method further includes: extracting suspected fireworks regions from the dual-modal image dataset of small fireworks targets, and labeling each suspected fireworks region with a fireworks type label to construct a suspected fireworks region classification dataset; initializing the training parameters of the FireSmokeCNN dual-branch convolutional network, using the cross-entropy loss function, and iteratively training the FireSmokeCNN dual-branch convolutional network using the suspected fireworks region classification dataset until the model converges, to obtain the pre-trained FireSmokeCNN dual-branch convolutional network.

[0012] Secondly, this application proposes an unmanned aerial vehicle (UAV)-borne small smoke target detection device. The device includes: an image acquisition module for acquiring a dual-modal image to be detected of a fire monitoring area, the dual-modal image to be detected including a near-infrared image and a visible light image; a suspected region extraction module for extracting a dual-modal image of a suspected smoke region from the dual-modal image to be detected using a pre-trained improved YOLOv11s network, wherein the backbone network and feature pyramid network of the improved YOLOv11s network are embedded with convolutional block attention modules, the convolutional block attention modules including channel attention branches and spatial attention branches, which are respectively used to enhance the semantic features and pixel-level spatial features of the small smoke target; and a detection module for extracting near-infrared features and visible light features from the dual-modal image of the suspected smoke region using a pre-trained FireSmokeCNN dual-branch convolutional network, fusing the near-infrared features and the visible light features in a high-dimensional feature fusion to obtain a fused feature vector of the suspected smoke region, and detecting small smoke targets in the suspected smoke region based on the fused feature vector.

[0013] Thirdly, this application proposes an unmanned aerial vehicle (UAV)-borne small target detection system for fireworks. The system is mounted on a rotary-wing UAV platform and includes: a dual-modal camera unit for acquiring near-infrared and visible light dual-modal images; an end-side computing unit electrically connected to the dual-modal camera unit, the end-side computing unit including a memory and a processor, the memory storing a computer program, which, when executed by the processor, implements the UAV-borne small target detection method for fireworks as described in the first aspect; and an early warning and communication unit electrically connected to the end-side computing unit for receiving the small target detection results for fireworks and issuing early warnings, and uploading the detection results to a background monitoring system.

[0014] In some embodiments, the edge computing unit is a Rockchip RV1126 core board with a built-in NPU neural network acceleration unit, used to perform INT8 format quantization on the improved YOLOv11s detection model and the FireSmokeCNN dual-branch convolutional network.

[0015] The UAV-borne small target detection method, apparatus, and system according to the embodiments of this application optimize the small target features by adding an attention mechanism to the YOLOv11s network, and achieves deep cross-modal feature fusion of visible light and near-infrared light through the FireSmokeCNN dual-branch convolutional network, making full use of the feature complementarity of near-infrared and visible light. By combining the improved YOLOv11s network with the FireSmokeCNN dual-branch convolutional network, the real-time performance and accuracy of small target detection of fireworks are improved while maintaining the lightweight nature of the detection model.

[0016] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0017] Figure 1 This is a flowchart of the UAV-borne small target detection method for smoke and fire according to an embodiment of this application; Figure 2 This is a flowchart illustrating the extraction of bimodal images of suspected fireworks areas using an improved YOLOv11s network, as described in an embodiment of this application. Figure 3 This is a flowchart of a process for detecting small fire targets in a suspected fire area based on a fused feature vector, according to an embodiment of this application. Figure 4 This is a flowchart illustrating the acquisition of a pre-trained improved YOLOv11s network according to an embodiment of this application; Figure 5 This is a flowchart illustrating the process of obtaining a pre-trained improved FireSmokeCNN dual-branch convolutional network according to an embodiment of this application; Figure 6 This is a block diagram of an unmanned aerial vehicle (UAV) carrying a small smoke and fire target detection device according to an embodiment of this application; Figure 7 This is a block diagram of an unmanned aerial vehicle (UAV)-borne small fire target detection system according to an embodiment of this application. Detailed Implementation

[0018] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0019] The following description, with reference to the accompanying drawings, describes an unmanned aerial vehicle (UAV) method, apparatus, and system for detecting small smoke and fire targets according to embodiments of this application.

[0020] Figure 1This is a flowchart of the UAV-borne small target detection method for smoke and fire according to an embodiment of this application.

[0021] In some embodiments of this application, reference is made to Figure 1 As shown, the method for detecting small targets such as smoke and fire carried by UAVs includes: S1, acquire dual-modal images of the fire monitoring area to be detected, including near-infrared images and visible light images.

[0022] Specifically, in this embodiment of the application, a dual-modal camera unit mounted on a rotary-wing UAV acquires dual-modal images of the fire monitoring area. The dual-modal camera unit includes a near-infrared camera and a visible light camera. The near-infrared camera has a detection wavelength of 8 to 14 micrometers and is used to acquire near-infrared images of the fire monitoring area. The visible light camera is a 2-megapixel high-definition wide-angle camera used to acquire visible light images of the fire monitoring area. The two cameras maintain frame rate synchronization to ensure temporal consistency of the acquired dual-modal images.

[0023] In addition, the acquired bimodal images need to be preprocessed. Specifically, in this embodiment of the application, Gaussian filtering is used to remove noise from the bimodal images, enhance the effective signal of the bimodal images, and Z-score standardization is used to eliminate the difference in the dimensions of pixel values ​​of the bimodal images, avoid gradient explosion caused by excessively large pixel values, stabilize the inference calculation process of the model, and uniformly scale the bimodal images to the same size to adapt to the input of subsequent models.

[0024] S2. A bimodal image of the suspected smoke region in the bimodal image to be detected is extracted by a pre-trained improved YOLOv11s network. The backbone network and feature pyramid network of the improved YOLOv11s network are embedded with convolutional block attention modules. The convolutional block attention modules include channel attention branches and spatial attention branches, which are used to enhance the semantic features and pixel-level spatial features of small smoke targets, respectively.

[0025] It should be noted that the YOLOv11s network is a lightweight general-purpose object detection model. In this embodiment of the application, a convolutional block attention module (CBAM) is embedded in the C3k2 module of the backbone network and the key layer of the PANet (PathAggregation Network) to improve the feature capture capability of the YOLOv11s network for small targets such as fireworks.

[0026] Specifically, the convolutional block attention module includes a channel attention branch and a spatial attention branch. The channel attention branch enhances the semantic features of small fireworks targets through average pooling, max pooling, and multilayer perceptron, and filters key feature channels. The spatial attention branch focuses on the pixel-level spatial features of small fireworks targets through 7x7 convolution, highlighting weak target areas.

[0027] Therefore, in this embodiment of the application, by embedding convolutional block attention modules in the C3k2 module of the backbone network of the YOLOv11s network and the key layers of the feature pyramid network, the feature capture of small fireworks targets is enhanced from both channel and spatial dimensions, so as to optimize the YOLOv11s network model for the features of small fireworks targets, thereby improving the accuracy of small fireworks target detection.

[0028] Furthermore, in some embodiments of this application, reference is made to... Figure 2 As shown, a bimodal image of the suspected smoke region in the bimodal image to be detected is extracted using a pre-trained improved YOLOv11s network, including: S21, Input the dual-modal image to be detected into the pre-trained improved YOLOv11s detection model to obtain the detection box coordinates and first confidence score of the suspected fireworks area; Specifically, in this embodiment of the application, after the preprocessed bimodal image to be detected is input into the pre-trained improved YOLOv11 network, the pre-trained improved YOLOv11 network will output the detection box coordinates and the first confidence score of the suspected smoke and fire area in the bimodal image to be detected. The detection box coordinates of the suspected smoke and fire area indicate the position of the suspected smoke and fire area in the bimodal image to be detected, and the first confidence score indicates the probability that there is a small smoke and fire target in the suspected smoke and fire area.

[0029] S22, when the first confidence level is greater than the first preset confidence level, the suspected smoke and fire region is extracted from the dual-modal image to be detected according to the detection box coordinates, so as to obtain the dual-modal image of the suspected smoke and fire region.

[0030] Specifically, in this embodiment of the application, when the first confidence level is greater than the first preset confidence level, it indicates that there are small fire targets in the suspected fire area. Therefore, the dual-modal image to be detected (including near-infrared image and visible light image) can be cropped according to the detection box coordinates to obtain the dual-modal image of the suspected fire area.

[0031] In addition, after acquiring the bimodal images of the suspected fireworks area, it is necessary to uniformly scale the bimodal images of the suspected fireworks area to the same size to adapt to the input of subsequent models.

[0032] S3 extracts near-infrared and visible light features from the dual-modal image of the suspected smoke area using a pre-trained FireSmokeCNN dual-branch convolutional network, and fuses the near-infrared and visible light features into high-dimensional features to obtain the fused feature vector of the suspected smoke area. Based on the fused feature vector, small smoke targets are detected in the suspected smoke area.

[0033] Specifically, in some embodiments of this application, the FireSmokeCNN dual-branch convolutional network includes a near-infrared feature extraction branch, a visible light feature extraction branch, a cross-modal fusion layer, and a classification output layer. The near-infrared feature extraction branch and the visible light feature extraction branch each include four convolutional blocks connected in sequence and a convolutional block attention module. Each convolutional block includes a convolutional layer, a batch normalization (BN) layer, a ReLU activation function, and a MaxPool pooling layer connected in sequence. The cross-modal fusion layer is used for high-dimensional feature fusion. The classification output layer includes a fully connected layer, a Dropout layer, and a Softmax activation function connected in sequence.

[0034] More specifically, in some embodiments of this application, after inputting the bimodal image of the suspected fireworks area after uniform size into the pre-trained FireSmokeCNN bimodal convolutional network, near-infrared features of the suspected fireworks area are extracted through the near-infrared feature extraction branch of the FireSmokeCNN bimodal convolutional network, and visible light features of the suspected fireworks area are extracted through the visible light feature extraction branch of the FireSmokeCNN bimodal convolutional network. Furthermore, near-infrared features and visible light features are fused in a high-dimensional manner through the cross-modal fusion layer of the FireSmokeCNN bimodal convolutional network to obtain the fused feature vector of the suspected fireworks area. Finally, small fireworks targets are detected in the suspected fireworks area based on the fused feature vector through the classification output layer of the FireSmokeCNN bimodal convolutional network.

[0035] Therefore, in this embodiment of the application, the near-infrared and visible light features of the dual-modal image of the suspected smoke and fire area are extracted by the pre-trained FireSmokeCNN dual-branch convolutional network, and the near-infrared and visible light features are fused in a high-dimensional manner. This enables deep cross-modal feature fusion of small smoke and fire targets, making full use of the complementary features of near-infrared image's resistance to illumination interference and visible light image's rich texture. This overcomes the limitations of single-modal image detection and maintains high detection accuracy in scenarios such as night, strong light, and complex backgrounds.

[0036] Furthermore, in some embodiments of this application, reference is made to... Figure 3As shown, the classification output layer of the FireSmokeCNN dual-branch convolutional network performs small fire target detection in suspected fire areas based on the fused feature vector, including: S31, through the classification output layer of the FireSmokeCNN dual-branch convolutional network, obtain the smoke type and second confidence of the suspected smoke area corresponding to the fused feature vector, where the smoke type includes smoke and fire coexisting, pure smoke, pure flame and no smoke; Specifically, in this embodiment of the application, the types of fireworks are divided into four categories based on the visual characteristics of the small fireworks targets: fireworks and smoke coexisting, pure smoke, pure flame, and no fireworks. The classification output layer of the FireSmokeCNN dual-branch convolutional network can obtain the fireworks type and second confidence level of the suspected fireworks area based on the fused feature vector. The second confidence level represents the probability that the fireworks type of the suspected fireworks area is the predicted type.

[0037] S32, when the type of smoke and fire in the suspected smoke and fire area corresponding to the fused feature vector is smoke and fire coexisting, pure smoke or pure flame, and the second confidence level is greater than the second preset confidence level, it is determined that there is a small smoke and fire target in the suspected smoke and fire area corresponding to the fused feature vector.

[0038] Specifically, in this embodiment of the application, when the type of smoke and fire in the suspected smoke and fire area corresponding to the fused feature vector is smoke and fire coexisting, pure smoke, or pure flame, and the second confidence level is greater than the second preset confidence level, it indicates that there is a small smoke and fire target in the suspected smoke and fire area. At this time, the rotary-wing UAV will issue an audible and visual warning and hover over the target area for continuous monitoring. At the same time, the rotary-wing UAV will transmit the detection results such as the type, location, and confidence level of the small smoke and fire target to the background monitoring system so that staff can detect and extinguish the smoke and fire in a timely manner.

[0039] Furthermore, in some embodiments of this application, reference is made to... Figure 4 As shown, the pre-trained improved YOLOv11s network is obtained through the following steps: S41. Construct a dual-modal image dataset of small fireworks targets. The dataset includes pairs of near-infrared images and visible light images, and a detection box is marked for the small fireworks targets in each image. The dataset also includes background images without small fireworks targets as interference images.

[0040] Specifically, in this embodiment of the application, a rotary-wing UAV is used to inspect the fire monitoring area and collect paired near-infrared and visible light images (including images of small fire targets such as early weak flames and thin smoke, as well as background interference images such as sunlight reflection, fluorescent lights, car taillights, and tree shadows). The detection boxes of small fire targets in each image are manually labeled to construct a dual-modal image dataset of small fire targets.

[0041] In addition, data augmentation processing such as image flipping, random scaling, brightness and contrast adjustment, and Gaussian noise addition is required on the dataset to improve the model's generalization ability.

[0042] S42, embedding convolutional block attention modules into the backbone network and feature pyramid network of the YOLOv11s network, resulting in an improved YOLOv11s network.

[0043] Specifically, in this embodiment of the application, CBAM is embedded in the C3k2 module of the YOLOv11s backbone network and the key downsampling and upsampling layers of the feature pyramid network. CBAM includes a channel attention branch and a spatial attention branch. The channel attention branch enhances the semantic features of small fireworks targets through average pooling, max pooling, and an MLP network, filtering key feature channels. The spatial attention branch focuses on the pixel-level spatial features of small fireworks targets through 7x7 convolutions, highlighting weak target regions. Thus, without changing the overall lightweight architecture of the YOLOv11s network, the small target feature capture capability of the YOLOv11s network is enhanced.

[0044] S43. Initialize the training parameters of the improved YOLOv11s network, and use the CIoU loss function. Iteratively train the improved YOLOv11s network using the fireworks small target bimodal image dataset until the model converges to obtain the pre-trained improved YOLOv11s network.

[0045] Specifically, in this embodiment of the application, the enhanced dual-modal image dataset of small fireworks targets is divided into a training set, a validation set, and a test set in a 7:2:1 ratio. An initial learning rate of 0.001 is set, and a cosine annealing strategy is used to adjust the learning rate. The batch size for each training iteration is set to 16, and the number of iterations is 300. The CIoU loss function is used as the bounding box regression loss function, and the cross-entropy loss function is used as the classification loss. The improved YOLOv11s network is iteratively trained using the dual-modal image dataset of small fireworks targets until the model converges, thus obtaining a pre-trained improved YOLOv11s network.

[0046] Furthermore, in some embodiments of this application, reference is made to... Figure 5 As shown, the pre-trained FireSmokeCNN dual-branch convolutional network is obtained through the following steps: S51, extract suspected fireworks regions from the dual-modal image dataset of small fireworks targets, and label each suspected fireworks region with a fireworks type label to construct a suspected fireworks region classification dataset; Specifically, in this embodiment of the application, suspected fireworks regions are cropped from the dual-modal image dataset of small fireworks targets based on manually labeled fireworks target detection boxes, and each suspected fireworks region is labeled with a fireworks type label. Then, the suspected fireworks regions are scaled to a uniform size to complete the classification dataset of suspected fireworks regions.

[0047] S52, initialize the training parameters of the FireSmokeCNN dual-branch convolutional network, and use the cross-entropy loss function and the FireSmokeCNN dual-branch convolutional network to iteratively train it until the model converges, so as to obtain the pre-trained FireSmokeCNN dual-branch convolutional network.

[0048] Specifically, in this embodiment of the application, the suspected fireworks region classification dataset is divided into a training set, a validation set, and a test set in an 8:1:1 ratio. The initial learning rate is set to 0.0001, the batch size for each training iteration is 32, and the number of iterations is 200. The cross-entropy loss function is used as the model's loss function, and the network parameters are updated using the Adam optimizer. The FireSmokeCNN dual-branch convolutional network is iteratively trained using the suspected fireworks region classification dataset until the model converges, thus obtaining a pre-trained FireSmokeCNN dual-branch convolutional network.

[0049] It should be noted that before training begins, the He initialization strategy is used to assign values ​​to all convolutional layer parameters of the FireSmokeCNN dual-branch convolutional network to ensure the convergence of network training. During training, the classification accuracy of the model is monitored using a validation set. If the accuracy on the validation set does not improve for 20 consecutive rounds, the model can be considered to have converged.

[0050] In summary, the UAV-borne small target detection method for fireworks in this application optimizes small target features by adding an attention mechanism to the YOLOv11s network and using a FireSmokeCNN dual-branch convolutional network to achieve deep cross-modal feature fusion of visible light and near-infrared light. This fully utilizes the complementary features of near-infrared images' resistance to illumination interference and visible light images' rich texture. Furthermore, by combining the improved YOLOv11s network with the FireSmokeCNN dual-branch convolutional network, and employing a two-stage architecture that first detects suspected regions using the improved YOLOv11s network and then accurately identifies them using the FireSmokeCNN dual-branch convolutional network, the method improves the real-time performance and accuracy of small target detection for fireworks while maintaining the lightweight nature of the detection model.

[0051] Figure 6 This is a block diagram of an unmanned aerial vehicle (UAV)-borne small target detection device for smoke and fire, according to an embodiment of this application.

[0052] In some embodiments of this application, reference is made to Figure 6 As shown, the UAV-borne small target detection device 800 for smoke and fire includes: an image acquisition module 10, a suspected area extraction module 20, and a detection module 30.

[0053] The system includes an image acquisition module 10, which acquires a dual-modal image of the fire monitoring area, comprising near-infrared and visible light images; a suspected region extraction module 20, which extracts a dual-modal image of a suspected smoke and fire area from the dual-modal image using a pre-trained improved YOLOv11s network, wherein the backbone and feature pyramid of the improved YOLOv11s network are embedded with convolutional block attention modules, which include channel attention branches and spatial attention branches, respectively used to enhance the semantic features and pixel-level spatial features of small smoke and fire targets; and a detection module 30, which extracts near-infrared and visible light features from the dual-modal image of the suspected smoke and fire area using a pre-trained FireSmokeCNN dual-branch convolutional network, performs high-dimensional feature fusion of the near-infrared and visible light features to obtain a fused feature vector of the suspected smoke and fire area, and performs small smoke and fire target detection in the suspected smoke and fire area based on the fused feature vector.

[0054] Furthermore, the suspected region extraction module 20 is specifically used to input the bimodal image to be detected into the pre-trained improved YOLOv11s detection model to obtain the detection box coordinates and first confidence level of the suspected fireworks region; when the first confidence level is greater than the first preset confidence level, the suspected fireworks region is extracted from the bimodal image to be detected based on the detection box coordinates to obtain the bimodal image of the suspected fireworks region.

[0055] Furthermore, the detection module 30 is specifically used to extract near-infrared features from the near-infrared image of the suspected smoke area through the near-infrared feature extraction branch of the FireSmokeCNN dual-branch convolutional network, and to extract visible light features from the visible light image of the suspected smoke area through the visible light feature extraction branch of the FireSmokeCNN dual-branch convolutional network; to perform high-dimensional feature fusion of near-infrared features and visible light features through the cross-modal fusion layer of the FireSmokeCNN dual-branch convolutional network to obtain a fused feature vector of the suspected smoke area; and to perform small smoke target detection in the suspected smoke area based on the fused feature vector through the classification output layer of the FireSmokeCNN dual-branch convolutional network.

[0056] Furthermore, the detection module 30 is also used to obtain the smoke type and second confidence level of the suspected smoke area corresponding to the fused feature vector through the classification output layer of the FireSmokeCNN dual-branch convolutional network. The smoke type includes smoke and fire coexisting, pure smoke, pure flame, and no smoke. When the smoke type of the suspected smoke area corresponding to the fused feature vector is smoke and fire coexisting, pure smoke, or pure flame, and the second confidence level is greater than the second preset confidence level, it is determined that there is a small smoke target in the suspected smoke area corresponding to the fused feature vector.

[0057] It should be understood that the specific implementation of the UAV-borne small target detection device 800 of this application can be referred to the specific implementation of the data feedback method of the intelligent driving system in the aforementioned embodiments of this application. In order to reduce redundancy, it will not be described again here.

[0058] Figure 7 This is a block diagram of an unmanned aerial vehicle (UAV)-borne small fire target detection system according to an embodiment of this application.

[0059] In some embodiments of this application, reference is made to Figure 7 As shown, the UAV-borne small target detection system 1000 includes: a dual-modal camera unit 100 for acquiring near-infrared and visible light dual-modal images, including a near-infrared camera, a visible light camera, and a three-axis anti-shake gimbal. The gimbal drives the infrared camera and the visible light camera to achieve 360° image acquisition without blind spots; an end-side computing unit 200 electrically connected to the dual-modal camera unit 100, the end-side computing unit 200 including a memory and a processor, the memory storing a computer program, and when the computer program is executed by the processor, it implements the aforementioned UAV-borne small target detection method; and an early warning and communication unit 300 electrically connected to the end-side computing unit 200, for receiving the small target detection results and issuing early warnings, and uploading the detection results to the background monitoring system 700.

[0060] Furthermore, in some embodiments of this application, the edge computing unit 200 is a Rockchip RV1126 core board with a built-in NPU neural network acceleration unit, used to perform INT8 format quantization on the improved YOLOv11s detection model and the FireSmokeCNN dual-branch convolutional network.

[0061] Specifically, in this embodiment of the application, by embedding an NPU neural network acceleration unit in the edge computing unit 200, the improved YOLOv11s detection model and the FireSmokeCNN dual-branch convolutional network are quantized using the RKNN model, and the model is converted into INT8 format, thereby improving the edge inference speed, meeting the real-time requirements of UAV inspection, and solving the problem of adapting complex algorithms to edge computing resources.

[0062] Furthermore, in some embodiments of this application, reference is made to Figure 7 As shown, the UAV-borne small target detection system 1000 also includes: a flight control unit 400, electrically connected to the end-side computing unit 200, used to receive detection results and control the flight status of the rotary-wing UAV 1100, while providing flight attitude and position information to each unit; an obstacle avoidance unit 500, used to perceive obstacle information in the environment around the rotary-wing UAV 1100 in real time, and send obstacle avoidance control commands to the flight control unit 400 to realize dynamic autonomous obstacle avoidance during the flight of the rotary-wing UAV 1100, ensuring the safety of inspection operations; and a power supply unit 600, used to provide stable DC power to the dual-mode camera unit, the end-side computing unit, the flight control unit, and the early warning and communication unit.

[0063] In summary, the UAV-borne small fire target detection system 1000 of this application has each hardware unit deeply adapted to the small fire target detection algorithm. It utilizes the dual-modal camera unit 100 to ensure the temporal and spatial consistency of the acquired dual-modal images, utilizes the end-side computing unit 200 to realize the efficient operation of the small fire target detection algorithm, and uses the flight control unit 400 and the early warning and communication unit 300 to realize real-time feedback and remote monitoring of the detection results. The system has strong applicability and can be directly applied to engineering practice.

[0064] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0065] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0066] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0067] In the description of this application, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise", "axial", "radial", "circumferential", etc., indicating the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this application.

[0068] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0069] In this application, unless otherwise expressly specified and limited, the terms "installation," "connection," "joining," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components, unless otherwise expressly limited. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0070] In this application, unless otherwise expressly specified and limited, "above" or "below" the second feature can mean that the first feature is in direct contact with the second feature, or that the first feature is in indirect contact with the second feature through an intermediate medium. Furthermore, "above," "on top of," and "over" the second feature can mean that the first feature is directly above or diagonally above the second feature, or simply that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature can mean that the first feature is directly below or diagonally below the second feature, or simply that the first feature is at a lower horizontal level than the second feature.

[0071] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A method for detecting small smoke and fire targets carried by an unmanned aerial vehicle (UAV), characterized in that, The method includes: Acquire dual-modal images of the fire monitoring area, the dual-modal images of the fire monitoring area including near-infrared images and visible light images; The bimodal image of the suspected smoke region in the bimodal image to be detected is extracted by a pre-trained improved YOLOv11s network. The backbone network and feature pyramid network of the improved YOLOv11s network are embedded with convolutional block attention modules. The convolutional block attention modules include channel attention branches and spatial attention branches, which are used to enhance the semantic features and pixel-level spatial features of small smoke targets, respectively. Near-infrared and visible light features are extracted from the bimodal image of the suspected fireworks area using a pre-trained FireSmokeCNN dual-branch convolutional network. The near-infrared and visible light features are then fused in a high-dimensional manner to obtain a fused feature vector of the suspected fireworks area. Small fireworks targets are then detected in the suspected fireworks area based on the fused feature vector.

2. The method for detecting small smoke and fire targets carried by an unmanned aerial vehicle according to claim 1, characterized in that, The step of extracting the bimodal image of the suspected smoke region in the bimodal image to be detected using a pre-trained improved YOLOv11s network includes: The dual-modal image to be detected is input into the pre-trained improved YOLOv11s detection model to obtain the detection box coordinates and first confidence level of the suspected fireworks area; When the first confidence level is greater than the first preset confidence level, the suspected smoke and fire region is extracted from the bimodal image to be detected based on the detection box coordinates to obtain the bimodal image of the suspected smoke and fire region.

3. The method for detecting small smoke and fire targets carried by an unmanned aerial vehicle according to claim 1, characterized in that, The FireSmokeCNN dual-branch convolutional network includes: The near-infrared feature extraction branch and the visible light feature extraction branch each include four convolutional blocks connected in sequence and a convolutional block attention module. Each convolutional block includes a convolutional layer, a BN layer, a ReLU activation function and a MaxPool pooling layer connected in sequence. A cross-modal fusion layer is used to perform the high-dimensional feature fusion; The classification output layer consists of a fully connected layer, a Dropout layer, and a Softmax activation function connected in sequence.

4. The method for detecting small smoke and fire targets carried by an unmanned aerial vehicle according to claim 3, characterized in that, The process involves extracting near-infrared and visible light features from the bimodal image of the suspected fireworks area using a pre-trained FireSmokeCNN dual-branch convolutional network, fusing the near-infrared and visible light features in a high-dimensional manner to obtain a fused feature vector for the suspected fireworks area, and performing small fire target detection in the suspected fireworks area based on the fused feature vector. Near-infrared features are extracted from the near-infrared image of the suspected fireworks area through the near-infrared feature extraction branch of the FireSmokeCNN dual-branch convolutional network, and visible light features are extracted from the visible light image of the suspected fireworks area through the visible light feature extraction branch of the FireSmokeCNN dual-branch convolutional network. The near-infrared features and the visible light features are fused in a high-dimensional manner through the cross-modal fusion layer of the FireSmokeCNN dual-branch convolutional network to obtain the fused feature vector of the suspected smoke region. The classification output layer of the FireSmokeCNN dual-branch convolutional network is used to detect small fire targets in the suspected fire area based on the fused feature vector.

5. The method for detecting small smoke and fire targets carried by an unmanned aerial vehicle according to claim 4, characterized in that, The step of detecting small fire targets in the suspected fire area using the classification output layer of the FireSmokeCNN dual-branch convolutional network based on the fused feature vector includes: The classification output layer of the FireSmokeCNN dual-branch convolutional network is used to obtain the smoke type and second confidence level of the suspected smoke region corresponding to the fused feature vector, wherein the smoke type includes smoke and fire coexistence, pure smoke, pure flame and no smoke; If the type of smoke and fire in the suspected smoke and fire area corresponding to the fused feature vector is smoke and fire coexisting, pure smoke, or pure flame, and the second confidence level is greater than the second preset confidence level, it is determined that there is a small smoke and fire target in the suspected smoke and fire area corresponding to the fused feature vector.

6. The method for detecting small smoke and fire targets carried by an unmanned aerial vehicle according to claim 1, characterized in that, The method further includes: A dual-modal image dataset of small fireworks targets is constructed. The dataset includes pairs of near-infrared images and visible light images, and detection boxes are marked for small fireworks targets in each image. The dataset also includes background images without small fireworks targets as interference images. By embedding convolutional block attention modules into the backbone network and feature pyramid network of the YOLOv11s network, an improved YOLOv11s network is obtained. The training parameters of the improved YOLOv11s network are initialized, and the CIoU loss function is adopted. The improved YOLOv11s network is iteratively trained using the dual-modal image dataset of small fireworks targets until the model converges, so as to obtain the pre-trained improved YOLOv11s network.

7. The method for detecting small smoke and fire targets carried by an unmanned aerial vehicle according to claim 6, characterized in that, The method further includes: Suspected fire areas are extracted from the dual-modal image dataset of small fire targets, and each suspected fire area is labeled with a fire type label to construct a suspected fire area classification dataset. The training parameters of the FireSmokeCNN dual-branch convolutional network are initialized, and the cross-entropy loss function is adopted. The FireSmokeCNN dual-branch convolutional network is iteratively trained using the suspected fireworks region classification dataset until the model converges, so as to obtain the pre-trained FireSmokeCNN dual-branch convolutional network.

8. A drone-borne smoke and fire small target detection device, characterized in that, The device includes: The image acquisition module is used to acquire dual-modal images to be detected in the fire monitoring area, including near-infrared images and visible light images; The suspected region extraction module is used to extract the bimodal image of the suspected smoke region in the bimodal image to be detected by using a pre-trained improved YOLOv11s network. The backbone network and feature pyramid network of the improved YOLOv11s network are embedded with convolutional block attention modules. The convolutional block attention modules include channel attention branches and spatial attention branches, which are used to enhance the semantic features and pixel-level spatial features of small smoke targets, respectively. The detection module is used to extract near-infrared and visible light features from the bimodal image of the suspected fireworks area using a pre-trained FireSmokeCNN dual-branch convolutional network, and to fuse the near-infrared and visible light features into a high-dimensional feature vector to obtain the fused feature vector of the suspected fireworks area, and to detect small fireworks targets in the suspected fireworks area based on the fused feature vector.

9. A UAV-borne smoke and fire small target detection system, characterized in that, The UAV-borne smoke and fire small target detection system is mounted on a rotary-wing UAV platform and includes: A dual-modal camera unit is used to acquire near-infrared and visible light dual-modal images; An edge computing unit is electrically connected to the dual-modal camera unit. The edge computing unit includes a memory and a processor. The memory stores a computer program. When the computer program is executed by the processor, it implements the UAV-borne small target detection method for smoke and fire as described in any one of claims 1-5. The early warning and communication unit is electrically connected to the end-side computing unit and is used to receive the detection results of small fire targets and issue early warnings, and upload the detection results to the background monitoring system.

10. The UAV-borne smoke and fire small target detection system according to claim 9, characterized in that, The edge computing unit is a Rockchip RV1126 core board with a built-in NPU neural network acceleration unit, used to perform INT8 format quantization on the improved YOLOv11s detection model and the FireSmokeCNN dual-branch convolutional network.

Images