Fire point detection and area estimation method for unmanned aerial vehicle inspection

Abstract

The invention discloses a fire point detection and area estimation method for unmanned aerial vehicle inspection, and relates to the technical field of artificial intelligence. Comprising the following steps: S1, preparing a data set of fire points, people and vehicles; s2, on the basis of an improved lightweight instance segmentation algorithm MobileNetV3-YOLOv8-Seg, the data set is trained, and a target detection model is obtained; s3, converting the target detection model into a target segmentation model; s4, the target segmentation model is called through a reasoning program to carry out target detection on continuous image frames shot by the unmanned aerial vehicle, and masks and rectangular marking boxes of fire points, people and vehicle targets are obtained; and S5, calculating the area of the fire point in the actual scene according to the mask and the rectangular marking frame box which are obtained through detection. According to the method, a lightweight target segmentation network model is realized, the model has the advantages of high reasoning efficiency, low resource occupancy rate, high detection precision, strong small target detection capability and the like, and a thread adaptive method is proposed to realize high-performance accelerated reasoning in a domestic embedded platform RK3588.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a fire point detection and area estimation method for unmanned aerial vehicle inspection. Background Art

[0002] Fire is one of the most serious disasters. Indoor fires can usually be monitored by smoke sensors, temperature sensors, etc., but outdoor fires usually have complex and changeable terrain, high concealment of fire points, and wide coverage, which makes them difficult to extinguish. In particular, forest fires have hidden fire points and are difficult to detect. Therefore, achieving real-time monitoring of large-scale fires and accurate estimation of fire area is of great significance for early fire prevention, real-time rescue, and resource allocation.

[0003] Traditional fire monitoring and fire coverage estimation mainly rely on manual surveys, satellite remote sensing and other methods. Among them, the satellite remote sensing method needs to transmit the image back to the ground and then detect it through the fire line image segmentation method. It has problems such as insufficient real-time efficiency, poor preventiveness, and high cost. It is also easy to sharply reduce the detection effect in the harsh environment of rain, snow and fog. At the same time, with the advancement of drone technology and artificial intelligence technology, various types of payloads are loaded in the optoelectronic pod of the drone, which can obtain high-resolution video image data on the ground in real time, so that drones gradually have the ability to obtain fire point images during inspections and transmit them back to the ground for detection and identification.

[0004] In the related technology, Chinese patent 202410589271.9, named "Forest fire monitoring system and method based on drone vision", is mainly based on the flame image segmentation method, with a low detection rate, weak real-time and practical performance, and too much reliance on the flame data set for accuracy. When the effect is not obvious in the early stage of the fire, it will cause a high false alarm rate. At the same time, in the related technology, only the single target of the fire point or flame is detected, and no moving targets such as people and vehicles are detected. The application of intelligent computing power is not sufficient, which is not conducive to subsequent rescue and early warning.

[0005] Moreover, traditional lightweight fire detection algorithms rely on NVIDIA's series of high-performance training cards. The generated models are difficult to adapt to the operation of low-power domestically produced independent and controllable AI chips, and have a low degree of autonomy and controllability and insufficient generalizability.

[0006] At the same time, fire point detection based on traditional image processing technology is mainly achieved based on the texture and color of the image, and relies on manual feature extraction, resulting in problems such as low detection accuracy and poor real-time performance. Summary of the invention

[0007] In view of the above problems, the present invention provides a fire point detection and area estimation method for UAV inspection, which can autonomously realize environmental perception and obtain the fire point location, and can quickly estimate the fire point coverage area.

[0008] The technical solution of the present invention is: a fire point detection and area estimation method for drone inspection, comprising the following steps:

[0009] S1: Preparation of datasets of fire points, people, and vehicles;

[0010] S2: Based on the improved lightweight instance segmentation algorithm MobileNetV3-YOLOv8-Seg, the dataset is trained to obtain the target detection model;

[0011] S3: Convert the object detection model into an object segmentation model;

[0012] S4: The target segmentation model is called through the inference program to perform target detection on the continuous image frames taken by the drone, and the masks and rectangular annotation boxes of the fire points, people and vehicles are obtained;

[0013] S5: Calculate the area of ​​the fire point in the actual scene based on the mask and rectangular annotation box obtained by detection.

[0014] In step S1, various image data with fire points, people, and vehicles taken from the perspective of drones are collected, and the fire points, people, and vehicle areas are labeled using the Labelme image annotation tool. A corresponding json format annotation file is generated for each image. After the annotation is completed, all annotation files are converted into coco format annotation files to form a data set.

[0015] In step S2, the improved lightweight instance segmentation algorithm MobileNetV3-YOLOv8-Seg includes:

[0016] The backbone network of the YOLOv8-Seg algorithm uses the lightweight MobileNetV3 network suitable for mobile deployment, and the ultra-lightweight dynamic upsampling operator DySample is used in the neck network; at the same time, the spatial context perception module SCAM is introduced, combining the spatial attention and channel attention mechanisms;

[0017] After training is completed, the trained model file is converted and exported into the *.onnx intermediate model conversion format.

[0018] In step S3, the domestic embedded intelligent chip RK3588 is used as the drone image processing module, and the model conversion tool rknn-toolkit2 is used to convert the *.onnx format model into a target segmentation model in the *.rknn format that is suitable for the RK3588 platform.

[0019] In step S4,

[0020] The reasoning program is divided into three parts: data preprocessing, model reasoning, and post-processing of reasoning results.

[0021] in,

[0022] The data preprocessing part scales, fills, transforms channels, normalizes, and upgrades the acquired image frames to obtain input data of size [1,3,640,640];

[0023] The model inference part uses the output of the data preprocessing part as the input of the *.rknn model, and obtains a set of unprocessed basic masks and rectangular boxes after performing inference;

[0024] The post-processing part includes two steps: first, non-maximum suppression is performed on the rectangular box, and the remaining rectangular box is decoded into a rectangular annotation box corresponding to the original image; then the mask corresponding to the original image is obtained by combining the basic masks.

[0025] The reasoning program uses a thread-adaptive approach, including:

[0026] Data preprocessing and model inference are cached through thread queues and passed to subsequent processing; multiple model inference threads are used to process queue cache data in parallel; at the same time, after data preprocessing is completed, the processed data and the serial number combination corresponding to the image frame are stored in the preprocessing queue, and the model inference thread and the inference result post-processing thread both obtain data from the queue in sequence, and store the processed data in the inference queue and the post-processing queue in sequence.

[0027] The thread adaptive allocation method includes: generating random data of the same size as the data captured by the camera as input;

[0028] After starting the inference program, the computational time of data preprocessing, model inference, and inference result post-processing can be obtained, which are t1, t2, and t3 respectively;

[0029] Compare and obtain the minimum value among t1, t2, and t3, take the minimum value, and then calculate the ratio of the remaining time consumption to the minimum value and round up.

[0030] In step S5, according to the mask mask and the rectangular annotation box obtained by detection, the ratio between the number of pixels occupied by the fire point target in the image and the number of all pixels in the rectangular annotation box is calculated;

[0031] Then, based on the similarity principle, the width and height of the rectangular area in the actual scene corresponding to the detection frame are calculated in combination with the camera's pixel size, focal length and the distance value obtained by laser ranging; combined with the obtained ratio relationship, the area of ​​the fire point in the actual scene is calculated.

[0032] Step S5 specifically includes:

[0033] a) Use OpenCV's findContours interface to process the mask to obtain the contour information of the fire point target, and use OpenCV's contourArea interface to obtain the total number of pixels occupied by the fire point target. fire ;

[0034] b) Calculate the number of pixels in the rectangular box based on the obtained box box ; Get the ratio between the number of pixels occupied by the fire point and the number of all pixels in the detection frame, as follows:

[0035]

[0036] c) Based on the distance D obtained by the drone laser ranging module, as well as the camera’s inherent parameters of focal length f and pixel size p pixel , combined with the above-obtained Ratio, calculate the length L and width W of the actual range corresponding to the box with a length of l and a width of w. The formula is as follows:

[0037]

[0038] d) Obtain the actual area S of the fire point area based on the obtained length L, width W and ratio Ratio fire =L*W*Ratio.

[0039] In the work, the present invention comprehensively considers the computing, memory and communication limited scenarios of small wing or fixed-wing UAVs, takes lightweight, modular, adaptable and customizable as the overall design principles, and adopts a lightweight MobileNetV3 feature extraction model as the backbone network of the YOLOv8-Seg algorithm, replaces the upsampling UpSample module in the neck network with an ultra-lightweight dynamic upsampling operator DySample, introduces a spatial context perception module SCAM and other improvement measures to realize a lightweight target segmentation network model. The model has the advantages of high reasoning efficiency, low resource occupancy, high detection accuracy and strong small target detection capability, and proposes a thread adaptive method to realize high-performance accelerated reasoning in the domestic embedded platform RK3588.

[0040] Next, based on the mask and rectangular annotation box of the fire point target output by the target segmentation network, combined with the camera parameters and laser ranging values, the fire point area is calculated by the similarity principle. In this step, only the segmented part of the fire point image needs to be processed, which has a lower amount of calculation than processing the entire image, so it has the advantages of being simple and efficient. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the specific implementation of the present invention or the technical solution in the prior art, the following briefly introduces the drawings required for the specific implementation or the prior art description. In the drawings, each part is not necessarily drawn according to the actual scale.

[0042] Figure 1 is a flow chart of the method of the present invention,

[0043] Figure 2 This is the structure diagram of the improved lightweight instance segmentation algorithm MobilNetV3-YOLOv8-Seg.

[0044] Figure 3 It is a schematic diagram of adaptive threading.

[0045] Figure 4 It is a schematic diagram of flame area estimation. DETAILED DESCRIPTION

[0046] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0047] In the description of the present invention, it should be understood that the terms "upper", "lower", "left", "right", "vertical", "horizontal", etc. indicate positions or positional relationships based on the positions or positional relationships shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific position, be constructed and operated in a specific position, and therefore cannot be understood as limiting the present invention. In the description of the present invention, unless otherwise specified, "plurality" means two or more.

[0048] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0049] The present invention Figure 1 A method for fire point detection and area estimation for drone inspection is provided, comprising the following steps:

[0050] S1: Preparation of datasets of fire points, people, and vehicles;

[0051] S2: Based on the improved lightweight instance segmentation algorithm MobileNetV3-YOLOv8-Seg, the dataset is trained to obtain the target detection model;

[0052] S3: Convert the target detection model into a target segmentation model;

[0053] S4: The target segmentation model is called through the inference program to perform target detection on the continuous image frames taken by the drone, and the masks and rectangular annotation boxes of the fire points, people and vehicles are obtained;

[0054] S5: Calculate the area of ​​the fire point in the actual scene based on the mask and rectangular annotation box obtained by detection.

[0055] The present invention firstly improves the YOLOv8-Seg model, replaces the backbone network and upsampling module with a lightweight MobileNetV3 feature extraction network and an ultra-lightweight dynamic upsampling operator DySample, and introduces a spatial context perception module SCAM according to the small sample characteristics, thereby realizing a lightweight and high-precision MobileNetV3-YOLOv8-Seg fire point detection model, and realizing high-performance accelerated reasoning in the RK3588 embedded platform through a thread adaptive method; then, the estimated area of ​​the fire point is obtained according to the similarity principle by combining the camera parameters and the UAV laser ranging values.

[0056] The present invention fully considers the scenarios in which computing, memory and network communication resources are limited on domestic embedded platforms, does not rely on the ground control center, and implements terminal model development and optimization based on a lightweight computing framework, thereby reducing equipment costs, minimizing resource consumption, and improving model operation performance.

[0057] The method proposed in the present invention can quickly and accurately locate the positions of fire points, people, vehicles and other targets, and segment the fire point area, effectively reducing the error in fire point area estimation. It has the advantages of high accuracy, real-time and high efficiency, and can provide strong support for the timely discovery, precise positioning and subsequent rescue response of fires.

[0058] In application, specifically,

[0059] S1: Collect various image data with fire points, people, and vehicles taken from the perspective of drones, and use the Labelme image annotation tool to annotate the fire points, people, and vehicle areas. Generate a corresponding json format annotation file for each image. After the annotation is completed, convert all annotation files into coco format annotation files, and divide the data set into training set and validation set in a ratio of 7:3.

[0060] S2: Based on the improved lightweight instance segmentation algorithm MobileNetV3-YOLOv8-Seg, the dataset is trained. The lightweight computing framework is used for modeling. The framework integrates high-performance basic operators optimized and tailored for embedded terminal platforms, involving neural network operators, convolution operators, and custom operators, and can provide high-ratio compression tools for GPU and NPU. During the training process, the training efficiency is improved by loading the MobileNetV3 pre-trained model and data parallelism. After the training is completed, the trained model file is converted and exported to the *.onnx intermediate model conversion format.

[0061] The MobileNetV3-YOLOv8-Seg algorithm is an improvement based on the YOLOv8-Seg algorithm. The model structure is as follows Figure 2 As shown. In order to enable the model to achieve high-performance deployment and application on resource-constrained embedded platforms, the backbone network of the YOLOv8-Seg algorithm adopts the MobileNetV3 lightweight network suitable for mobile deployment, and replaces the upsampling UpSample module in the neck network with the ultra-lightweight dynamic upsampling operator DySample, which effectively improves the feature extraction efficiency and upsampling speed, and reduces the computational complexity and parameter amount of the overall network; at the same time, the spatial context perception module SCAM is introduced to improve the attention to small target areas by combining spatial attention and channel attention, so as to enhance the network's detection ability for small targets.

[0062] In step S2, a lightweight feature extraction network is developed to adapt to domestic low-power embedded platforms; the detection and segmentation algorithms are integrated; and the attention mechanism is introduced to capture small target fire points.

[0063] S3: The domestic embedded intelligent chip (RK3588) is used as the drone image processing module, and the model conversion tool rknn-toolkit2 is used to convert the *.onnx format model into a target segmentation model in the *.rknn format that is suitable for the RK3588 platform.

[0064] S4: In the RK3588 platform, the *.rknn format target segmentation model is called through the inference program to perform target detection on the continuous image frames taken by the drone, and the masks and rectangular annotation boxes of the fire points, people and vehicles are obtained;

[0065] The inference program is divided into three parts: data preprocessing, model inference, and inference result post-processing. The content that needs to be implemented in each part is as follows:

[0066] 1) The data preprocessing part is used to scale, fill, transform channels, normalize and upgrade the acquired image frames to obtain input data of size [1,3,640,640];

[0067] 2) The model inference part is used to use the output of the data preprocessing part as the input of the *.rknn model, and obtain a set of unprocessed basic masks and rectangular boxes after performing inference;

[0068] 3) The post-processing part includes two steps: first, non-maximum suppression is performed on the rectangular boxes to remove the rectangular boxes with high overlap, and the remaining rectangular boxes are decoded into rectangular annotation boxes corresponding to the original image; then, the mask corresponding to the original image is obtained by combining the basic masks.

[0069] The model reasoning program innovatively proposes a thread adaptive method, which can improve computing efficiency through parallel computing. The specific implementation process is as follows: Figure 3 shown.

[0070] Data preprocessing and model inference are cached through thread queues and passed to subsequent steps for processing. Since data preprocessing takes much less time than model inference, multiple model inference threads are required to process queue cache data in parallel. At the same time, considering the image sequence problem in the video stream processing process, after data preprocessing is completed, the processed data and the serial number combination corresponding to the image frame are stored in the preprocessing queue. The model inference thread and the inference result post-processing thread both obtain data from the queue in sequence, and store the processed data in the inference queue and the post-processing queue in sequence, so as to ensure that the image data finally obtained is arranged in order.

[0071] The thread adaptive allocation method is specifically as follows: generating random data of the same size as the data captured by the camera as input, such as 1920*1080*3;

[0072] After starting the inference program, you can get the computational time of data preprocessing, model inference, and inference result post-processing, which are t1, t2, and t3 respectively; compare and get the minimum value among t1, t2, and t3, taking t1 as the minimum value as an example, and then calculate the ratio of t2, t3 to t1 and round up. It should be noted that the model inference thread is related to the number of cores of the NPU in RK3588. If it exceeds the number of cores, only the value corresponding to the number of cores is taken. For example, if the data preprocessing takes 6ms, the model inference takes 13ms, and the inference result post-processing takes 8ms, then 1 data preprocessing thread, 3 model inference threads, and 2 inference result post-processing threads should be allocated to achieve acceleration.

[0073] The present invention innovatively proposes a thread adaptation method for multiple AI cores (NPUs), which can automatically allocate the number of threads according to the number of AI cores (NPUs) in the device and the time consumption of each part without manual preset; the timing issues in the multi-core processing process are taken into account, so this method can realize acceleration by utilizing multiple cores when processing video streams.

[0074] S5: Schematic diagram of flame area estimation Figure 4 As shown. According to the mask mask and rectangular annotation box obtained by detection, calculate the ratio between the number of pixels occupied by the fire point target in the image and the number of all pixels in the rectangular annotation box; then, based on the similarity principle, combine the camera's pixel size, focal length, and the distance value obtained by laser ranging to calculate the width and height of the rectangular area in the actual scene corresponding to the detection frame; combine the obtained ratio relationship to calculate the area of ​​the fire point in the actual scene. The detailed steps are as follows:

[0075] a) Use OpenCV's findContours interface to process the mask to obtain the contour information of the fire point target, and use OpenCV's contourArea interface to obtain the total number of pixels occupied by the fire point target. fire ;

[0076] b) Calculate the number of pixels in the rectangular box based on the obtained box box ; Obtain the ratio between the number of pixels occupied by the fire point and the number of all pixels in the detection frame according to the following formula.

[0077]

[0078] c) Based on the distance D obtained by the drone laser ranging module, as well as the camera’s inherent parameters of focal length f and pixel size p pixel , combined with the above-obtained Ratio, calculate the length L and width W of the actual range corresponding to the box with a length of l and a width of w. The formula is as follows:

[0079]

[0080] d) Obtain the actual area S of the fire point area based on the obtained length L, width W and ratio Ratio fire =L*W*Ratio.

[0081] The present invention is applicable to a UAV equipped with a laser distance measuring device, and can directly obtain the fire point area value based on the distance value obtained by the laser distance measuring device by adopting a simple similarity principle, without the need for complex coordinate transformation, with simple processing steps and high efficiency.

[0082] Regarding the contents disclosed in this case, there are a few points that need to be explained:

[0083] (1) The drawings of the embodiments disclosed in this case only involve the structures involved in the embodiments disclosed in this case, and other structures can refer to the general design;

[0084] (2) In the absence of conflict, the embodiments and features of the embodiments disclosed in this case can be combined with each other to obtain new embodiments;

[0085] The above are only specific implementation methods disclosed in this case, but the protection scope of the present disclosure is not limited thereto. The protection scope disclosed in this case should be based on the protection scope of the claims.

Claims

1. A fire point detection and area estimation method for drone inspection, characterized in that: The following steps are involved: S1: Preparation of datasets of fire points, people, and vehicles; S2: Based on the improved lightweight instance segmentation algorithm MobileNetV3-YOLOv8-Seg, the dataset is trained to obtain the target detection model; S3: Convert the object detection model into an object segmentation model; S4: The target segmentation model is called through the inference program to perform target detection on the continuous image frames taken by the drone, and the masks and rectangular annotation boxes of the fire points, people and vehicles are obtained; S5: Calculate the area of ​​the fire point in the actual scene based on the mask and rectangular annotation box obtained by detection.

2. The method for fire point detection and area estimation for drone inspection according to claim 1 is characterized in that: In step S1, various image data with fire points, people, and vehicles taken from the perspective of drones are collected, and the fire points, people, and vehicle areas are labeled using the Labelme image annotation tool. A corresponding json format annotation file is generated for each image. After the annotation is completed, all annotation files are converted into coco format annotation files to form a data set.

3. The method for fire point detection and area estimation for drone inspection according to claim 2 is characterized in that: In step S2, the improved lightweight instance segmentation algorithm MobileNetV3-YOLOv8-Seg includes: The backbone network of the YOLOv8-Seg algorithm uses the lightweight MobileNetV3 network suitable for mobile deployment, and the ultra-lightweight dynamic upsampling operator DySample is used in the neck network; at the same time, the spatial context perception module SCAM is introduced, combining the spatial attention and channel attention mechanisms; After training is completed, the trained model file is converted and exported into the *.onnx intermediate model conversion format.

4. The method for fire point detection and area estimation for drone inspection according to claim 3 is characterized in that: In step S3, the domestic embedded intelligent chip RK3588 is used as the drone image processing module, and the model conversion tool rknn-toolkit2 is used to convert the *.onnx format model into a target segmentation model in the *.rknn format that is suitable for the RK3588 platform.

5. The method for fire point detection and area estimation for drone inspection according to claim 4 is characterized in that: In step S4, The reasoning program is divided into three parts: data preprocessing, model reasoning, and post-processing of reasoning results. in, The data preprocessing part scales, fills, transforms channels, normalizes, and upgrades the acquired image frames to obtain input data of size [1,3,640,640]; The model inference part uses the output of the data preprocessing part as the input of the *.rknn model, and obtains a set of unprocessed basic masks and rectangular boxes after performing inference; The post-processing part includes two steps: first, non-maximum suppression is performed on the rectangular box, and the remaining rectangular box is decoded into a rectangular annotation box corresponding to the original image; then the mask corresponding to the original image is obtained by combining the basic masks.

6. The method for fire point detection and area estimation for drone inspection according to claim 5, characterized in that: The reasoning program uses a thread-adaptive approach, including: Data preprocessing and model inference are cached through thread queues and passed to subsequent processing; multiple model inference threads are used to process queue cache data in parallel; at the same time, after data preprocessing is completed, the processed data and the serial number combination corresponding to the image frame are stored in the preprocessing queue, and the model inference thread and the inference result post-processing thread both obtain data from the queue in sequence, and store the processed data in the inference queue and the post-processing queue in sequence.

7. The method for fire point detection and area estimation for drone inspection according to claim 6, characterized in that: The thread adaptive allocation method includes: generating random data of the same size as the data captured by the camera as input; After starting the inference program, the computational time of data preprocessing, model inference, and inference result post-processing can be obtained, which are t1, t2, and t3 respectively; Compare and obtain the minimum value among t1, t2, and t3, take the minimum value, and then calculate the ratio of the remaining time consumption to the minimum value and round up.

8. The method for fire point detection and area estimation for drone inspection according to claim 1, characterized in that: In step S5, according to the mask mask and the rectangular annotation box obtained by detection, the ratio between the number of pixels occupied by the fire point target in the image and the number of all pixels in the rectangular annotation box is calculated; Then, based on the similarity principle, the width and height of the rectangular area in the actual scene corresponding to the detection frame are calculated in combination with the camera's pixel size, focal length and the distance value obtained by laser ranging; combined with the obtained ratio relationship, the area of ​​the fire point in the actual scene is calculated.

9. The method for fire point detection and area estimation for drone inspection according to claim 8, characterized in that: Step S5 specifically includes: a) Use OpenCV's findContours interface to process the mask to obtain the contour information of the fire point target, and use OpenCV's contourArea interface to obtain the total number of pixels occupied by the fire point target. fire ; b) Calculate the number of pixels in the rectangular box based on the obtained box box ; Get the ratio between the number of pixels occupied by the fire point and the number of all pixels in the detection frame, as follows: c) Based on the distance D obtained by the drone laser ranging module, as well as the camera’s inherent parameters of focal length f and pixel size p pixel , combined with the above-obtained Ratio, calculate the length L and width W of the actual range corresponding to the box with a length of l and a width of w. The formula is as follows: d) Obtain the actual area S of the fire point area based on the obtained length L, width W and ratio Ratio fire =L*W*Ratio.

Images