Target real-time positioning system for resource complementary supply in multi-uav cooperative fire extinguishing

Abstract

The present application belongs to the technical field of target identification and positioning, and relates to a target real-time positioning system for resource complementary supply in multi-unmanned aerial vehicle cooperative fire extinguishing, which is composed of multiple unmanned aerial vehicles, each of which is loaded with a Bluetooth mesh communication module for establishing a communication link between the unmanned aerial vehicle and each of the remaining unmanned aerial vehicles to form a mesh network. Each unmanned aerial vehicle is also loaded with a fire perception module, a data analysis module, a selection trigger module, a supply request module, a data encoding module, a signal transmitting module, a supply response module, a queue generation module, a flight control module, a signal collecting module and a real-time positioning module. Through image processing based on the perspective of multiple spectral images, the accurate positioning of the target unmanned aerial vehicle can be obtained, thereby reducing the difficulty of unmanned aerial vehicle autonomous supply.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of target identification and positioning, and particularly relates to a target real-time positioning system for resource complementary supply in multi-unmanned aerial vehicle cooperative fire extinguishing. BACKGROUND

[0002] In recent years, unmanned aerial vehicles (UAVs) have been increasingly widely applied in forest fire extinguishing due to their advantages such as flexibility, maneuverability and convenient deployment. Multi-UAV cooperative operation can quickly cover a large area of fire area and perform tasks such as fire reconnaissance, fire extinguishing agent spraying and fire monitoring, thereby greatly improving the efficiency and safety of fire extinguishing. However, the limited endurance and insufficient carrying capacity of fire extinguishing agents of UAVs seriously restrict the sustained combat capability of UAVs in forest fire extinguishing tasks. When the power of a UAV is exhausted or the fire extinguishing agent is insufficient, if the UAV cannot be timely supplied, the fire extinguishing task will be interrupted, and the UAV may even crash, resulting in equipment loss.

[0003] In traditional UAV supply technology, manual recovery supply or fixed site supply is mainly adopted. Manual recovery supply requires the UAV to return to a specified location for charging or replenishment of fire extinguishing agents by manual operation, which is low in efficiency and time-consuming, and difficult to meet the urgent needs of forest fire extinguishing tasks. Fixed site supply requires the UAV to fly to a preset supply site, which also has the problems of poor flexibility and insufficient timeliness of supply, and it is difficult to set up a suitable fixed supply site in a complex forest area.

[0004] With the development of wireless charging technology and automatic docking technology, some researches attempt to realize autonomous supply of UAVs. However, the existing technologies have obvious deficiencies in positioning accuracy, tracking stability and docking accuracy. For example, the positioning method based on GPS is easily affected by tree shelter in the forest environment, resulting in large positioning error; the traditional target tracking algorithm is difficult to adapt to the attitude change of the UAV under complex weather conditions and dynamic flight scenarios, and tracking loss is likely to occur; in the supply docking link, due to the lack of effective visual feedback and precise control means, reliable docking of the UAV in flight state cannot be realized, and the success rate of supply is low. In addition, the forest fire scene has the characteristics of complex environment and many interference factors such as smoke, light change and strong airflow, which further increases the difficulty of precise supply of the UAV. SUMMARY

[0005] The technical problem to be solved by the present application is that the existing autonomous supply technology of UAVs cannot obtain accurate positioning of the target UAV.

[0006] To solve the above technical problems, the present application realizes the following technical scheme:

[0007] A real-time target positioning system is provided for resource complementarity in multi-UAV collaborative firefighting, characterized by comprising: a mesh network composed of multiple UAVs; each UAV includes:

[0008] The fire perception module is used to sense the real-time fire situation within the mission area of ​​this UAV;

[0009] The data analysis module is used to input the real-time fire situation and the real-time status of the UAV into the trained random forest model and output the classification results. The classification results include: unable to complete the fire extinguishing task and able to complete the fire extinguishing task.

[0010] The select trigger module is used to trigger the supply request module, data encoding module, and signal transmission module when the UAV cannot complete the firefighting mission, and to trigger the supply response module, queue generation module, flight control module, signal acquisition module, real-time positioning module, and target tracking module when the UAV can complete the firefighting mission.

[0011] The supply request module is used to send supply request signals to the other drones via the mesh network;

[0012] The data encoding module is used to encode the identity data and current attitude data of this UAV in real time;

[0013] The signal transmission module is used to transmit multispectral signals to the outside world based on real-time encoded data;

[0014] The supply response module is used to send a supply response signal to the other drones via the mesh network when a supply request signal is received.

[0015] The queue generation module is used to generate response priority queues based on each supply response signal;

[0016] The flight control module is used to control the UAV to fly to the signal acquisition area close to the current position in the resupply request signal when the UAV is at the top of the response priority queue;

[0017] The signal acquisition module is used to acquire multispectral signals within the signal acquisition area and generate multispectral images;

[0018] The real-time positioning module is used to obtain the real-time positioning of the target UAV based on multispectral images.

[0019] Compared with the prior art, the present application has the following advantages and beneficial effects: on the one hand, by virtue of the advantages of decentralization, multi-organization, self-healing and multi-hop transmission of the mesh self-organizing network technology, a mesh network with multiple unmanned aerial vehicles as nodes is established, good communication between multiple unmanned aerial vehicles in a complex environment is ensured, and the problem of positioning deviation caused by the shielding of GPS signals is solved, thereby providing a good network environment foundation for the collaborative resource complementation of multiple unmanned aerial vehicles; on the other hand, by sending a supply request, a supply response, and each unmanned aerial vehicle analyzing the task completion of itself, resource collaborative scheduling between multiple unmanned aerial vehicles is realized. In addition, the target unmanned aerial vehicle transmits a multi-spectrum signal carrying identity information and attitude information, which can avoid the influence of smoke and light changes in the forest fire environment on target positioning accuracy, and the supply unmanned aerial vehicle can obtain accurate positioning of the target unmanned aerial vehicle through perspective-based image processing of the multi-spectrum image, thereby reducing the difficulty of autonomous supply of the unmanned aerial vehicle. BRIEF DESCRIPTION OF DRAWINGS

[0020] The drawings described herein are used to provide further understanding of the embodiments of the present application, form a part of the present application, and do not constitute a limitation on the embodiments of the present application. In the drawings:

[0021] Figure 1 A mesh network structure composed of multiple unmanned aerial vehicles provided by the embodiments of the present application is shown in the figure;

[0022] Figure 2 A concentric double-ring structure provided by the embodiments of the present application is shown in the figure. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical scheme and advantages of the present application more clear and explicit, the present application is further described in detail below in combination with embodiments, the illustrative embodiments of the present application and the description thereof are used to explain the present application, and do not constitute a limitation on the present application, the embodiments described below are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor belong to the scope of protection of the present application.

[0024] In the following description, a large number of specific details are set forth in order to provide a thorough understanding of the present application. However, it is apparent to those skilled in the art that the present application can be implemented without necessarily adopting these specific details. In other embodiments, in order to avoid obscuring the present application, well-known structures, materials or methods are not specifically described. The materials, instruments and reagents used in the following embodiments, etc., can be obtained from commercial channels if not specifically stated. The technical means used in the embodiments, if not specifically stated, are conventional means known to those skilled in the art.

[0025] In addition, the terms "first", "second", etc. are used only for descriptive purposes and should not be construed as implying or suggesting relative importance or an indicated number of technical features. Thus, features defined with "first", "second", etc. can explicitly or implicitly include one or more of the features. In the description of the present application, the meaning of "multiple" is two or more, unless otherwise explicitly and specifically limited.

[0026] Embodiment: A target real-time positioning system for resource complementary supply in multi-UAV cooperative fire extinguishing is provided. The system is composed of multiple UAVs, each of which is equipped with a Bluetooth mesh communication module for establishing a communication link between the UAV and each of the remaining UAVs to form a mesh network as shown. Figure 1 Each UAV is also equipped with the following functional modules:

[0027] 1. Status monitoring module, network topology module, task allocation module, fire perception module and data analysis module

[0028] The purpose of the present application is to realize resource complementary supply between multiple UAVs in a cooperative fire extinguishing scenario. The prerequisite for resource complementary supply is that the UAV can complete its own fire extinguishing task and has resource surplus. Therefore, the UAV needs to autonomously determine whether it can complete the fire extinguishing task. The present embodiment uses the fire perception module and the data analysis module to achieve this purpose.

[0029] The fire perception module is used to perceive the real-time fire situation in the task execution area of the UAV, providing data support for the data analysis module. From the functional description of the fire perception module, it can be seen that the real-time fire situation is limited to the task execution area of the UAV. Therefore, the task execution area of the UAV needs to be clearly defined before the real-time fire situation. The present embodiment uses the network topology of multiple UAVs to filter out the backbone layer nodes in the network topology, and uses the backbone layer nodes to allocate fire extinguishing tasks to the remaining UAVs in the network topology, so that each UAV knows its task execution area. This is achieved through the following functional modules:

[0030] (1) Status monitoring module

[0031] The status monitoring module is used to monitor the real-time status of the UAV. The real-time status includes the current remaining power, the current amount of fire extinguishing agent, the communication radius, the current received signal strength indicator and the current computing power of the UAV. The status monitoring module is connected to the mesh communication module, and the real-time status of the UAV is sent to the remaining UAVs in the cellular subnetwork through the mesh communication module, so that each UAV in the mesh network can know the real-time status of each of the remaining UAVs.

[0032] In the status monitoring module: 1) A Battery Management System (BMS) can be used to monitor the drone's remaining battery power in real time. The BMS is integrated inside the drone's battery, collecting battery voltage, current, and temperature data in real time, and calculating the remaining battery power and health status. 2) A weight sensor can be used to monitor the drone's remaining fire extinguishing agent level in real time. The weight sensor directly weighs the container holding the fire extinguishing agent, and the remaining amount is obtained after deducting the weight of the empty container. 3) The NRF24L01 wireless module can be used to monitor the communication radius and received signal strength in real time. 4) A performance counter (such as an ARM Cortex PMU unit) can be used to monitor the drone's computing power in real time. The performance counter is built into the drone's control chip and monitors node computing power consumption by calling the top command or nodelet component in real time.

[0033] (2) Network topology module

[0034] Based on the status monitoring module, the network topology module establishes a hierarchical network topology containing multiple drones according to the real-time status of this drone and the received real-time status of other drones in this cellular subnet. The hierarchical network topology consists of backbone layer nodes and multiple access layer nodes.

[0035] This embodiment builds a hierarchical and distributed collaborative architecture through the network partitioning module, and further establishes a two-layer network topology of "backbone layer nodes + access layer nodes" for each cellular subnet through the network topology module.

[0036] Furthermore, the backbone layer nodes and access layer nodes are selected through a weight calculation unit, a first filtering unit, and a second filtering unit:

[0037] 1) Weight Calculation Unit

[0038] This is used to obtain the overall weight of this drone in the network topology based on the real-time status, and then send the overall weight to the other drones.

[0039] Based on the explanation of the status monitoring module, the real-time status of the UAV includes: current remaining battery power, current remaining fire extinguishing agent, communication radius, current received signal strength, and current computing power. Therefore, the comprehensive weight of the UAV in the network topology can be obtained through a weight calculation model. The expression for the weight calculation model is: Comprehensive Weight = a 1× Remaining battery power + a 2×Extinguishing Agent Residual + a 3×communication radius+ a 4× Received signal strength index+ a 5× computing power. Among them, a 1 represents the weighting factor for the remaining battery power. a 2 is the weighting coefficient for the residual extinguishing agent. a3 is a weight coefficient of communication radius, a 4 is a weight coefficient of received signal strength indicator, a 5 is a weight coefficient of computing power.

[0040] It should be noted that: <1> before calculating the comprehensive weight, the remaining, the extinguishing agent remaining amount, the communication radius, the received signal strength indicator and the computing power need to be normalized. The purpose is to convert the indicators of different dimensions and value ranges into unified dimensionless values, which are usually mapped to the interval [0, 1] or [-1, 1]. <2> Each weight coefficient is set according to the importance of the real-time state of each type. If the real-time state of each type of task is in the same important position, each weight coefficient can be set to be the same, for example a 1 to a 5 are all 0.2.

[0041] 2) the first screening unit.

[0042] For screening out the unmanned aerial vehicle corresponding to the comprehensive weight greater than the threshold.

[0043] 3) the second screening unit.

[0044] For screening out the unmanned aerial vehicle closest to the geometric center or geometric center of the network topology as the backbone layer unmanned aerial vehicle according to the real-time position of the screened out multiple unmanned aerial vehicles, and taking the remaining unmanned aerial vehicles as access layer nodes.

[0045] It should be noted that: in combination with the role of the backbone layer node, the unmanned aerial vehicle closest to the geometric center or geometric center of the network topology is taken as the backbone layer unmanned aerial vehicle, which is based on the requirements of the core targets of communication efficiency optimization, coverage balance, network stability and resource utilization maximization of the backbone layer node.

[0046] After the network topology module is used to screen out the backbone layer node, the backbone layer node can be used to distribute the corresponding fire extinguishing task for each unmanned aerial vehicle in the network topology. Therefore, each unmanned aerial vehicle also needs to carry a task distribution module. The premise of the work of the task distribution module is that the unmanned aerial vehicle is selected as the backbone layer node, so the selection trigger module is also used to trigger the task distribution module to work when the unmanned aerial vehicle is the backbone layer node.

[0047] (3) task distribution module

[0048] For generating the fire extinguishing task of each unmanned aerial vehicle according to each real-time fire and each real-time state, and sending each fire extinguishing task to each unmanned aerial vehicle.

[0049] Under the network topology architecture, after the backbone layer nodes receive the real-time fire information and real-time state information sent by each unmanned aerial vehicle, through multi-dimensional data processing and intelligent algorithm, the dynamic and accurate allocation of the fire extinguishing task of each unmanned aerial vehicle is realized, so as to improve the overall fire extinguishing efficiency and resource utilization. Specifically, the task allocation module includes:

[0050] 1) fire analysis unit

[0051] For dividing the management unit into multiple task execution areas according to the real-time fire situation. The task execution area includes: flame front area, flame spread area and cooling area. Among them, the flame front area is the area where the fire is currently most intense, the flame directly burns and is in a state of advancement; the flame spread area is located behind the flame front area, which is the area that the flame may spread to; the cooling area is an area where the fire has been extinguished or the fire has been effectively controlled, and the temperature is relatively low. For the flame front area: the fused image can be edge detected (such as Canny operator), and the flame edge contour can be extracted to determine the boundary area of the front end of the flame as the flame front area. For the flame spread area: machine learning algorithms (such as LSTM, GRU) can be used to learn the historical movement data of the flame front area to predict the future spread direction and range of the flame. For the cooling area: in the temperature distribution map, the area with temperature lower than the set threshold and no obvious flame is marked as the cooling area.

[0052] 2) priority quantization unit

[0053] For quantifying the priority of each flame front area, the priority of each flame spread area and the priority of each cooling area by a fire priority quantization model. The purpose of priority quantization is:

[0054] <1> Realize the reasonable allocation of resources: in the fire extinguishing task, the number of unmanned aerial vehicles, the carrying capacity of fire extinguishing agent, the flight time and other resources are limited. By quantifying the priority of different areas, the emergency degree and resource demand degree of each area can be clearly judged, so as to put the limited resources into the area with high priority first.

[0055] <2> Promote multi-machine collaborative operation: after the priority of different areas is quantified, each unmanned aerial vehicle clearly knows the importance of its own task and target, which helps to achieve better cooperation.

[0056] The expression of the fire priority quantization model is: (1); in formula (1), P is the fire priority quantization value, w 1 represents the weight of the flame spread speed, w 2 represents the weight of the fire influence area, V represents the flame spread speed, V max represents the maximum flame spread speed,S Indicates the area affected by the fire. S max Indicates the maximum fire area; w 1 and w 2. Calculated using the Analytic Hierarchy Process (AHP). For example, using a fire priority quantification model, fire priorities are divided into three levels: First Priority (… P ≥0.7), second priority (0.3 < P <0.7), third priority ( P (≤0.3). It should be noted that before calculating the overall weight, the flame spread rate, the area affected by the fire, and the maximum flame spread rate need to be normalized.

[0057] In addition, the quantification results are sent to the remaining backbone nodes through the mesh communication module to provide data support for subsequent priority matching.

[0058] 3) Priority matching unit

[0059] This is used to match the combined weights from each weight calculation unit with the priority levels from each priority quantization unit using a support vector machine, and to establish a priority matching table.

[0060] 4) Task allocation unit

[0061] This is used to assign firefighting tasks to each node in the current cellular subnet based on a priority matching table and a multi-objective optimization algorithm. Specifically:

[0062] First, define the core objective function for multi-objective optimization, including:

[0063] Fire extinguishing efficiency function: ,in, F eff This measures the area of ​​fire extinguished per unit of time. j This refers to the node numbering within the cellular subnet. k This indicates the number of nodes in the subnet. S shed,j Indicates the first j The area of ​​the fire extinguished by each node t total This indicates the total firefighting time.

[0064] Resource utilization function: ,in, F res Indicates the overall resource utilization rate. m use,j Indicates the first j The amount of extinguishing agent already used at each node, m total,j Indicates the firstj The total amount of extinguishing agent at each node, E left,j Indicates the first j The remaining power of each node E total,j Indicates the first j Total power of each node.

[0065] Task completion time function: F time =max{ t finish,1 , t finish,2 ,…, t finish,k},in, F time This represents the maximum value of all tasks completed. t finish,j Indicates the first j Task completion time for each node. j=1 , 2 , … , k .

[0066] The multi-objective optimization algorithm described in this embodiment is the NSGA-II algorithm. Based on the NSGA-II algorithm, fire suppression tasks are assigned to each node in this cellular subnet. Specific operations include: population initialization, fitness calculation, non-dominated sorting, crowding calculation, genetic operations, population update iteration, and scheme selection.

[0067] Initialize the population – randomly generate a certain number (e.g., 100) of task allocation schemes as the initial population, ensuring that each task is covered by at least one node (avoiding invalid solutions) and satisfying node energy and communication constraints.

[0068] Fitness calculation – Based on the three objective functions mentioned above, calculate the fitness value for each individual.

[0069] Non-dominated ranking—This involves stratifying individuals in a population according to their non-dominant relationships. Individuals within the same stratum are mutually non-dominant, with lower stratum numbers indicating better individuals. For example, for any two individuals A and B in the population, if A is not inferior to B in all objectives and is superior to B in at least one objective, then A is said to dominate B. By ranking the population hierarchically, different non-dominant levels are formed (level 1 is optimal).

[0070] Crowding density calculation reflects the distribution density of individuals in the solution space and is used to maintain population diversity. Crowding density calculation involves calculating the sum of distances between adjacent individuals in each target dimension for individuals within the same non-dominated layer. A higher crowding density indicates a sparser distribution of individuals around that individual, and thus better diversity.

[0071] Genetic operations include: selection operation: adopt tournament selection method, randomly select k individuals from the population, select individuals with high non-dominated level and large crowding degree into the mating pool; crossover operation: adopt two-point crossover or uniform crossover, randomly exchange task allocation fragments of selected chromosomes, while ensuring the legality of the solution after crossover (such as task coverage constraint); mutation operation: flip the gene in the chromosome with low probability (0→1 or 1→0), and repair the solution that violates the constraint (such as reassigning nodes to uncoated tasks).

[0072] Population update iteration: merge the parent and child populations, retain the top 100 individuals (based on non-dominated sorting and crowding degree), form a new generation of population; repeat the selection, crossover and mutation operations until the maximum number of iterations is reached or the Pareto frontier converges.

[0073] Scheme selection: select the most suitable task allocation scheme for the current fire situation and node state from the Pareto frontier solution set. Specifically, the most suitable task allocation scheme for the current fire situation and node state is selected from the Pareto frontier solution set, which needs to combine dynamic evaluation index, real-time data feedback and decision strategy to convert multi-objective optimization results into executable specific scheme:

[0074] <1> Dynamic evaluation index

[0075] Including: fire emergency level (flame spread speed, threat area population / asset value and burning material type), task timeliness (remaining time from the best fire extinguishing window period), spatial distribution (fire source location, spread direction), unmanned aerial vehicle resources (remaining power, range, fire extinguishing agent capacity, current location and distance from the fire source), historical task load (number of executed tasks, cumulative flight time), etc.

[0076] <2> Real-time data feedback

[0077] According to the real-time data, the weight of each target is allocated, and the multi-objective problem is converted into a single-objective optimization problem. For example, according to the preset expert field knowledge, the weight of each target is adjusted in real time. Among them, the expert field knowledge is: when the fire level is ≥3, the "fire extinguishing speed" weight is >0.6, and the "unmanned aerial vehicle safety" weight is >0.3, when there are people trapped in a certain area, the "priority to cover the area" weight is increased to 0.5. Real-time adjustment of target allocation weight can dynamically correct the weight through sensor data (such as thermal imaging, wind speed), for example, when the wind speed increases, the weight of "quickly blocking the direction of fire spread" increases.

[0078] <3> Decision strategy

[0079] A fuzzy inference system is constructed, with input variables being fire emergency degree (low / medium / high) and node health degree (excellent / good / poor), and output being weight coefficients of each target. For example, when "fire emergency degree = high" and "node health degree = excellent", the "task timeliness" weight = 0.7, and the "energy consumption" weight = 0.3; when "fire emergency degree = medium" and "node health degree = poor", the "balanced load" weight = 0.6, and the "completion rate" weight = 0.4.

[0080] <4> Scheme screening

[0081] Each scheme in the Pareto solution set is quantitatively scored by using the evaluation index, and the scheme most suitable for the current state is screened out. The approximation ideal solution ranking method can be used for screening. The method steps are as follows: first, the index data of each scheme is standardized (such as normalized power, distance, etc.); then, the "positive ideal solution" (the optimal value of each index) and the "negative ideal solution" (the worst value of each index) are determined; finally, the Euclidean distance of each scheme from the positive / negative ideal solution is calculated to obtain the comprehensive score: C i = d i - / ( d i - + d i + ), wherein, C i represents the comprehensive score, C i the closer to 1 the better the scheme, d i - is the negative ideal solution, d i + is the positive ideal solution.

[0082] According to the above explanation and description of the state monitoring module, the network topology module and the task allocation module, it can be known that the fire extinguishing task described in the embodiment is essentially the task execution area planned by the backbone layer node for each unmanned aerial vehicle. After each unmanned aerial vehicle is allocated a corresponding fire extinguishing task (task execution area), the unmanned aerial vehicle can use the fire condition perception module to collect real-time fire conditions in the task execution area.

[0083] (4) Fire condition perception module

[0084] The fire condition perception module is composed of the following functional units:

[0085] 1) Visible light camera

[0086] The visible light camera is used to capture images of fire in the area where the UAV is performing its mission in real time. The visible light camera can identify regular flames and smoke, and combined with a convolutional neural network (such as YOLOv5), it can quickly detect fire targets.

[0087] The YOLOv5 model is a real-time target detection model based on a convolutional neural network, with the characteristics of fast speed and high accuracy, suitable for rapid detection of fire targets. Before using YOLOv5 to detect fire targets, the YOLOv5 model needs to be trained and tested. The training and testing method of the YOLOv5 model is as follows:

[0088] Collect image data of flames and smoke from various channels, including but not limited to pictures taken at actual fire scenes, publicly available data sets on the Internet, and images taken by simulating fire scenes, etc., to ensure the diversity and richness of the data, covering different lighting conditions, background environments, burning substances, etc.

[0089] Preprocess the collected fire image data, including rotation, sharpening, cropping, and adjusting the image size to the input size required by the YOLOv5 model, such as 416x416.

[0090] Label the preprocessed fire image data, including the location of the flame and the location of the smoke. Divide the labeled fire image set into a training set and a test set.

[0091] Use the training set to train the YOLOv5 model. During the training process, data augmentation techniques such as random cropping and color jittering can be used to further improve the robustness of the model. At the same time, use the complete intersection over union (CIoU) loss function to speed up the convergence of the model and improve the robustness of the model. Use the test set to verify the accuracy, recall rate, mAP, etc. of the YOLOv5 model.

[0092] Input the fire image captured by the visible light camera in real time into the trained YOLOv5 model to detect fire targets, and output the detection results, including the location of the flame and the location of the smoke.

[0093] 2) Thermal imaging infrared sensor

[0094] The thermal imaging infrared sensor can select a high-precision platinum resistance temperature sensor (such as Pt100). The platinum resistance temperature sensor can capture hot spots, fire points, and smoke areas in real time, and can discover fire sources even at night or under smoke cover. The platinum resistance temperature sensor generates thermal images in real time by detecting infrared radiation emitted by objects. These thermal images can clearly show the temperature distribution in the monitoring area, including hot spots, fire points, and smoke areas. In the platinum resistance temperature sensor, the position of the fire source and the temperature change trend are extracted from the thermal image according to the temperature distribution, and the area with temperature anomaly is identified.

[0095] 3) Temperature sensor

[0096] The temperature distribution graph of the management unit is collected in real time. In an indoor environment, a high-precision digital temperature sensor such as DS18B20 can be selected. The digital temperature sensor can monitor the local environmental temperature rise, and through image data fusion, it can assist in judging the severity of the fire. Specifically, the image data of the temperature distribution graph is fused with the image data of the thermal image and the image data of the visible light image by using weighted average method, Kalman filtering method, etc. For example, first, the data of the thermal image and the visible light image are normalized to make their numerical ranges consistent. For example, the data of the thermal image and the visible light image can be normalized to the range of [0, 1]. Then, the image data of the temperature distribution graph is used to assign weights to the image data of the thermal image and the image data of the visible light image. Finally, the weighted average value of the image data of the thermal image and the image data of the visible light image is calculated as a comprehensive evaluation index of the severity of the fire. wherein, R is the weighted average value, N is the total number of pixel points in the image, w i is the weight of the i th pixel point, I 1 is the normalized intensity value of the i th pixel point of the thermal image, I 2 is the normalized intensity value of the i th pixel point of the visible light image.

[0097] 4) Feature extraction unit

[0098] The visible light image, thermal image and temperature distribution image are fused by using an image fusion algorithm, and a convolutional neural network is used to extract features from the fused image to obtain feature information. Specifically, before image fusion, the visible light image, thermal image and temperature distribution image are preprocessed to unify the image size, eliminate noise and correct color and gray difference. The SIFT algorithm or SURF algorithm is used to extract feature points in the visible light image, thermal image and temperature distribution image, respectively, and feature matching is performed. The matched features are fused, and the fused image is reconstructed by interpolation, transformation and other methods according to the fused feature information.

[0099] Specifically, the following steps are included:

[0100] 1. Extract feature points in the visible light image, thermal image and temperature distribution image.

[0101] (1) Select the SIFT (Scale-Invariant Feature Transform) algorithm or the SURF (Speeded-Up Robust Features) algorithm to extract feature points. The SIFT algorithm has good robustness to scale, rotation and illumination changes, while the SURF algorithm is faster in extraction speed and suitable for real-time applications.

[0102] (2) Extraction process - input the visible light image into the SIFT or SURF algorithm, use the algorithm to automatically detect feature points in the image, and calculate the descriptor of each key point; input the thermal image into the SIFT or SURF algorithm, use the algorithm to automatically detect feature points in the image; for the temperature distribution image, first convert it to image format, then use the SIFT or SURF algorithm to extract feature points.

[0103] 2. Feature matching for the visible light image, thermal image and temperature distribution image.

[0104] The matched feature points are fused to form a comprehensive feature set. The fusion method can be a simple weighted average. The fusion process is as follows: the descriptors of the corresponding feature points in the visible light image, thermal image and temperature distribution image are weighted and averaged to obtain the fused feature information.

[0105] 3. Image reconstruction according to the fused feature set.

[0106] (1) According to the fused feature point information, a grid is constructed to divide the image into multiple small regions.

[0107] (2) In each small region, the values of other points in the image are calculated using interpolation methods. For example, in bilinear interpolation, the value of the target point is calculated according to the values of the four nearest feature points.

[0108] (3) The calculated values of each point are spliced together to form a complete fused image.

[0109] Based on the fused feature point information, interpolation methods are used to reconstruct the fused image. Common interpolation methods include bilinear interpolation and bicubic interpolation. These methods can calculate the values ​​of other points in the image based on known feature point values, thereby generating a complete image.

[0110] 5) Information processing unit

[0111] This is used to analyze feature information using deep learning models (such as the YOLOv5 model) to obtain real-time fire information. Real-time fire information includes: the area affected by the fire and the rate of flame spread.

[0112] It should be noted that for calculating the area affected by the fire: the YOLOv5 model calculates the bounding box area based on the bounding box coordinates (top-left corner coordinates, top-right corner coordinates), optimizes the region within the bounding box through morphological operations (such as erosion and dilation), and then converts the pixel area into the actual physical area by combining the image resolution. Specifically, first, the YOLOv5 model performs inference on the input image and outputs the bounding box coordinates of the flame target, in the form of top-left corner coordinates (top-left corner coordinates, top-right ... x 1, y 1) and the coordinates of the lower right corner ( x 2, y 2) The minimum bounding area of ​​the corresponding rectangle. The pixel area for a single flame target. S px The calculation formula is: S px =( x 2- x 1)×( y 2- y 1) If multiple flame targets exist, the area of ​​each target needs to be calculated separately and then summed to obtain the total pixel area. Then, morphological operations are used to remove noise from the detection results (such as falsely detected small areas) and fill holes within the flame area to smooth the boundaries and improve the accuracy of area calculation. This includes: erosion operation—using a structuring element of a specified size (such as a 3×3 or 5×5 rectangular or elliptical kernel) to traverse the binarized flame mask within the bounding box, deleting edge pixels and eliminating small particle noise; dilation operation—the shape and size of the structuring element need to be adjusted according to the resolution and noise characteristics of the flame image, usually determined experimentally (e.g., kernel size of 5×5, iteration count of 1-2). Finally, the pixel area is converted to physical area. The mapping relationship between image resolution and actual physical size needs to be pre-calibrated, i.e., the actual area corresponding to a unit pixel (e.g., meters). 2 (per pixel). There are two cases: if the camera position is fixed, the pixel density can be calculated using a reference object of known size (such as a ground marker). , ,in,d 1 represents the actual physical width (in meters); d 2 represents the image width (in pixels), h 1 represents the actual physical height (in meters), h 2 represents the image height (in pixels). If it involves perspective transformation (such as oblique shooting), the image needs to be corrected through the homography matrix, and the physical area is calculated after converting to the bird's eye view. Finally, the total physical area S 总 = S px × .

[0113] For flame propagation speed calculation: after the YOLOv5 model obtains continuous multiple frames of images, flame detection is performed on each frame of image respectively, and the bounding box coordinates of multiple flame targets are obtained, the corresponding relationship of flame targets between different frames is determined, and the displacement distance of flame centroid between different frames is calculated to indirectly calculate the flame propagation speed. Specifically, first, continuous frame flame detection and target tracking are performed - YOLOv5 detection is performed on continuous N frames of images (such as 30 frames per second, taking the last 5 frames) in turn to obtain the bounding box coordinates of flame targets in each frame; a target tracking algorithm (such as IOU-based greedy tracking or DeepSORT) is used to associate the same flame target in different frames to ensure the accuracy of the cross-frame correspondence; the Hungarian Algorithm is used to match the intersection over union (IOU) of the current frame detection box and the previous frame tracking box to avoid target ID jumping. Then, the centroid is calculated - for the bounding box of a single flame target, the centroid ( c x , c y ) takes the center of the bounding box: c x =( x 1+ x 2) / 2, c y =( y 1+ y 2) / 2. Next, calculate the frame displacement - let the centroid of the target in the t th frame be ( c x t , c y t ), and the centroid of the t +1th frame be ( c x t+1 , c y t+1), then the pixel displacement distance is d px , Convert the pixel displacement distance to the actual physical displacement d phys , d phys =d px × , is the scale factor between the pixel displacement distance and the actual physical displacement, which can be obtained through experiments or calibration process. Finally, the flame propagation speed is calculated - assuming the video frame rate is f (such as 30 FPS), then the time interval△ t =1 / f second between two adjacent frames, then the average speed of the flame in the inside v is v = d phys / △ t (meters / second).

[0114] (5) Data analysis module

[0115] for inputting real-time fire and real-time state of the unmanned aerial vehicle into the trained random forest model, and outputting classification results. The classification results include: unable to complete the fire extinguishing task and able to complete the fire extinguishing task.

[0116] 2, Select trigger module

[0117] for triggering the supply request module, data encoding module and signal transmitting module to work when the unmanned aerial vehicle cannot complete the fire extinguishing task, and triggering the supply response module, queue generation module, flight control module, signal collection module, real-time positioning module and target tracking module to work when the unmanned aerial vehicle can complete the fire extinguishing task. The above-mentioned each functional module will be explained and described respectively as follows.

[0118] (1) Supply request module

[0119] for sending the supply request signal to the remaining unmanned aerial vehicles through the mesh network. The triggering premise of the supply request module is that the unmanned aerial vehicle judges that it cannot complete the fire extinguishing task by using the current resource amount through analysis, and then sends the supply request signal to the remaining unmanned aerial vehicles through the mesh network.

[0120] (2) Data encoding module and signal transmitting module

[0121] The resource-deficient unmanned aerial vehicle sends a supply request signal to the other unmanned aerial vehicles, and waits for the other unmanned aerial vehicles to supply resources while performing the remaining fire extinguishing task. In order to enable the unmanned aerial vehicle providing resource supply to accurately find and obtain the accurate position of the unmanned aerial vehicle waiting for resource supply, the unmanned aerial vehicle waiting for resource supply provides its identifiable information to the outside world by using a data encoding module and a signal sending module.

[0122] 1) Signal encoding module

[0123] The signal encoding module is used for real-time encoding of identity data and current attitude data of the unmanned aerial vehicle. The signal encoding module comprises:

[0124] <1> A first encoding unit.

[0125] The first encoding unit is used for Manchester encoding of the identity data to obtain a first encoding result.

[0126] It should be noted that in the embodiment, the identity data of the unmanned aerial vehicle is a MAC address hash value. Manchester encoding is a kind of encoding method for encoding digital data into analog signals. The time interval of each bit is divided into two equal parts. At the middle position of each bit, the signal will jump once, thereby representing the logic state of the data. Specifically, logic 0 represents that the signal jumps from a high level to a low level; and logic 1 represents that the signal jumps from a low level to a high level. Further, the identity data of the unmanned aerial vehicle is a unique identifier of the unmanned aerial vehicle, which needs to be absolutely avoided to be decoded incorrectly. The jump feature of Manchester encoding can eliminate the direct current component, and adapt to the signal baseline fluctuation caused by the change of light in the forest environment. Therefore, the identity data of the unmanned aerial vehicle is Manchester encoded in the embodiment.

[0127] <2> A second encoding unit

[0128] The second encoding unit is used for Gray encoding of the attitude data to obtain a second encoding result.

[0129] In the embodiment, the pitch angle, the azimuth angle and the roll angle of the unmanned aerial vehicle are used to represent the real-time flight attitude of the unmanned aerial vehicle, and the attitude data is Gray encoded. The Gray encoding has the feature that only one bit is different between any two adjacent values. In the process of performing the task, the flight attitude of the unmanned aerial vehicle continuously changes, and the angle change between adjacent frames is usually less than 10°. The feature that only one bit is different between adjacent values of the Gray encoding can minimize the error code caused by rotation. Therefore, the attitude data of the unmanned aerial vehicle is Gray encoded in the embodiment.

[0130] <3> PPS signal modulation unit

[0131] The PPS signal modulation unit is used for modulating the first encoding result and the second encoding result into a PPS signal, and sending the PPS signal to the other unmanned aerial vehicles through a mesh network, so as to realize GPS clock alignment.

[0132] 2) signal transmitting module

[0133] The signal transmitting module is used for transmitting multispectral signals to the outside world according to the real-time coded data. The multispectral signals include visible light signals, infrared light signals and ultraviolet light signals. The light signals can be used to generate images with rich details and meet the signal collection under normal lighting conditions. The infrared light signals have good penetration ability and can meet the target detection under night or low-visibility conditions, such as smoke environment in fire scenes and long-distance tracking under night lightless conditions. The ultraviolet light can suppress sunlight images and meet the signal collection under strong sunlight interference environment.

[0134] As can be known from the above explanation and description of the data coding module, the identity data and flight attitude data of the unmanned aerial vehicle are coded in different ways. The identity data is used to identify the unmanned aerial vehicle, and the flight attitude data is used to provide the positioning of the unmanned aerial vehicle. Therefore, the signal transmitting module also adopts two data transmitting modes.

[0135] Specifically, the signal transmitting module includes: Figure 2The first LED ring array and the second LED ring array are shown, and the first LED ring array and the second LED ring array constitute a concentric double-ring structure (the positions of the first LED ring array and the second LED ring array can be interchanged). In each of the first LED ring array and the second LED ring array, a plurality of LED units are provided, and each LED unit emits a multi-spectrum signal to the outside world in a flashing manner (a flashing frequency feature of Manchester coding) according to a first coding result. Further, the second LED ring array also has a plurality of LED units, which are used to transmit the pitch angle data, the azimuth angle data and the roll angle data to the outside environment through the multi-spectrum signal. Therefore, the second LED ring array provided by the embodiment includes three sub-zones corresponding to the pitch angle, the azimuth angle and the roll angle, and one sub-zone is responsible for reflecting the change mode of one angle data. In addition, each sub-zone includes four LED units arranged at equal intervals in a clockwise or counterclockwise direction; each LED unit is used to emit a visible light signal, an infrared light signal and an ultraviolet light signal to the outside world; along the arrangement direction, the first LED unit corresponds to the binary code 00, the second LED unit corresponds to the binary code 01, the third LED unit corresponds to the binary code 10, and the fourth LED unit corresponds to the binary code 11. Specifically, since the adjacent values in the Gray coding only change by 1 bit, that is, there are only four combination modes of 00, 01, 10 and 11 between the adjacent values in the Gray coding, therefore, the code sequence of the flight attitude data after the Gray coding can also be represented by the above four modes. For example, the code sequence of the pitch angle 5° after the Gray coding is 001110, and three combinations of 00, 11 and 10 are obtained by splitting the code sequence two by two, then in the sub-zone corresponding to the pitch angle (clockwise or counterclockwise), 00 is displayed by the first LED unit, 11 is displayed by the fourth LED unit, and 10 is displayed by the third LED unit. According to the order of 00, 11 and 10 in the code sequence of the pitch angle, the lighting order of the LED units in the sub-zone corresponding to the pitch angle is first LED unit, fourth LED unit, third LED unit in turn. The lighting rules of the LED units in the sub-zones corresponding to the azimuth angle and the roll angle are the same. Therefore, the second LED ring array controls the LED units to emit the multi-spectrum signal to the outside world in a manner of changing the lighting order according to the second coding result.

[0136] It should be noted that the ring array is adopted in the embodiment, so that the plurality of LED units can be observed from any angle.

[0137] (3) Supply response module and queue generation module

[0138] In order to realize efficient cooperation of multiple unmanned aerial vehicles and avoid supply conflicts between multiple unmanned aerial vehicles, the embodiment adopts a response priority sorting manner to preferentially select an unmanned aerial vehicle with the highest response priority for resource supply. The following functional modules are used to realize the above.

[0139] 1) replenishment response module

[0140] for sending replenishment response signals to the rest of the unmanned aerial vehicles when receiving replenishment request signals.

[0141] In order to avoid replenishment errors and replenishment conflicts, and to achieve efficient cooperation of multiple unmanned aerial vehicles, each replenishment unmanned aerial vehicle needs to know the content of the replenishment request signal and the content of the replenishment response signal sent by the rest of the replenishment unmanned aerial vehicles. Therefore, the replenishment request signal of the embodiment carries the replenishment request type (including: power replenishment request and extinguishing agent replenishment request), and the replenishment response signal carries the surplus power and the surplus amount of extinguishing agent, so that the replenishment unmanned aerial vehicle can judge whether the resource surplus of each unmanned aerial vehicle meets the resource demand. For example, if the replenishment request type in the replenishment request signal sent by a certain unmanned aerial vehicle is power replenishment request, and the remaining resource of a certain replenishment unmanned aerial vehicle is only extinguishing agent, then the replenishment unmanned aerial vehicle cannot meet the replenishment demand. In addition, the replenishment request signal also carries the identity data of the unmanned aerial vehicle (for identity recognition of the replenishment unmanned aerial vehicle) and the current position (for preliminary positioning of the replenishment unmanned aerial vehicle), and the replenishment response signal also carries the identity data, the current position and the response time of the unmanned aerial vehicle. Among them, the current position is used to calculate the spatial distance between the replenishment unmanned aerial vehicle and the target unmanned aerial vehicle, as one of the response priority indicators (the farther the spatial distance, the lower the response efficiency); the response time is used as another response priority indicator (the longer the response time, the lower the response efficiency). Further, the surplus power can be obtained by the surplus power prediction unit, and the surplus amount of extinguishing agent can be obtained by the extinguishing agent surplus prediction unit.

[0142] <1> surplus power prediction unit

[0143] The remaining time, power consumption rate, current flight speed, current flight height and current fire extinguishing agent remaining amount of the unmanned aerial vehicle are input into a time sequence prediction model (such as LSTM / GRU neural network), and the remaining power of each node is output. Specifically, the essence of the remaining power prediction is to capture the dynamic correlation between multiple features and power consumption through time sequence modeling, and to provide endurance capability prediction for task scheduling. Before prediction, the remaining time, power consumption rate, current flight speed, current flight height and current fire extinguishing agent remaining amount are first subjected to data cleaning, normalization processing and feature extraction, and the processed data is input into the trained time sequence prediction model (taking long short-term memory network as an example). In the LSTM, the input layer receives sequence data within a time window, and the input dimension is Kx5 (K is the length of the time window, and 5 is the number of features); in the LSTM hidden layer, the number of layers is selected according to the data complexity (such as 1-3 layers), and each layer contains N hidden units (such as N=64); in the output layer, the output of the LSTM is mapped to a single value prediction through a fully connected layer (Dense), and the remaining power is output.

[0144] <2> Fire extinguishing agent remaining amount prediction unit

[0145] The executed fire extinguishing area of the unmanned aerial vehicle, the average temperature of the management unit, the average fire extinguishing time of each node and the flame spread speed are input into a regression model (such as gradient boosting tree XGBoost / LightGBM), and the fire extinguishing agent remaining amount of each node is output. Specifically, the gradient boosting tree algorithm can capture the complex nonlinear relationship between features (such as the synergistic effect of temperature and spread speed), is not sensitive to outliers, is suitable for processing noise that may exist in sensor data, and can output the contribution of each feature to the prediction result, which helps to understand the key driving factors of fire extinguishing agent consumption. In actual tasks, select the appropriate gradient boosting tree according to the data size, accuracy requirement, scene complexity and real-time requirement. In this embodiment, the LightGBM model is selected, which is suitable for scenarios with large data size and high real-time requirement (such as dynamic update of cellular subnets). Similarly, before prediction, the fire extinguishing area, the average temperature of the management unit, the average fire extinguishing time of each node and the flame spread speed are subjected to data cleaning, normalization processing and feature extraction. The processed data is input into the trained LightGBM model. In the LightGBM model, the model outputs the fire extinguishing agent remaining amount by weighted sum of the prediction results of multiple trees.

[0146] 2) Queue generation module

[0147] For generating a response priority queue according to each supply response signal. It includes:

[0148] <1> Distance calculation unit

[0149] a spatial distance between the current position in the replenishment request signal and the current position in each replenishment response signal is calculated.

[0150] <2> priority calculation unit

[0151] when the replenishment request type is the power replenishment request, a first priority calculation model is used to calculate the response priority of each unmanned aerial vehicle, and when the replenishment request type is the fire extinguishing agent replenishment request, a second priority calculation model is used to calculate the response priority of each unmanned aerial vehicle.

[0152] The expression of the first priority calculation model is: p 1= b 1×spatial distance+ b 2×response time+ b 3×surplus power;

[0153] The expression of the second priority calculation model is: p 2= b 1×spatial distance+ b 2×response time+ b 4×fire extinguishing agent surplus amount;

[0154] wherein, p 1 is the response priority when the replenishment request type is the power replenishment request, p 2 is the response priority when the replenishment request type is the fire extinguishing agent replenishment request, b 1 is a weight coefficient of the spatial distance, b 2 is a weight coefficient of the response time, b 3 is a weight coefficient of the surplus power, b 4 is a weight coefficient of the fire extinguishing agent surplus amount.

[0155] <3> priority sorting unit

[0156] The response priority of each unmanned aerial vehicle is sorted in descending order of value to obtain a response priority queue.

[0157] (4) flight control module

[0158] When the unmanned aerial vehicle is located at the head of the response priority queue, the flight control module controls the unmanned aerial vehicle to fly to a signal collection area close to the current position in the replenishment request signal.

[0159] When the unmanned aerial vehicle is located at the head of the response priority queue, it means that the unmanned aerial vehicle has the highest priority compared with the other unmanned aerial vehicles, so it is determined to perform the current replenishment task. The flight control module randomly generates a target address (which is close to the current position in the replenishment request signal) and controls the unmanned aerial vehicle to fly to the signal collection area where the target address is located.

[0160] (5) Signal acquisition module

[0161] When the UAV flies to the signal acquisition area, the multispectral signals in the area are immediately acquired to generate a multispectral image. Corresponding to the types of the multispectral signals described above, the signal acquisition module includes:

[0162] 1) First signal acquisition unit

[0163] used for acquiring the visible light signals in the multispectral signals in the signal acquisition area in real time to generate corresponding visible light images. The first signal acquisition unit can also be selected as a visible light camera.

[0164] 2) Second signal acquisition unit

[0165] used for acquiring the infrared light signals in the multispectral signals in the signal acquisition area in real time to generate corresponding thermal imaging images. The second signal acquisition unit can also be selected as a thermal imaging infrared sensor.

[0166] 3) Third signal acquisition unit

[0167] used for acquiring the infrared light signals in the multispectral signals in the signal acquisition area in real time to generate corresponding ultraviolet images. The third signal acquisition unit can be selected as an ultraviolet light sensor, such as a photodiode type ultraviolet light sensor, a photoresistor type ultraviolet light sensor, etc.

[0168] The images output by the above three signal acquisition units are different imaging results of the concentric double-ring structure.

[0169] (6) Real-time positioning module

[0170] used for obtaining the real-time positioning of the target UAV according to the multispectral image.

[0171] After the multispectral image is acquired, the real-time positioning module performs image processing based on a perspective image processing method to obtain the real-time positioning of the target UAV. As known from the above, the real-time positioning of the target UAV is reflected by the flight attitude data. Specifically, the real-time positioning of the target UAV includes the coordinates of the target UAV in the camera coordinate system and the actual roll angle of the target UAV. The coordinates of the target UAV in the camera coordinate system (represented by the pitch angle and the azimuth angle) and the actual roll angle of the target UAV. Therefore, the positioning module of the present embodiment includes:

[0172] 1) Target recognition unit

[0173] The first encoding result corresponding to the first LED ring array is extracted from the multispectral image, it is judged whether the first encoding result matches the identity data in the replenishment request signal, if it matches, the unmanned aerial vehicle emitting the multispectral signal is marked as a target unmanned aerial vehicle, and the distance calculation unit, the azimuth angle calculation unit, the pitch angle calculation unit, the three-dimensional coordinate reconstruction unit and the roll angle calculation unit are driven to work, if it does not match, the signal acquisition module is driven to collect other multispectral signals.

[0174] 2) Distance calculation unit

[0175] The actual distance between the unmanned aerial vehicle and the target unmanned aerial vehicle is calculated.

[0176] The actual distance calculation formula is: ; wherein, d represents the actual distance between the unmanned aerial vehicle and the target unmanned aerial vehicle, f represents the focal length of the first signal acquisition unit, L 1 represents the actual diameter of the concentric double-ring structure, L 2 represents the diameter of the concentric double-ring structure in the visible light image;

[0177] 3) Azimuth angle calculation unit

[0178] The horizontal pixel offset between the center coordinates of the multispectral image and the center coordinates of the concentric double-ring structure in the multispectral image is obtained, and the horizontal pixel offset is input into the azimuth angle calculation formula to output the actual azimuth angle of the target unmanned aerial vehicle.

[0179] The azimuth angle calculation formula is: ; wherein, A represents the actual azimuth angle of the target unmanned aerial vehicle, dx represents the horizontal pixel offset between the center coordinates of the visible light image and the center coordinates of the concentric double-ring structure in the visible light image, W represents the image width of the visible light image, FOV x represents the horizontal field of view angle of the first signal acquisition unit.

[0180] 4) Pitch angle calculation unit

[0181] The vertical pixel offset between the center coordinates of the multispectral image and the center coordinates of the concentric double-ring structure in the multispectral image is obtained, and the vertical pixel offset is input into the pitch angle calculation formula to output the actual pitch angle of the target unmanned aerial vehicle.

[0182] The pitch angle calculation formula is: ; wherein, E represents the actual pitch angle of the target unmanned aerial vehicle, dyrepresents a vertical pixel offset between a center coordinate of the visible light image and a center coordinate of the concentric double-ring structure in the visible light image, H represents an image height of the visible light image, FOV y represents a vertical field of view angle of the first signal acquisition unit.

[0183] 5)three-dimensional coordinate reconstruction unit

[0184] For obtaining the coordinate in the camera coordinate system of the target UAV according to the actual azimuth angle and the actual pitch angle of the target UAV. The coordinate in the camera coordinate system of the target UAV is: .

[0185] 6)roll angle calculation unit

[0186] For extracting the center pixel point coordinates of the plurality of LED units of the second LED ring array in the multispectral image, inputting each center pixel point coordinate into the OpenCV software library to perform ellipse fitting on the second LED ring array, obtaining a plane equation (ellipse equation) of the second LED ring array, inputting the ellipse equation into the OpenCV software library, and outputting the actual roll angle of the target UAV.

[0187] It should be understood that the "system", "device", "unit" and / or "module" used in the specification is a method for distinguishing different components, elements, parts, portions or assemblies at different levels. However, if other words can achieve the same purpose, the words can be replaced by other expressions.

[0188] As shown in the specification and claims, unless the context clearly indicates otherwise, "one", "a", "an" and / or "the" do not refer to the singular, but can also include the plural. Generally, the terms "comprise" and "include" only indicate the inclusion of the steps and elements explicitly identified, and these steps and elements do not constitute an exclusive list, and the method or device can also include other steps or elements.

[0189] The above specific embodiments further illustrate the purpose, technical solutions and beneficial effects of the present application. It should be understood that the above description is only a specific embodiment of the present application and does not limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. within the spirit and principles of the present application shall be included in the protection scope of the present application.

[0190] It should be noted that the structure, proportion, size, etc. shown in the drawings attached to the present specification are only used to cooperate with the content disclosed in the specification, so that those skilled in the art can understand and read, and are not used to limit the conditions that the present application can be implemented, so they do not have technical significance. Any modification of the structure, change of the proportion relationship or adjustment of the size, without affecting the effect and purpose that the present application can produce, should still fall within the scope of the technical content disclosed by the present application. At the same time, the terms such as "upper", "lower", "left", "right", "middle" and the like in the present specification are only for the convenience of clear description, and are not used to limit the scope of the present application, and the change or adjustment of the relative relationship is also considered as the implementation of the present application without substantial changes in technical content.

Claims

1. A target real-time positioning system for resource complementary supply in multi-UAV cooperative fire extinguishing, characterized in that, The application relates to a fire extinguishing system comprising a mesh network composed of multiple unmanned aerial vehicles (UAVs), each UAV comprising a fire perception module configured to perceive real-time fire conditions in a task execution area of the UAV; a data analysis module configured to input the real-time fire conditions and real-time state of the UAV into a trained random forest model to output a classification result, the classification result comprising: unable to complete a fire extinguishing task and able to complete the fire extinguishing task; a selection trigger module configured to trigger a supply request module, a data encoding module and a signal emission module to work when the UAV is unable to complete the fire extinguishing task, and trigger a supply response module, a queue generation module, a flight control module, a signal acquisition module, a real-time positioning module and a target tracking module to work when the UAV is able to complete the fire extinguishing task; the supply request module is configured to send a supply request signal to the remaining UAVs through the mesh network; the data encoding module is configured to encode real-time identity data and current attitude data of the UAV; the signal emission module is configured to emit a multi-spectral signal to the outside world according to the real-time encoding data; the supply response module is configured to send a supply response signal to the remaining UAVs through the mesh network when the supply request signal is received; the queue generation module is configured to generate a response priority queue according to the supply response signals; the flight control module is configured to control the UAV to fly to a signal acquisition area close to a current position in the supply request signal when the UAV is located at the head of the response priority queue; the signal acquisition module is configured to acquire a multi-spectral signal in the signal acquisition area to generate a multi-spectral image; and the real-time positioning module is configured to obtain real-time positioning of a target UAV according to the multi-spectral image. Each UAV further comprises a real-time positioning module configured to obtain a current position of the UAV in real time and send the current position to the remaining UAVs through the mesh network; a state monitoring module configured to monitor a real-time state of the UAV and send the real-time state to the remaining UAVs through the mesh network, the real-time state comprising: remaining power, extinguishing agent amount, communication radius, received signal strength indicator and computing power; a network topology module configured to establish a hierarchical network topology of the multiple UAVs according to the real-time states, the hierarchical network topology comprising: a backbone layer node and multiple access layer nodes; the network topology module comprises: a weight calculation unit configured to obtain a comprehensive weight of the UAV in the mesh network according to the real-time state and send the comprehensive weight to the remaining UAVs through the mesh network; a first screening unit configured to screen out multiple UAVs corresponding to the comprehensive weight greater than a threshold value; and a second screening unit configured to screen out a UAV closest to a geometric center or a geometric gravity center of the network topology as a backbone layer UAV according to real-time positions of the multiple screened UAVs, and take the remaining UAVs in the mesh network as access layer nodes. Each UAV further comprises a task allocation module configured to generate a fire extinguishing task of each UAV according to the real-time fire conditions and the real-time states; and the selection trigger module is further configured to trigger the task allocation module to work when the UAV is the backbone layer node; the task allocation module comprises: ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ 2.The target real-time positioning system for resource complementary supply in multi-UAV cooperative fire extinguishing according to claim 1, wherein, ​ ​ ​ ​ ​ ​ ​ ​ 3.The target real-time positioning system for resource complementary supply in multi-UAV cooperative fire extinguishing according to claim 2, wherein, ​ ​ ​ The fire analysis unit is configured to divide the fire area into a plurality of task execution areas according to real-time fire conditions, wherein the task execution areas include a flame front area, a flame spread area and a cooling area; The priority quantification unit is configured to quantize the priority of each flame front area, the priority of each flame spread area and the priority of each cooling area by using a fire condition priority quantification model; The priority matching unit is configured to match the comprehensive weight of each unmanned aerial vehicle with the quantified priority by using a support vector machine to establish a priority matching table; The task allocation unit is configured to allocate a corresponding fire extinguishing task to each unmanned aerial vehicle by using a multi-objective optimization algorithm according to the priority matching table, wherein one fire extinguishing task corresponds to one task execution area.

4. The target real-time positioning system for resource complementary supply of multiple unmanned aerial vehicles in cooperative fire extinguishing according to claim 3, wherein the fire condition sensing module comprises: Comprehensive weight = a 1 x remaining power + a 2 x extinguishing agent remaining amount + a 3 x communication radius + a 4 x received signal strength indicator + a 5 x computing power; wherein, a 1 is the weight coefficient of the remaining power, a 2 is the weight coefficient of the extinguishing agent remaining amount, a 3 is the weight coefficient of the communication radius, a 4 is the weight coefficient of the received signal strength indicator, a 5 is the weight coefficient of the computing power; The expression of the fire priority quantification model is: (1); in formula (1), P is a fire priority quantification value, w 1 represents a weight of the flame spread speed, w 2 represents a weight of the fire influence area, V represents the flame spread speed, V max represents the maximum flame spread speed, S represents the fire influence area, S max represents the maximum fire influence area.

5. The target real-time positioning system for resource complementary supply in multi-UAV cooperative fire extinguishing according to claim 3, characterized in that, a visible light camera configured to collect a fire condition image in the task execution area of the unmanned aerial vehicle in real time; a thermal imaging infrared sensor configured to collect a thermal imaging image in the task execution area of the unmanned aerial vehicle in real time; a temperature sensor configured to collect a temperature distribution map in the task execution area of the unmanned aerial vehicle in real time; a feature extraction unit configured to fuse the fire condition image, the thermal imaging image and the temperature distribution map by using an image fusion algorithm, extract features of the fused image by using a convolutional neural network, and obtain feature information; an information processing unit configured to analyze the feature information by using a deep learning model to obtain real-time fire conditions, wherein the real-time fire conditions include a fire influence area and a flame spread speed. The data encoding module comprises:

6. The target real-time positioning system for resource complementary supply in multi-UAV cooperative fire extinguishing according to claim 5, characterized in that, a first encoding unit configured to perform Manchester encoding on the identity data to obtain a first encoding result; a second encoding unit configured to perform Gray encoding on the attitude data to obtain a second encoding result, wherein the attitude data includes a pitch angle, a yaw angle and a roll angle. The data encoding module further comprises:

7. The target real-time positioning system for resource complementary supply in multi-UAV cooperative fire extinguishing according to claim 6, characterized in that, a PPS signal modulation unit configured to modulate the first encoding result and the second encoding result into a PPS signal, and transmit the PPS signal to the other unmanned aerial vehicles through a mesh network. The signal transmitting module comprises a first LED ring array and a second LED ring array, and the first LED ring array and the second LED ring array form a concentric double-ring structure; 8. The target real-time positioning system for resource complementary supply in multi-UAV cooperative fire extinguishing according to claim 6 or 7, characterized in that, The second LED ring array comprises three sub-zones corresponding to the pitch angle, the yaw angle and the roll angle, each sub-zone comprises four LED units arranged at equal intervals in a clockwise or counterclockwise direction, each LED unit is configured to emit visible light signals, infrared light signals and ultraviolet light signals to the outside, and along the arrangement direction, the first LED unit corresponds to the binary code 00, the second LED unit corresponds to the binary code 01, the third LED unit corresponds to the binary code 10, and the fourth LED unit corresponds to the binary code 11; The first encoding result is used to control the multi-spectrum signal emission state of each LED unit in the first LED ring array; The second encoding result is used to control the multi-spectrum signal emission sequence of each LED unit in the second LED ring array. Each unmanned aerial vehicle further comprises: 9.The target real-time positioning system for resource complementary supply in multi-UAV cooperative fire extinguishing according to claim 5, wherein, ​ The surplus power prediction unit is configured to input the endurance time, the power consumption rate, the current flight speed, the current flight height and the current fire extinguishing agent surplus of the unmanned aerial vehicle into a trained time series prediction model, and output the surplus power of the unmanned aerial vehicle; The fire extinguishing agent surplus prediction unit is configured to input the executed fire extinguishing area, the average temperature in the task execution area, the average fire extinguishing time of the unmanned aerial vehicle and the flame spread speed in the task execution area into a trained regression model, and output the fire extinguishing agent surplus of each node.

10. The target real-time positioning system for resource complementary supply in multi-unmanned aerial vehicle cooperative fire extinguishing according to claim 9, characterized in that, The supply request signal carries the identity data, the current position and the supply request type of the unmanned aerial vehicle, and the supply request type includes power supply request and fire extinguishing agent supply request; The supply response signal carries the identity data, the current position, the surplus power, the fire extinguishing agent surplus and the response time of the unmanned aerial vehicle; The queue generation module comprises: The distance calculation unit is configured to calculate the spatial distance between the current position in the supply request signal and the current position in each supply response signal; The priority calculation unit is configured to calculate the response priority of each unmanned aerial vehicle by using a first priority calculation model when the supply request type is power supply request, and calculate the response priority of each unmanned aerial vehicle by using a second priority calculation model when the supply request type is fire extinguishing agent supply request; The priority sorting unit is configured to sort the response priority of each unmanned aerial vehicle in descending order to obtain a response priority queue; The expression of the first priority calculation model is: p 1= b 1 x spatial distance + b 2 x response time + b 3 x surplus power; The expression of the second priority calculation model is: p 2= b 1 x spatial distance b 2 x response time b 4 x extinguishing agent surplus wherein, p 1 is a response priority when the replenishment request type is an electric power replenishment request, p 2 is a response priority when the replenishment request type is a fire extinguishing agent replenishment request, b 1 is a weight coefficient of a spatial distance, b 2 is a weight coefficient of a response time, b 3 is a weight coefficient of a surplus electric power, b 4 is a weight coefficient of a surplus fire extinguishing agent.

11. The target real-time positioning system for resource complementary supply in multi-UAV cooperative fire extinguishing according to claim 10, characterized in that, The signal acquisition module comprises: The first signal acquisition unit is configured to acquire the visible light signal in the multi-spectral signal in the signal acquisition area in real time to generate a corresponding visible light image; The second signal acquisition unit is configured to acquire the infrared light signal in the multi-spectral signal in the signal acquisition area in real time to generate a corresponding thermal imaging image; The third signal acquisition unit is configured to acquire the infrared light signal in the multi-spectral signal in the signal acquisition area in real time to generate a corresponding ultraviolet image.

12. The target real-time positioning system for resource complementary supply in multi-UAV cooperative fire extinguishing according to claim 11, characterized in that, The real-time positioning of the target unmanned aerial vehicle comprises the coordinate of the target unmanned aerial vehicle in the camera coordinate system and the actual roll angle of the target unmanned aerial vehicle; The real-time positioning module comprises: The target identification unit is configured to extract the first encoding result corresponding to the first LED ring array from the multi-spectral image, judge whether the first encoding result matches the identity data in the supply request signal, mark the unmanned aerial vehicle emitting the multi-spectral signal as the target unmanned aerial vehicle if the first encoding result matches the identity data, and drive the distance calculation unit, the azimuth calculation unit, the pitch angle calculation unit, the three-dimensional coordinate reconstruction unit and the roll angle calculation unit to work, and drive the signal acquisition module to acquire other multi-spectral signals if the first encoding result does not match the identity data; The distance calculation unit is configured to calculate the actual distance between the unmanned aerial vehicle and the target unmanned aerial vehicle; The azimuth calculation unit is configured to obtain the horizontal pixel offset between the center coordinate of the multi-spectral image and the center coordinate of the concentric double-ring structure in the multi-spectral image, input the horizontal pixel offset into an azimuth calculation formula, and output the actual azimuth of the target unmanned aerial vehicle. The pitch angle solving unit is configured to obtain a vertical pixel offset between a center coordinate of the multispectral image and a center coordinate of the concentric double-ring structure in the multispectral image, input the vertical pixel offset into a pitch angle solving formula, and output an actual pitch angle of the target UAV. The three-dimensional coordinate reconstruction unit is configured to obtain a coordinate in a camera coordinate system of the target UAV according to the actual azimuth angle and the actual pitch angle. The roll angle solving unit is configured to extract center pixel point coordinates of a plurality of LED units of the second LED ring array in the multispectral image, perform elliptical fitting on the second LED ring array according to the plurality of center point pixel coordinates, input an elliptical fitting result into OpenCV, and output an actual roll angle of the target UAV.

13. The target real-time positioning system for resource complementary supply in multi-UAV cooperative fire extinguishing according to claim 12, characterized in that, The actual distance calculation formula is: ; wherein, d represents the actual distance between the unmanned aerial vehicle and the target unmanned aerial vehicle, f represents the focal length of the first signal acquisition unit, L 1 represents the actual diameter of the concentric double-ring structure, L 2 represents the diameter of the concentric double-ring structure in the visible light image; The azimuth angle solving formula is: ; wherein, A represents the actual azimuth angle of the target unmanned aerial vehicle, dx represents the horizontal pixel offset between the center coordinate of the visible light image and the center coordinate of the concentric double-ring structure in the visible light image, W represents the image width of the visible light image, FOV x represents the horizontal field of view angle of the first signal acquisition unit; The pitch angle calculation formula is: ; wherein, E represents the actual pitch angle of the target unmanned aerial vehicle, dy represents the vertical pixel offset between the center coordinates of the visible light image and the center coordinates of the concentric double-ring structure in the visible light image, H represents the image height of the visible light image, FOV y represents the vertical field of view angle of the first signal acquisition unit; The coordinates in the target unmanned aerial vehicle camera coordinate system are: .

Images