Smart forest fire prevention monitoring and early warning method and device based on unmanned aerial vehicle group

Abstract

The invention discloses a smart forest fire prevention monitoring and early warning method and device based on an unmanned aerial vehicle group, and relates to the technical field of forest fire prevention monitoring, and the method comprises the steps: obtaining a patrol topological graph of a current environment, real-time map information, the initial position of each unmanned aerial vehicle, and the current observation state of each unmanned aerial vehicle; establishing an unmanned aerial vehicle environment space model based on the acquired information; inputting the current observation state of the unmanned aerial vehicle into the trained multi-agent reinforcement learning model, and determining the optimal patrol node of each unmanned aerial vehicle at the next moment; each unmanned aerial vehicle is controlled to execute a patrol task, and the position and the current observation state of each unmanned aerial vehicle in the unmanned aerial vehicle environment space model are updated; and based on the updated current observation state of each unmanned aerial vehicle, re-determining the optimal patrol node at the next moment until all patrol nodes in the patrol topological graph are covered. The problems that unmanned aerial vehicle patrol is low in efficiency and the patrol algorithm is not robust are solved, and the patrol efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of forest fire monitoring technology, and in particular to an intelligent forest fire monitoring and early warning method and device based on a swarm of drones. Background Art

[0002] Patrolling, i.e., continuous monitoring of the environment, is an important activity in related application fields such as environmental safety, forest fire monitoring and prevention. Using multiple drones for patrolling simultaneously helps improve patrol performance and helps to detect abnormal factors in a timely manner. Therefore, there have been different research perspectives in the past to analyze and study the multi-robot patrol (MRP) problem, including problem modeling, optimal solution, theoretical analysis, implementation, and experimental verification.

[0003] There have been many theoretical studies on the Multi-Robot Patrol Problem (MRPP), and it has been shown that this problem is NP-hard. It is generally believed in the literature that when the environment is represented topologically and global / centralized information is used, if all robots follow the same TSP or Hamiltonian circuit and are evenly distributed in time and space, the optimal patrol path can be obtained and the optimal patrol effect can be achieved. However, in sparse topologies (the case of most real-world environments), these cycles are not easy to compute and do not even exist in most cases.

[0004] Therefore, the research on patrol problems needs to be oriented towards fully autonomous agents. Agents independently determine their patrol routes based on the state of the system without the need for a central planner, which allows agents to effectively coordinate autonomously, and this method is robust to robot failures. Compared with centralized solutions, decentralized methods have no single point of failure and provide more flexible application implementations. However, other decentralized methods such as DTAG based on task allocation, DTAP algorithm, and Bayesian method CBLS algorithm need to frequently resolve conflicts in order to adapt to the dynamic changes in the environment. The amount of calculation is large and time-consuming, and the calculation results may not be optimal (need to be explored), and the strategy is not robust enough.

[0005] The existing patrol method based on multi-agent reinforcement learning algorithm does not require prior knowledge of the environment, can effectively achieve cooperation, better explore and learn some better actions, and produce more robust strategies. However, the patrol algorithm based on reinforcement learning still has shortcomings. On the one hand, due to partial observation and non-stationary problems, individual exploration conflicts occur, strategies are unstable, and the effect is poor. On the other hand, due to the inconsistency between local and global information, the agent needs to strike a balance between local and global perspectives, otherwise, it may lead to insufficient or redundant exploration. Summary of the invention

[0006] The purpose of this application is to provide a smart forest fire monitoring and early warning method and device based on drone swarms, which can solve the problems of inefficient drone patrols and non-robust algorithms and improve patrol efficiency.

[0007] To achieve the above objectives, this application provides the following solutions:

[0008] In the first aspect, the present application provides a smart forest fire monitoring and early warning method based on drone swarms, including:

[0009] Obtain patrol topology, real-time map information and drone swarm information of the current environment; the drone swarm information includes the starting position of each drone and the current observation status of the drone; the current observation status of the drone includes the patrol nodes within the drone observation range and the status of other drones within the drone observation range; the patrol topology includes several patrol nodes.

[0010] A drone environment spatial model is established based on the patrol topology map, real-time map information and drone swarm information.

[0011] The current observation state of the drone is input into a trained multi-agent reinforcement learning model to determine the optimal patrol node of each drone in the drone group at the next moment; the multi-agent reinforcement learning model is a model based on the MAPPO algorithm.

[0012] According to the optimal patrol node of each drone in the drone swarm at the next moment, each drone is controlled to perform the patrol mission, and the position and current observation status of each drone in the drone environment space model are updated.

[0013] Based on the updated current observation status of each drone, re-execute the following steps: input the current observation status of the drone into the trained multi-agent reinforcement learning model to determine the optimal patrol node of each drone in the drone swarm at the next moment until all patrol nodes in the patrol topology map are covered.

[0014] Optionally, the training process of the multi-agent reinforcement learning model is:

[0015] Build a multi-agent reinforcement learning model.

[0016] Initialize the actor network parameters, critic network parameters and experience replay pool in the multi-agent reinforcement learning model, and set the learning rate.

[0017] For each UAV, the current observation state of the UAV is used as the input of the actor network in the MAPPO algorithm, and the UAV action is calculated through the actor network.

[0018] Based on the accident minimization reinforcement learning method, the drone's actions interact with the environment to obtain rewards from environmental feedback and minimize accidental rewards.

[0019] Execute the drone action and get the experience tuple for the current time step when the drone moves to the next position.

[0020] Based on the experience tuple at each time step, the advantage function and complete trajectory of the drone swarm are determined.

[0021] Split the complete trajectory into chunks of length L and store the chunks into the experience replay pool.

[0022] When the storage capacity of the experience replay pool reaches the maximum, a set number of samples are drawn from the experience replay pool.

[0023] Based on the samples, update the weights of the actor network and the critic network in the multi-agent reinforcement learning model.

[0024] Until the actor network and the critic network converge, a trained multi-agent reinforcement learning model is obtained.

[0025] Optionally, the experience tuple is specifically:

[0026] e t =(s t , o t ,h t，π ,h t，V ,u t , r t ,s t+1 , o t+1 ).

[0027] Among them, s t is the global state at time t; o t is the state perceived by the drone at time t; h t，π is the hidden state of the actor network, h t，V is the hidden state of the critic network, u t is the drone action at time t, r t An immediate reward consisting of a reward for environmental feedback and minimizing unexpected rewards; t+1 is the global state at time t+1; o t+1 The state perceived by the drone at time t+1.

[0028] Optionally, based on the experience tuple at each time step, the advantage function and the complete trajectory of the drone swarm are determined, including:

[0029] According to the formula Determine the advantage function of the drone swarm; where, γ is the discount factor, λ is the GAE parameter, T is the time step, For criticism c The output of the network.

[0030] Based on a large number of experience tuples generated during the interaction between the drone and the environment, a complete trajectory is obtained.

[0031] Optionally, the block with a length of L is specifically:

[0032]

[0033] Among them, τ[l:l+L] is a trajectory sequence of length L, is of length L Advantage value sequence, is a discounted return sequence of length L.

[0034] Optionally, the updating of the actor network in the multi-agent reinforcement learning model, c Riti c The weight of the network, specifically including:

[0035] Update the actor network according to maximizing the objective function L(θ).

[0036] Update the critic network by minimizing the value function L(φ).

[0037] Optionally, the maximization objective function L(θ) is specifically:

[0038]

[0039] Among them, B is the batch size, n is the number of agents, is the parameter update ratio, is the observation value of agent i, is the current action taken by agent i, is the current network strategy, To update the previous network strategy, ∈ is a hyperparameter used to control the range of clipping, and clip() is a parameter clipping method. Limited to [1-∈, 1+∈], To obtain the advantage function, is the strategy taken according to the observed value, H is the entropy of the strategy, and λ is the coefficient of controlling entropy.

[0040] The minimization value function L(φ) is specifically:

[0041]

[0042] in, For discount rewards, is the state value output by the critic network, For the crit before the update c The state value output by the network.

[0043] In the second aspect, the present application provides a smart forest fire monitoring and early warning device based on a drone swarm, comprising:

[0044] The information acquisition module is used to obtain the patrol topology map, real-time map information and drone swarm information of the current environment; the drone swarm information includes the starting position of each drone and the current observation status of the drone; the current observation status of the drone includes the patrol nodes within the drone observation range and the status of other drones within the drone observation range; the patrol topology map includes a number of patrol nodes.

[0045] The environment space model building module is used to build the drone environment space model based on the patrol topology map, real-time map information and drone group information.

[0046] The computing module is used to input the current observation state of the drone into the trained multi-agent reinforcement learning model to determine the optimal patrol node of each drone in the drone group at the next moment; the multi-agent reinforcement learning model is a model based on the MAPPO algorithm.

[0047] The status update module is used to control each drone in the drone swarm to perform patrol tasks according to the optimal patrol node of each drone at the next moment, and to update the position and current observation status of each drone in the drone environment space model.

[0048] The observation module is used to re-execute the steps based on the updated current observation status of each drone: input the current observation status of the drone into the trained multi-agent reinforcement learning model, determine the optimal patrol node of each drone in the drone group at the next moment, until all patrol nodes in the patrol topology map are covered.

[0049] Optionally, a model training module is also included.

[0050] The model training module specifically includes:

[0051] The learning model construction submodule is used to build a multi-agent reinforcement learning model.

[0052] The parameter initialization submodule is used to initialize the actor network parameters, critic network parameters and experience replay pool in the multi-agent reinforcement learning model, and set the learning rate.

[0053] The action calculation submodule is used to calculate the current observation state of each drone as the input of the actor network in the MAPPO algorithm, and obtain the drone action through the actor network.

[0054] The reward calculation submodule is used to interact the drone's actions with the environment based on the accident minimization reinforcement learning method, obtain rewards from environmental feedback, and minimize accidental rewards.

[0055] The action execution submodule is used to execute the UAV action and obtain the experience tuple of the current time step when the UAV moves to the next position.

[0056] The advantage function and complete trajectory determination submodule is used to determine the advantage function and complete trajectory of the drone swarm based on the experience tuple at each time step.

[0057] The trajectory splitting submodule is used to split the complete trajectory into a trajectory sequence of length L and store the trajectory sequence in the experience replay pool.

[0058] The sample extraction submodule is used to extract a set number of samples from the experience replay pool when the storage capacity of the experience replay pool reaches the maximum.

[0059] The model training submodule is used to update the weights of the actor network and the critic network in the multi-agent reinforcement learning model based on the samples until the actor network and the critic network converge to obtain a trained multi-agent reinforcement learning model.

[0060] Optionally, the advantage function and complete trajectory determination submodule specifically include:

[0061] Advantage function determination unit, used to determine the formula Determine the advantage function of the drone swarm; where, γ is the discount factor, λ is the GAE parameter, T is the time step, is the output of the critic network.

[0062] The complete trajectory determination unit is used to obtain a complete trajectory based on a large number of experience tuples generated during the interaction between the UAV and the environment.

[0063] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0064] The present application provides a method and device for monitoring and early warning of forest fire prevention based on a drone swarm. The method includes: obtaining the patrol topology map, real-time map information and drone swarm information of the current environment is the first step, which helps to understand the area that the drone needs to cover and the current environmental status, and provides basic data for subsequent decision-making. Establishing a drone environment space model is the second step. By combining the patrol topology map, real-time map information and drone swarm information, an accurate environmental model can be constructed to provide a basis for the path planning and decision-making of the drone. Inputting the current observation state of the drone into the trained multi-agent reinforcement learning model, and determining the optimal patrol node of each drone in the drone swarm at the next moment is the third step. This step uses advanced algorithms to dynamically adjust the patrol path of the drone to ensure the efficient execution of the patrol mission. According to the optimal patrol node of each drone in the drone swarm at the next moment, controlling each drone to perform the patrol mission, and updating the position and current observation status of each drone in the drone environment space model is the fourth step. This ensures that the drone can be adjusted according to the latest environmental information and mission requirements, improving the flexibility and efficiency of patrols. Finally, based on the updated current observation status of each drone, the current observation status of the drone is input into the trained multi-agent reinforcement learning model again to determine the optimal patrol node of each drone in the drone group at the next moment until all patrol nodes in the patrol topology map are covered. This cyclic process ensures that drones can continuously and efficiently cover all patrol nodes, solving the problems of low efficiency and non-robust algorithms in traditional methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0066] Figure 1 A flowchart of a smart forest fire monitoring and early warning method based on a drone swarm is provided in accordance with an embodiment of the present application.

[0067] Figure 2 A schematic diagram of a network structure provided for an embodiment of the present application.

[0068] Figure 3 A schematic diagram of the functional modules of a smart forest fire monitoring and early warning device based on a drone swarm provided in one embodiment of the present application. DETAILED DESCRIPTION

[0069] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0070] Forests cover a large area, and fire is one of the biggest factors in forest damage. Existing forest fire prevention generally adopts the construction of watchtowers for manual inspection and the establishment of video surveillance systems to achieve fire prevention. The biggest problem with artificial watchtowers is that they require monitoring personnel to pay attention at all times, which can easily lead to missed reports or untimely reports; the establishment of a video surveillance system for fire prevention is to summarize the collected videos and then complete centralized monitoring manually, which also lacks intelligence and has high labor costs. On the other hand, the use of drone patrols mostly adopts fixed route patrols, the patrol algorithm is relatively simple, and the algorithm strategy is not robust enough.

[0071] The purpose of this application is to provide a smart forest fire monitoring and early warning method and device based on drone swarms, which can solve the problems of inefficient drone patrols and non-robust algorithms and improve patrol efficiency.

[0072] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0073] Embodiment 1

[0074] like Figure 1 As shown, this embodiment provides a smart forest fire monitoring and early warning method based on a drone swarm, including:

[0075] Step 101: Obtain a patrol topology map, real-time map information and drone swarm information of the current environment; the drone swarm information includes the starting position of each drone and the current observation status of the drone; the current observation status of the drone includes the patrol nodes within the drone observation range and the status of other drones within the drone observation range; the patrol topology map includes a number of patrol nodes.

[0076] Step 102: Establish a drone environment space model based on the patrol topology map, real-time map information and drone group information.

[0077] Step 103: Input the current observation state of the drone into the trained multi-agent reinforcement learning model to determine the optimal patrol node of each drone in the drone group at the next moment; the multi-agent reinforcement learning model is a model based on the MAPPO algorithm.

[0078] Step 104: According to the optimal patrol node of each drone in the drone group at the next moment, each drone is controlled to perform a patrol mission, and the position and current observation status of each drone in the drone environment space model are updated.

[0079] Step 105: Based on the updated current observation status of each drone, re-execute step: input the current observation status of the drone into the trained multi-agent reinforcement learning model to determine the optimal patrol node of each drone in the drone group at the next moment until all patrol nodes in the patrol topology map are covered.

[0080] In some embodiments, when executing steps 101-102, the specific steps may be as follows:

[0081] Obtain the current environment patrol topology and real-time map information so that the drone can obtain patrol node information within the observable range in real time. The state of the drone includes the patrol nodes and other drone information within the drone's observation range as the state. The nodes are given by the topology map pre-generated in the previous step of the environment space. At this time, each drone knows the structural information of the graph in advance, including the number and location of nodes, and the connection status of the edges between nodes.

[0082] Based on the patrol topology map, real-time map information and drone group information, the drone environment space model is established. The simulation scene (environmental space) is a two-dimensional space composed of drones and patrol topology maps. The starting position of the drone and the position of the node in the topology map are randomly set in the two-dimensional Cartesian rectangular coordinate system.

[0083] When executing step 103, the specific steps may be as follows:

[0084] Get the current observation status of the drone.

[0085] Specifically, the state o perceived by the drone at the current time t t =(o i , o j ), where o i is the status of each patrol node observed by the UAV within the observation range, o j is the state of other drones within the observation range. At the same time, the global state s at the current time t is generated t At the beginning of each episode, the drone is placed at a random location in the environment topology map.

[0086] The current observation status of the drone is o t As the input of the MAPPO algorithm actor network in the trained multi-agent reinforcement learning model, the drone action u is calculated through the actor network t , according to the drone action ut As the optimal control action that the UAV needs to perform in the current state, the optimal patrol node of the UAV at the next moment is determined.

[0087] Among them, in this embodiment, if Figure 2 As shown in the figure, the multi-agent reinforcement learning algorithm used is the Multi-agent Proximal Policy Optimization algorithm, and the algorithm network model includes: multiple actor networks and multiple critic networks. Each actor network shares parameters, and the actor network maps from the state space to the action space, inputs the observed state, and outputs the action. Each critic evaluates the decisions made by multiple actor networks, inputs the global state, and outputs the generalized advantage estimate of the value of the joint action taken in this state. The actor and critic networks are updated accordingly.

[0088] Specifically, the training process of the multi-agent reinforcement learning model is:

[0089] Step 201: Build a multi-agent reinforcement learning model.

[0090] Step 202: Initialize the actor network parameters, critic network parameters and experience replay pool in the multi-agent reinforcement learning model, and set the learning rate.

[0091] Step 203: For each drone, the current observation state of the drone is used as the input of the actor network in the MAPPO algorithm, and the drone action is calculated through the actor network.

[0092] Step 204: Based on the surprise minimization reinforcement learning method, the drone action interacts with the environment to obtain rewards from environmental feedback and minimize surprise rewards.

[0093] Step 205: Execute the drone action, and when the drone moves to the next position, obtain the experience tuple of the current time step.

[0094] Step 206: Determine the advantage function and complete trajectory of the drone swarm based on the experience tuple at each time step.

[0095] Step 207: Split the complete trajectory into blocks of length L, and store the blocks in the experience replay pool.

[0096] Step 208: When the storage capacity of the experience replay pool reaches a maximum, a set number of samples are drawn from the experience replay pool.

[0097] Step 209: Based on the sample, update the weights of the actor network and the critic network in the multi-agent reinforcement learning model until the actor network and the critic network converge to obtain a trained multi-agent reinforcement learning model.

[0098] When executing steps 201-202, each drone a is regarded as an intelligent agent. At the beginning, the actor network parameters θ and the critic network parameters φ are initialized; the learning rate α is set, and the experience replay pool is initialized.

[0099] When executing step 203, the current observation state of the drone is t As the input of the MAPPO algorithm actor network, the action output u is calculated through the actor network t , as the control action that the drone needs to perform in the current state. The critic network receives the global state s t Output state value v t .

[0100] When executing step 204, the drone executes the action u given by the actor network in the MAPPO algorithm. t , and interact with the environment to get rewards from environmental feedback and minimizing unexpected rewards These two together constitute the instant reward r t , and the UAV moves to the next position to obtain the next observation state o t+1 , the global state is converted to s t+1 At this time, we get the experience tuple e at time t t =(s t , o t ,h t，π ,h t，V ,u t , r t ,s t+1 , o t+1 ), i.e., samples, and store the tuples into the trajectory τ.

[0101] Specifically, in patrol problems, patrol topology graphs are often sparse, and the number of graph nodes is often greater than the number of drones. Therefore, patrolling in a dynamic environment requires minimizing the impact of conflicts in drone access to nodes, requiring drone swarm collaborative exploration to further improve the robustness of cooperation between drones, and is conducive to accelerating strategy exploration. Based on this, this embodiment proposes a technology based on surprise minimization reinforcement learning, which gives surprise minimization rewards to environmental conflicts to achieve the above purpose.

[0102] The surprise minimization reward of this embodiment is understood as follows: In a realistic open world environment, natural forces that drones cannot control have provided an excellent source of novelty: from other drones to unexpected natural forces. In such an environment, drones must deal with constantly emerging unexpected events, learn to reject interference and maintain a stable balance. By reducing the entropy of the drone access state, the drone patrol strategy learns to find stable and repeatable situations to counter the ubiquitous entropy sources in the environment. The drone itself can learn to avoid other drones and find a stable and balanced drone patrol trajectory in the face of interfering forces. Therefore, a common surprise minimization reward for drones is given from a global perspective, that is, drones in the environment guide drones to learn stable and predictable behaviors by minimizing the global access state entropy, which minimizes the access conflicts of drone nodes.

[0103] When executing steps 205-206, the agent executes the strategy in the environment and collects the experience tuple e t =(s t , o t ,h t，π ,h t，V ,u t , r t ,s t+1 , o t+1 ), for each time step t, calculate the advantage function The formula is

[0104] In the formula, γ is the discount factor, λ is the GAE parameter, T is the time step, is the output of the critic network.

[0105] Specifically, for a data tuple e t =(s t , o t ,h t，π ,h t,V ,u t , r t ,s t+1 , o t+1 )Change the reward value to: r t =r task (o t )+αr SMiRL (s t ).

[0106] In the formula, r task (o t ) is the external reward, i.e. the reward from the environment itself, r SMiRL (s t) is to minimize the unexpected reward. α>0 indicates the weight of the latter consideration.

[0107] Among them, for r task (o t ) calculation: r task (o t )=r local +r terminal .

[0108] In the formula, the global reward is the average idle time of the global target Minimization: When the drone reaches the maximum training time T When the training maximum time T is not reached terminal = 0; use a local reward, which is at the node v i Reward a single drone: When the drone reaches the next patrol node Among them, ξ(v i ) is the node v i The idle time of a node is the time interval since the node was last visited. is the average idle time of all nodes, m is the number of nodes, and ε is a very small value to avoid r local Divisible by 0.

[0109] r SMiRL (s t ) calculation: Suppose the historical trajectory of the global state is s1,...,s t , Among them, μ t is the mean of the historical states σ t is the variance of the historical state After calculating the global state entropy minimization reward, update μ and σ. The update method is: Therefore, the whole process alternates between two steps: improving the strategy to seek more predictable stimuli, that is, encouraging the agent to seek stable strategies by minimizing surprise rewards; learning a density model to evaluate the degree of surprise of the stimulus, that is, updating μ and σ, so that the drone will get greater rewards when visiting more familiar states.

[0110] Since a large number of experience tuples are generated during the interaction between drones and the environment, the complete trajectory of the drone swarm is obtained through a large number of experience tuples.

[0111] When executing steps 207-209, the complete trajectory τ is split into blocks of length L. Where τ[l:l+L] is a trajectory sequence of length L, containing L experience tuples, The length is L Advantage value sequence, is a discounted reward sequence of length L. These blocks are stored in the experience replay pool, and when the maximum batch_size is reached, the experience replay pool is used to The number of randomly sampled samples b in is used to train the network and update the weights of the actor network and the critic network until the actor network and the critic network converge.

[0112] Specifically, the actor network is updated to maximize the objective function L(θ), as follows:

[0113]

[0114] Among them, B is the batch size, n is the number of agents, is the parameter update ratio, is the observed value of agent i, is the current action taken by agent i, is the current network strategy, To update the previous network strategy, ∈ is a hyperparameter used to control the range of clipping, and clip() is a parameter clipping method. Limited to [1-∈, 1+∈], To obtain the advantage function, is the strategy adopted according to the observed value, H is the entropy of the strategy, λ is the coefficient of control entropy, and the calculation formula is λ=Tanh(Norm(H(s i ))-Norm(H(o i ))). Tanh is a squeezing function that limits the parameter λ to between (-1, 1), and the Norm() normalization function normalizes the entropy H, where H(s i ) calculation process and r SMiRL (s t ) is consistent, H(o i ) is similar to the calculation process of H(s i ), except that the state transitions used are local observations of agent i.

[0115] Specifically, update the critic network and minimize the value function L(φ), as follows:

[0116]

[0117] in For discount rewards, is the state value output by the critic network, It is the state value output by the critic network before updating.

[0118] Use gradient descent to update the parameters of the actor network and the critic network separately.

[0119] Embodiment 2

[0120] like Figure 3 As shown, this embodiment provides a smart forest fire monitoring and early warning device based on a drone swarm, including:

[0121] The information acquisition module 301 is used to obtain the patrol topology map, real-time map information and drone swarm information of the current environment; the drone swarm information includes the starting position of each drone and the current observation status of the drone; the current observation status of the drone includes the patrol nodes within the drone observation range and the status of other drones within the drone observation range; the patrol topology map includes a number of patrol nodes.

[0122] The environment space model building module 302 is used to build a drone environment space model based on the patrol topology map, real-time map information and drone group information.

[0123] The computing module 303 is used to input the current observation state of the drone into the trained multi-agent reinforcement learning model to determine the optimal patrol node of each drone in the drone group at the next moment; the multi-agent reinforcement learning model is a model based on the MAPPO algorithm.

[0124] The state update module 304 is used to control each drone in the drone group to perform a patrol mission according to the optimal patrol node of each drone at the next moment, and to update the position and current observation state of each drone in the drone environment space model.

[0125] The observation module 305 is used to re-execute the steps based on the updated current observation status of each drone: input the current observation status of the drone into the trained multi-agent reinforcement learning model to determine the optimal patrol node of each drone in the drone group at the next moment until all patrol nodes in the patrol topology map are covered.

[0126] Among them, the device also includes a model training module.

[0127] The model training module specifically includes:

[0128] The learning model construction submodule is used to build a multi-agent reinforcement learning model.

[0129] The parameter initialization submodule is used to initialize the actor network parameters, critic network parameters and experience replay pool in the multi-agent reinforcement learning model, and set the learning rate.

[0130] The action calculation submodule is used to calculate the current observation state of each drone as the input of the actor network in the MAPPO algorithm, and obtain the drone action through the actor network.

[0131] The reward calculation submodule is used to interact the drone's actions with the environment based on the accident minimization reinforcement learning method, obtain rewards from environmental feedback, and minimize accidental rewards.

[0132] The action execution submodule is used to execute the UAV action and obtain the experience tuple of the current time step when the UAV moves to the next position.

[0133] The advantage function and complete trajectory determination submodule is used to determine the advantage function and complete trajectory of the drone swarm based on the experience tuple at each time step.

[0134] The trajectory splitting submodule is used to split the complete trajectory into a trajectory sequence of length L and store the trajectory sequence in the experience replay pool.

[0135] The sample extraction submodule is used to extract a set number of samples from the experience replay pool when the storage capacity of the experience replay pool reaches the maximum.

[0136] The model training submodule is used to update the actor network in the multi-agent reinforcement learning model based on the sample. c Riti c The weights of the network, up to the actor network and the critic c The network converges and a trained multi-agent reinforcement learning model is obtained.

[0137] The advantage function and complete trajectory determination submodule specifically include:

[0138] Advantage function determination unit, used to determine the formula Determine the advantage function of the drone swarm; where, γ is the discount factor, λ is the GAE parameter, T is the time step, is the output of the critic network.

[0139] The complete trajectory determination unit is used to obtain a complete trajectory based on a large number of experience tuples generated during the interaction between the UAV and the environment.

[0140] In some other embodiments, a smart forest fire monitoring and early warning device based on a drone swarm is provided, comprising:

[0141] Map module: used to obtain the current environment patrol topology map and real-time map information. The drone can obtain patrol node information within the observable range in real time.

[0142] Patrol module: During training, the UAV uses the local observation information obtained by the map module as input, and iterates through training until the UAV can autonomously select the next globally optimal patrol node. During execution, the node selection operation is performed using the trained patrol algorithm based on the local observation information.

[0143] Control module: The UAV autonomously navigates to the next node planned by the patrol module at a constant flight speed.

[0144] In summary, this application has the following technical effects:

[0145] First, this application draws on the idea of ​​minimizing surprises, allowing drones to experience familiar states as much as possible and maintain a relatively dispersed distribution in space. This method introduces an auxiliary reward for minimizing surprises, hoping to improve the problem of conflicting node access conflicts under poor communication conditions, that is, to disperse drones as much as possible to avoid local conflicts. The principle of the above scheme of this application is as follows: organisms are able to shape an ecological niche that can maintain relative predictability in an environment with increasing entropy around them. In a more realistic world environment, natural forces that drones cannot control have provided an excellent source of novelty: from other drones to unexpected natural forces, drones must deal with constantly emerging unexpected events. In this case, how drones reject interference and maintain a stable balance is more challenging than seeking novelty. Based on this observation, an auxiliary reward is introduced, which aims to reduce the entropy of drone access states. Suppose the historical trajectory of the global state s1,...,s t , Among them, μ t is the mean of the historical states σ t is the variance of the historical state After calculating the global state entropy minimization reward, update μ and σ. The update method is: Therefore, the whole process alternates between two steps: improving the strategy to seek more predictable stimuli, that is, minimizing rewards through surprises, and learning density models to evaluate the degree of surprise of stimuli, that is, updating μ and σ, so that the drone will get greater rewards when visiting more familiar states. Through these two steps, the drone is helped to learn a strategy to change the high entropy of the environment to low entropy, making the states experienced by the drone more familiar and more predictable, further improving the robustness of the drone strategy and reducing the impact under poor communication conditions.

[0146] Second, due to the inconsistency between local information and global information, the agent needs to strike a balance between the local and global perspectives. Therefore, a method for controlling the exploration of a single agent by using the difference between the individual local observation access state entropy and the global access state entropy is proposed. The global state entropy and the individual state entropy measure the degree to which the global and individual are in a steady state, respectively, and the entropy difference between the two measures the degree of exploration of the individual at this time. The approach is to use the policy entropy Add the regularization coefficient λ before the equation = Tanh(Norm(H(s i ))-Norm(H(o i ))) to adjust the degree of individual exploration, individual entropy is greater than the global entropy (λ<0), maximizing L(θ) means becomes smaller, that is, the degree of individual exploration is weakened; the individual entropy is lower than the global entropy (λ>0), and maximizing L(θ) means The larger the value, the stronger the individual exploration. This balances the individual exploration and the global steady state, minimizing the impact of the inconsistency between local and global information.

[0147] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0148] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A smart forest fire monitoring and early warning method based on drone swarms, characterized in that the method include: Obtain patrol topology, real-time map information, and drone swarm information of the current environment; The drone group information includes the starting position of each drone and the current observation status of the drone; the current observation status of the drone includes the patrol nodes within the drone observation range and the status of other drones within the drone observation range; The patrol topology diagram includes a plurality of patrol nodes; Establish a UAV environment spatial model based on patrol topology, real-time map information, and UAV swarm information; The current observation state of the drone is input into a trained multi-agent reinforcement learning model to determine the optimal patrol node of each drone in the drone group at the next moment; the multi-agent reinforcement learning model is a model based on the MAPPO algorithm; According to the optimal patrol node of each drone in the drone swarm at the next moment, each drone is controlled to perform the patrol mission, and the position and current observation status of each drone in the drone environment space model are updated; Based on the updated current observation status of each drone, re-execute the following steps: input the current observation status of the drone into the trained multi-agent reinforcement learning model to determine the optimal patrol node of each drone in the drone swarm at the next moment until all patrol nodes in the patrol topology map are covered.

2. The intelligent forest fire monitoring and early warning method based on drone swarm according to claim 1 is characterized in that: The training process of the multi-agent reinforcement learning model is: Build a multi-agent reinforcement learning model; Initialize the actor network parameters, critic network parameters and experience replay pool in the multi-agent reinforcement learning model, and set the learning rate; For each drone, the current observation state of the drone is used as the input of the actor network in the MAPPO algorithm, and the drone action is calculated through the actor network; Based on the accident minimization reinforcement learning method, the drone's actions interact with the environment to obtain rewards from environmental feedback and minimize accidental rewards; Execute the drone action and get the experience tuple of the current time step when the drone moves to the next position; Determine the advantage function and complete trajectory of the drone swarm based on the experience tuple at each time step; Split the complete trajectory into blocks of length L and store the blocks in the experience replay pool; When the storage capacity of the experience replay pool reaches the maximum, a set number of samples are drawn from the experience replay pool; Based on the samples, the weights of the actor network and the critic network in the multi-agent reinforcement learning model are updated until the actor network and the critic network converge to obtain a trained multi-agent reinforcement learning model.

3. The intelligent forest fire monitoring and early warning method based on drone swarm according to claim 2 is characterized in that: The experience tuple is specifically: e t =(s t ,o t ,h t,π ,h t,V ,u t ,r t ,s t+1 ,o t+1 ); Among them, s t is the global state at time t; o t is the state perceived by the drone at time t; h t,π is the hidden state of the actor network, h t,V is the hidden state of the critic network, u t is the drone action at time t, r t An immediate reward consisting of a reward for environmental feedback and minimizing unexpected rewards; t+1 is the global state at time t+1; o t+1 The state perceived by the drone at time t+1.

4. The intelligent forest fire monitoring and early warning method based on drone swarm according to claim 2 is characterized in that: According to the experience tuple at each time step, the advantage function and complete trajectory of the drone group are determined, including: According to the formula Determine the advantage function of the drone swarm; where, γ is the discount factor, λ is the GAE parameter, T is the time step, is the output of the critic network; Based on a large number of experience tuples generated during the interaction between the drone and the environment, a complete trajectory is obtained.

5. The intelligent forest fire monitoring and early warning method based on drone swarm according to claim 2 is characterized in that: The block of length L is specifically: Among them, τ[l:l+L] is a trajectory sequence of length L, is of length L Advantage value sequence, is a discounted return sequence of length L.

6. The intelligent forest fire monitoring and early warning method based on drone swarm according to claim 2 is characterized in that: The updating of the weights of the actor network and the critic network in the multi-agent reinforcement learning model specifically includes: Update the actor network according to maximizing the objective function L(θ); Update the critic network by minimizing the value function L(φ).

7. The intelligent forest fire monitoring and early warning method based on drone swarm according to claim 6 is characterized in that: The maximization objective function L(θ) is specifically: Among them, B is the batch size, n is the number of agents, is the parameter update ratio, is the observation value of agent i, is the current action taken by agent i, is the current network strategy, To update the previous network strategy, ∈ is a hyperparameter used to control the range of clipping, and clip() is a parameter clipping method. Limited to [1-∈, 1+∈], To obtain the advantage function, is the strategy adopted according to the observed value, H is the entropy of the strategy, and λ is the coefficient of controlling entropy; The minimization value function L(φ) is specifically: in, For discount rewards, is the state value output by the critic network, It is the state value output by the critic network before updating.

8. An intelligent forest fire monitoring and early warning device based on drone swarms, characterized in that: The device includes: Information acquisition module, used to obtain patrol topology map, real-time map information and drone group information of the current environment; The drone group information includes the starting position of each drone and the current observation status of the drone; the current observation status of the drone includes the patrol nodes within the drone observation range and the status of other drones within the drone observation range; the patrol topology map includes a number of patrol nodes; The environment space model building module is used to build the drone environment space model based on the patrol topology map, real-time map information and drone group information; A computing module, used to input the current observation state of the drone into a trained multi-agent reinforcement learning model to determine the optimal patrol node of each drone in the drone group at the next moment; the multi-agent reinforcement learning model is a model based on the MAPPO algorithm; The state update module is used to control each drone in the drone group to perform patrol tasks according to the optimal patrol node of each drone at the next moment, and to update the position and current observation state of each drone in the drone environment space model; The observation module is used to re-execute the steps based on the updated current observation status of each drone: input the current observation status of the drone into the trained multi-agent reinforcement learning model, determine the optimal patrol node of each drone in the drone group at the next moment, until all patrol nodes in the patrol topology map are covered.

9. The intelligent forest fire monitoring and early warning device based on drone swarm according to claim 8 is characterized in that: Also includes,model training module; The model training module specifically includes: The learning model construction submodule is used to build a multi-agent reinforcement learning model; A parameter initialization submodule, used to initialize the actor network parameters, critic network parameters and experience replay pool in the multi-agent reinforcement learning model, and set the learning rate; The action calculation submodule is used for each UAV to use the current observation state of the UAV as the input of the actor network in the MAPPO algorithm and calculate the UAV action through the actor network; The reward calculation submodule is used to interact the drone's actions with the environment based on the accident minimization reinforcement learning method, obtain rewards from environmental feedback, and minimize accidental rewards; The action execution submodule is used to execute the UAV action and obtain the experience tuple of the current time step when the UAV moves to the next position; The advantage function and complete trajectory determination submodule is used to determine the advantage function and complete trajectory of the drone swarm based on the experience tuple at each time step; The trajectory splitting submodule is used to split the complete trajectory into a trajectory sequence of length L and store the trajectory sequence in the experience replay pool; The sample extraction submodule is used to extract a set number of samples from the experience replay pool when the storage capacity of the experience replay pool reaches the maximum; The model training submodule is used to update the weights of the actor network and the critic network in the multi-agent reinforcement learning model based on the samples until the actor network and the critic network converge to obtain a trained multi-agent reinforcement learning model.

10. The intelligent forest fire monitoring and early warning device based on drone swarm according to claim 9 is characterized in that: The advantage function and complete trajectory determination submodule specifically include: Advantage function determination unit, used to determine the formula Determine the advantage function of the drone swarm; where, γ is the discount factor, λ is the GAE parameter, T is the time step, is the output of the critic network; The complete trajectory determination unit is used to obtain a complete trajectory based on a large number of experience tuples generated during the interaction between the UAV and the environment.

Images