A wisdom forest fire prevention monitoring and early warning method and device based on a UAV group

Abstract

The application discloses a kind of wisdom forest fire prevention monitoring early warning method and device based on unmanned aerial vehicle group, it is related to forest fire prevention monitoring technical field, the method includes obtaining the patrol topological graph of current environment, real-time map information, the starting position of each unmanned aerial vehicle and the current observation state of unmanned aerial vehicle;Based on the information obtained, establish unmanned aerial vehicle environment space model;The current observation state of unmanned aerial vehicle is input into the multi-agent reinforcement learning model trained, and the optimal patrol node of each unmanned aerial vehicle in next time is determined;Control each unmanned aerial vehicle to execute patrol task, and the position and current observation state of each unmanned aerial vehicle in unmanned aerial vehicle environment space model are updated;Based on the current observation state of each unmanned aerial vehicle after updating, the optimal patrol node of next time is determined again, until all patrol nodes in patrol topological graph are covered.The application solves the problem that unmanned aerial vehicle patrol is inefficient, and the patrol algorithm is not robust, and improves the patrol efficiency.

Description

Technical Field

[0001] This application relates to the field of forest fire monitoring technology, and in particular to a smart forest fire monitoring and early warning method and device based on unmanned aerial vehicle (UAV) swarms. Background Technology

[0002] Patrol, or continuous environmental monitoring, is an important activity in fields such as environmental safety, forest fire monitoring and prevention. Using multiple drones for patrols simultaneously can improve patrol performance and facilitate the timely detection of anomalies. Therefore, previous research has analyzed the multi-robot patrol (MRP) problem from various perspectives, including problem modeling, optimal solutions, theoretical analysis, implementation, and experimental verification.

[0003] The Multi-Robot Patrol Problem (MRPP) has been extensively studied theoretically and proven to be NP-hard. The literature generally holds that when using environment-based topological representations and global / centralized information, optimal patrol paths and results can be achieved if all robots follow the same TSP or Hamiltonian cycle and are uniformly distributed in time and space. However, in sparse topologies (the case in most real-world environments), these cycles are not easily computed and, in most cases, do not exist.

[0004] Therefore, research on the patrol problem needs to focus on fully autonomous agents. Agents independently determine their patrol routes based on the system's state, without the need for a central planner. This allows for effective autonomous coordination and makes the method robust to robot failures. Compared to centralized solutions, decentralized methods eliminate single points of failure and offer more flexible application implementations. However, other decentralized methods, such as the task-assignment-based DTAG and DTAP algorithms, and the Bayesian CBLS algorithm, require frequent conflict resolution to adapt to dynamically changing environments. This results in high computational cost, long processing times, and the calculated results may not be optimal (requiring further exploration), making the strategies less robust.

[0005] Existing patrol methods based on multi-agent reinforcement learning algorithms do not require prior environmental knowledge, can effectively achieve cooperation, and can better explore and learn some superior actions, generating more robust policies. However, reinforcement learning-based patrol algorithms still have shortcomings. On the one hand, conflicts arise in individual explorations due to partial observations and non-stationarity, leading to unstable policies and poor results. On the other hand, due to the inconsistency between local and global information, agents need to strike a balance between local and global perspectives; otherwise, insufficient or redundant exploration may occur. Summary of the Invention

[0006] The purpose of this application is to provide a smart forest fire monitoring and early warning method and device based on drone swarms, which can solve the problems of inefficient drone patrols and unrobust algorithms, and improve patrol efficiency.

[0007] To achieve the above objectives, this application provides the following solution:

[0008] Firstly, this application provides a smart forest fire monitoring and early warning method based on unmanned aerial vehicle (UAV) swarms, including:

[0009] The system acquires the current patrol topology map, real-time map information, and drone swarm information. The drone swarm information includes the starting position of each drone and its current observation status. The current observation status of the drones includes the patrol nodes within the drones' observation range and the status of other drones within the drones' observation range. The patrol topology map includes several patrol nodes.

[0010] A drone environmental spatial model is established based on patrol topology, real-time map information, and drone swarm information.

[0011] The current observation state of the UAV is input into the trained multi-agent reinforcement learning model to determine the optimal patrol node for each UAV in the UAV swarm at the next moment; the multi-agent reinforcement learning model is a model based on the MAPPO algorithm.

[0012] Based on the optimal patrol node of each drone in the drone swarm at the next moment, control each drone to perform patrol tasks, and update the position and current observation status of each drone in the drone environmental space model.

[0013] Based on the updated current observation status of each UAV, the following steps are repeated: input the current observation status of the UAV into the trained multi-agent reinforcement learning model to determine the optimal patrol node for each UAV in the UAV swarm at the next moment, until all patrol nodes in the patrol topology are covered.

[0014] Optionally, the training process of the multi-agent reinforcement learning model is as follows:

[0015] Construct a multi-agent reinforcement learning model.

[0016] The actor network parameters, critic network parameters, and experience replay pool in the multi-agent reinforcement learning model are initialized, and the learning rate is set.

[0017] For each UAV, the current observation state of the UAV is used as the input to the actor network in the MAPPO algorithm, and the UAV action is calculated through the actor network.

[0018] Based on the surprise minimization reinforcement learning method, the drone's actions interact with the environment to obtain rewards from environmental feedback and minimize surprise rewards.

[0019] Execute drone actions, and when the drone moves to the next location, obtain the experience tuple for the current time step.

[0020] Based on the empirical tuples at each time step, the dominance function and complete trajectory of the drone swarm are determined.

[0021] The complete trajectory is divided into blocks of length L, and the blocks are stored in the experience replay pool.

[0022] When the storage capacity of the experience replay pool reaches its maximum, a set number of samples are extracted from the experience replay pool.

[0023] Based on the samples, update the weights of the actor network and critic network in the multi-agent reinforcement learning model.

[0024] The process continues until the actor network and the critic network converge, resulting in a well-trained multi-agent reinforcement learning model.

[0025] Optionally, the empirical tuple specifically refers to:

[0026] e t =(s t o t h t，π h t，V u t r t s t+1 o t+1 ).

[0027] Among them, s t Let o be the global state at time t; t h represents the state perceived by the drone at time t. t，π For the hidden state of the actor network, h t，V For the hidden state of the critic network, u t Let r be the drone's action at time t. t The immediate reward consists of environmental feedback rewards and minimized unexpected rewards; t+1 The global state at time t+1; o t+1 The state perceived by the drone at time t+1.

[0028] Optionally, based on the empirical tuples at each time step, the dominance function and complete trajectory of the UAV swarm are determined, specifically including:

[0029] According to the formula Determine the dominance function of the drone swarm; among which, γ is the discount factor, λ is the GAE parameter, and T is the time step. For criti c Network output.

[0030] A complete trajectory is obtained by generating a large number of empirical tuples during the interaction between the drone and the environment.

[0031] Optionally, the block of length L specifically refers to:

[0032]

[0033] Where τ[l:l+L] is a trajectory sequence of length L, It is of length L Dominance value sequence, It is a discount reward sequence of length L.

[0034] Optionally, the updated multi-agent reinforcement learning model includes an actor network, c riti c The network weights specifically include:

[0035] Update the actor network based on maximizing the objective function L(θ).

[0036] Update the critic network based on minimizing the value function L(φ).

[0037] Optionally, maximizing the objective function L(θ) specifically means:

[0038]

[0039] Where B is the batch size and n is the number of agents. Update the parameter ratio. Let i be the observation value of agent i. The action currently taken by agent i. For the current network strategy, To update the previous network strategy, ∈ is a hyperparameter used to control the range of clipping, and clip() is the parameter clipping method. Limited to [1-∈, 1+∈], To obtain the dominance function, Let H be the strategy adopted based on the observed values, H be the entropy of the strategy, and λ be the coefficient of the control entropy.

[0040] The minimized value function L(φ) is specifically as follows:

[0041]

[0042] in, As a discount reward, The state value output by the critic network. For the previous criti c The state value output by the network.

[0043] Secondly, this application provides a smart forest fire monitoring and early warning device based on a drone swarm, comprising:

[0044] The information acquisition module is used to acquire the patrol topology map, real-time map information and drone swarm information of the current environment; the drone swarm information includes the starting position of each drone and the current observation status of the drone; the current observation status of the drone includes the patrol nodes within the drone's observation range and the status of other drones within the drone's observation range; the patrol topology map includes several patrol nodes.

[0045] The environmental space model building module is used to build an environmental space model of UAVs based on patrol topology map, real-time map information and UAV swarm information.

[0046] The calculation module is used to input the current observation state of the UAV into the trained multi-agent reinforcement learning model to determine the optimal patrol node for each UAV in the UAV swarm at the next moment; the multi-agent reinforcement learning model is a model based on the MAPPO algorithm.

[0047] The state update module is used to control each UAV to perform patrol tasks based on the optimal patrol node of each UAV in the next moment, and to update the position and current observation status of each UAV in the UAV environmental space model.

[0048] The observation module is used to re-execute the following steps based on the updated current observation status of each UAV: ​​inputting the current observation status of the UAV into the trained multi-agent reinforcement learning model to determine the optimal patrol node for each UAV in the UAV swarm at the next moment, until all patrol nodes in the patrol topology map are covered.

[0049] Optionally, it also includes a model training module.

[0050] The model training module specifically includes:

[0051] The learning model building submodule is used to build multi-agent reinforcement learning models.

[0052] The parameter initialization submodule is used to initialize the actor network parameters, critic network parameters, and experience replay pool in the multi-agent reinforcement learning model, and to set the learning rate.

[0053] The action calculation submodule is used to calculate the drone's actions by taking the drone's current observation state as input to the actor network in the MAPPO algorithm for each drone.

[0054] The reward calculation submodule is used to interact the drone's actions with the environment based on the surprise minimization reinforcement learning method, so as to obtain the reward from the environment feedback and minimize the surprise reward.

[0055] The action execution submodule is used to execute drone actions. When the drone moves to the next location, it obtains the experience tuple for the current time step.

[0056] The dominance function and complete trajectory determination submodule is used to determine the dominance function and complete trajectory of the UAV swarm based on the empirical tuples at each time step.

[0057] The trajectory splitting submodule is used to split the complete trajectory into trajectory sequences of length L and store the trajectory sequences in the experience replay pool.

[0058] The sample extraction submodule is used to extract a set number of samples from the experience replay pool when the storage capacity of the experience replay pool reaches its maximum.

[0059] The model training submodule is used to update the weights of the actor network and the critic network in the multi-agent reinforcement learning model based on the samples, until the actor network and the critic network converge, thus obtaining the trained multi-agent reinforcement learning model.

[0060] Optionally, the advantage function and the complete trajectory determination submodule specifically include:

[0061] The dominant function determination unit is used to determine the dominant function based on the formula. Determine the dominance function of the drone swarm; among which, γ is the discount factor, λ is the GAE parameter, and T is the time step. This is the output of the critic network.

[0062] The complete trajectory determination unit is used to obtain the complete trajectory based on the large number of experience tuples generated during the interaction between the UAV and the environment.

[0063] According to the specific embodiments provided in this application, the following technical effects are disclosed:

[0064] This application provides a method and device for intelligent forest fire monitoring and early warning based on unmanned aerial vehicle (UAV) swarms. The method includes: First, acquiring the patrol topology map, real-time map information, and UAV swarm information of the current environment. This helps to understand the areas that the UAVs need to cover and the current environmental state, providing basic data for subsequent decision-making. Second, establishing a UAV environmental space model. By combining the patrol topology map, real-time map information, and UAV swarm information, an accurate environmental model can be constructed, providing a basis for UAV path planning and decision-making. Third, inputting the current observation state of the UAVs into a trained multi-agent reinforcement learning model to determine the optimal patrol node for each UAV in the swarm at the next moment is the third step. This step uses advanced algorithms to dynamically adjust the patrol paths of the UAVs, ensuring efficient execution of patrol tasks. Fourth, controlling each UAV to execute patrol tasks based on the optimal patrol node for each UAV in the swarm at the next moment, and updating the position and current observation state of each UAV in the UAV environmental space model is the fourth step. This ensures that the UAVs can adjust according to the latest environmental information and task requirements, improving the flexibility and efficiency of patrols. Finally, based on the updated observation states of each UAV, the process re-inputs the current observation states of the UAVs into the trained multi-agent reinforcement learning model to determine the optimal patrol node for each UAV in the swarm at the next time step, until all patrol nodes in the patrol topology are covered. This iterative process ensures that the UAVs can continuously and efficiently cover all patrol nodes, solving the problems of low efficiency and non-robustness in traditional methods. Attached Figure Description

[0065] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0066] Figure 1 This is a flowchart illustrating a smart forest fire monitoring and early warning method based on a drone swarm, as provided in one embodiment of this application.

[0067] Figure 2 This is a schematic diagram of a network structure provided in an embodiment of this application.

[0068] Figure 3 This is a schematic diagram of the functional modules of a smart forest fire monitoring and early warning device based on a drone swarm, provided as an embodiment of this application. Detailed Implementation

[0069] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0070] Forests cover a large area, and fire is one of the biggest factors causing forest damage. Current forest fire prevention methods generally rely on building lookout towers for manual inspection and establishing video surveillance systems. The biggest problem with manual lookout towers is that they require personnel to be constantly attentive, which can easily lead to missed reports or delayed reporting. Video surveillance systems, which collect and aggregate video data and then rely on manual monitoring, lack intelligence and are costly in terms of manpower. On the other hand, drone patrols often use fixed routes, have relatively simple patrol algorithms, and their strategies are not robust enough.

[0071] The purpose of this application is to provide a smart forest fire monitoring and early warning method and device based on drone swarms, which can solve the problems of inefficient drone patrols and unrobust algorithms, and improve patrol efficiency.

[0072] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0073] Example 1

[0074] like Figure 1 As shown, this embodiment provides a smart forest fire monitoring and early warning method based on unmanned aerial vehicle (UAV) swarms, including:

[0075] Step 101: Obtain the patrol topology map, real-time map information, and drone swarm information of the current environment; the drone swarm information includes the starting position of each drone and the current observation status of the drone; the current observation status of the drone includes the patrol nodes within the drone's observation range and the status of other drones within the drone's observation range; the patrol topology map includes several patrol nodes.

[0076] Step 102: Based on the patrol topology map, real-time map information and drone swarm information, establish a drone environmental space model.

[0077] Step 103: Input the current observation state of the UAV into the trained multi-agent reinforcement learning model to determine the optimal patrol node for each UAV in the UAV swarm at the next moment; the multi-agent reinforcement learning model is a model based on the MAPPO algorithm.

[0078] Step 104: Based on the optimal patrol node of each UAV in the UAV swarm at the next moment, control each UAV to perform patrol tasks, and update the position and current observation status of each UAV in the UAV environmental space model.

[0079] Step 105: Based on the updated current observation status of each UAV, re-execute the following steps: input the current observation status of the UAV into the trained multi-agent reinforcement learning model to determine the optimal patrol node for each UAV in the UAV swarm at the next moment, until all patrol nodes in the patrol topology are covered.

[0080] In some embodiments, when performing steps 101-102, the specific steps may be as follows:

[0081] The system acquires the current environmental patrol topology and real-time map information, enabling UAVs to obtain patrol node information within their observable range in real time. The UAV's state includes information about patrol nodes and other UAVs within its observation range. Nodes are defined by the pre-generated topology from the previous step in the environmental space; at this point, each UAV knows the graph's structure in advance, including the number and location of nodes and the connections between them.

[0082] Based on the patrol topology map, real-time map information, and UAV swarm information, a UAV environmental space model is established. The simulation scene (environment space) is a two-dimensional space composed of UAVs and the patrol topology map. The starting position of the UAVs and the positions of the nodes in the topology map are randomly set in a two-dimensional Cartesian coordinate system.

[0083] Specifically, when executing step 103, the following can be done:

[0084] Obtain the current observation status of the UAV.

[0085] Specifically, the state o perceived by the drone at the current moment t. t =(o i o j ), where o i The status of each patrol node as observed by the UAV within its observation range, o j This shows the state of other UAVs within the observation range. Simultaneously, the global state s at time t is generated. t At the start of each episode, the drone is placed at a random location on the environmental topology map.

[0086] The current observation status of the drone. t As input to the MAPPO algorithm actor network in the trained multi-agent reinforcement learning model, the drone action u is calculated through the actor network. t According to the drone's actionst As the optimal control action that the drone needs to perform in the current state, determine the optimal patrol node for the drone in the next moment.

[0087] In this embodiment, such as Figure 2 As shown, the multi-agent reinforcement learning algorithm used is the Multi-agent Proximal Policy Optimization algorithm. This algorithm's network model includes multiple actor networks and multiple critic networks. Each actor network shares parameters and maps from a state space to an action space, taking the observed state as input and outputting an action. Each critic evaluates the decisions generated by the multiple actor networks, taking the global state as input and outputting a value generalized advantage estimate of the joint action taken in that state. This evaluation then updates both the actor and critic networks.

[0088] Specifically, the training process of the multi-agent reinforcement learning model is as follows:

[0089] Step 201: Construct a multi-agent reinforcement learning model.

[0090] Step 202: Initialize the actor network parameters, critic network parameters, and experience replay pool in the multi-agent reinforcement learning model, and set the learning rate.

[0091] Step 203: For each UAV, use the current observation state of the UAV as the input to the actor network in the MAPPO algorithm, and calculate the UAV action through the actor network.

[0092] Step 204: Based on the surprise minimization reinforcement learning method, the drone's actions interact with the environment to obtain rewards from environmental feedback and minimize surprise rewards.

[0093] Step 205: Execute the drone action. When the drone moves to the next location, obtain the experience tuple for the current time step.

[0094] Step 206: Determine the dominance function and complete trajectory of the UAV swarm based on the empirical tuples at each time step.

[0095] Step 207: Divide the complete trajectory into blocks of length L and store the blocks in the experience replay pool.

[0096] Step 208: When the storage capacity of the experience replay pool reaches its maximum, a set number of samples are extracted from the experience replay pool.

[0097] Step 209: Based on the samples, update the weights of the actor network and the critic network in the multi-agent reinforcement learning model until the actor network and the critic network converge, thus obtaining the trained multi-agent reinforcement learning model.

[0098] In steps 201-202, each drone a is treated as an agent. At the beginning, the actor network parameters θ and the critic network parameters φ are initialized; the learning rate α is set, and the experience replay pool is initialized.

[0099] Specifically, during step 203, the current observation status of the UAV is set. t As input to the actor network of the MAPPO algorithm, the action output u is calculated through the actor network. t This refers to the control actions that the drone needs to perform in its current state. The critic network receives the global state s. t Output state value v t .

[0100] In step 204, the UAV executes the action u given by the actor network in the MAPPO algorithm. t It interacts with the environment and receives rewards in return. and minimize unexpected rewards These two together constitute the instant reward r t The drone then moves to the next location to obtain the next observation status. t+1 The global state is transformed into s t+1 At this point, we obtain the empirical tuple e at time t. t =(s t o t h t，π h t，V u t r t s t+1 o t+1 ), that is, the sample, and the tuple is stored in the trajectory τ.

[0101] Specifically, in the patrol problem, the patrol topology is often sparse, with the number of nodes typically exceeding the number of drones. Therefore, patrolling in dynamic environments requires minimizing the impact of drone-visited node conflicts. This necessitates collaborative exploration by drone swarms to further enhance the robustness of inter-drone cooperation and accelerate strategy discovery. Based on this, this embodiment proposes a technique based on surprise minimization reinforcement learning, assigning surprise minimization rewards to environmental conflicts to achieve the aforementioned objectives.

[0102] The surprise minimization reward in this embodiment is understood as follows: In a real-world open-world environment, uncontrollable natural forces provide an excellent source of novelty: from other drones to unexpected natural forces. In such an environment, drones must cope with constantly emerging unexpected events, learn to reject interference, and maintain a stable equilibrium. By reducing the drone's access state entropy, the drone patrol strategy learns to find stable and repeatable situations to counteract the pervasive entropy sources in the environment. The drone itself can learn to avoid other drones and find a stable and balanced drone patrol trajectory in the face of interfering forces. Therefore, from a global perspective, the common surprise minimization reward for drones is given, that is, drones in the environment are guided to learn stable and predictable behavior by minimizing global access state entropy, which minimizes drone node access conflicts as much as possible.

[0103] During steps 205-206, the agent executes policies in the environment and collects experience tuples e. t =(s t o t h t，π h t，V u t r t s t+1 o t+1 For each time step t, calculate the dominance function. The formula is

[0104] In the formula, γ is the discount factor, λ is the GAE parameter, and T is the time step. This is the output of the critic network.

[0105] Specifically, for a data tuple e t =(s t o t h t，π h t,V u t r t s t+1 o t+1 The reward value is modified to: r t =r task (o t )+αr SMiRL (s t ).

[0106] In the formula, r task (o t ) refers to external rewards, i.e., rewards from the environment itself, r SMiRL (s tTo minimize unexpected rewards. α>0 indicates the weight considered in the latter case.

[0107] Among them, for r task (o t Calculation of r: task (o t ) = r local +r terminal .

[0108] In the formula, the global reward is the average idle time of the global objective. Minimize: When the drone reaches the maximum training time T The maximum training time T was not reached. terminal =0; Use a local reward, which is given when reaching node v. i A reward is given to a single drone when it reaches the next patrol node. Where, ξ(v i ) is node v i Idle time, which is the time interval since the node was last accessed. It is the average idle time of all nodes, where m is the number of nodes, and ε is a very small value that can avoid r local Divisible by 0.

[0109] r SMiRL (s t Calculation of ) : Suppose there are historical trajectories of global state s1, ..., s2 t , Where, μ t The mean of historical states σ t The variance of historical states After calculating the reward that minimizes the global state entropy, update μ and σ. The update method is as follows: Therefore, the entire process alternates between two steps: improving the strategy to seek more predictable stimuli, i.e., encouraging the agent to seek stable strategies by minimizing surprise rewards; and learning a density model to assess the degree of surprise of the stimulus, i.e., updating μ and σ so that the drone will receive greater rewards for visiting more familiar states.

[0110] Because a large number of empirical tuples are generated during the interaction between the drones and the environment, the complete trajectory of the drone swarm can be obtained through a large number of empirical tuples.

[0111] Specifically, during steps 207-209, the complete trajectory τ is divided into blocks of length L. Where τ[l:l+L] is a trajectory sequence of length L, containing L empirical tuples. It is of length L Dominance value sequence, It is a discount reward sequence of length L. These blocks are stored in the experience replay pool, and when the maximum batch_size is reached, they are removed from the experience replay pool. A random sample b of size B is used to train the network, and the weights of the actor network and the critic network are updated until the actor network and the critic network converge.

[0112] Specifically, the actor network is updated by maximizing the objective function L(θ), as follows:

[0113]

[0114] Where B is the batch size and n is the number of agents. Update the parameter ratio. Let i be the observation value of agent i. The action currently taken by agent i. For the current network strategy, To update the previous network strategy, ∈ is a hyperparameter used to control the range of clipping, and clip() is the parameter clipping method. Limited to [1-∈, 1+∈], To obtain the dominance function, The strategy adopted based on the observed values, where H is the entropy of the strategy and λ is the coefficient of the control entropy, is calculated using the formula λ = Tanh(Norm(H(s)). i ))-Norm(H(o i ))). Here, Tanh is the squeezing function, restricting the parameter λ to the range (-1, 1), and the Norm() normalization function normalizes the entropy H, where H(s i The calculation process of r) SMiRL (s t H(o) is consistent with H(o) i The calculation process of H(s) and H(s) i Similar to that, except that the state used is transformed into the local observation of agent i.

[0115] Specifically, the critic network is updated by minimizing the value function L(φ), as follows:

[0116]

[0117] in As a discount reward, The state value output by the critic network. This represents the state value of the critic network output before the update.

[0118] Use gradient descent to update the parameters of the actor network and the critic network respectively.

[0119] Example 2

[0120] like Figure 3 As shown, this embodiment provides a smart forest fire monitoring and early warning device based on a drone swarm, including:

[0121] The information acquisition module 301 is used to acquire the patrol topology map, real-time map information and UAV swarm information of the current environment; the UAV swarm information includes the starting position of each UAV and the current observation status of the UAV; the current observation status of the UAV includes the patrol nodes within the observation range of the UAV and the status of other UAVs within the observation range of the UAV; the patrol topology map includes several patrol nodes.

[0122] The environmental space model building module 302 is used to build an environmental space model of UAVs based on patrol topology map, real-time map information and UAV swarm information.

[0123] The calculation module 303 is used to input the current observation state of the UAV into the trained multi-agent reinforcement learning model to determine the optimal patrol node of each UAV in the UAV swarm at the next moment; the multi-agent reinforcement learning model is a model based on the MAPPO algorithm.

[0124] The state update module 304 is used to control each UAV to perform patrol tasks based on the optimal patrol node of each UAV in the UAV swarm at the next moment, and to update the position and current observation status of each UAV in the UAV environmental space model.

[0125] The observation module 305 is used to re-execute the following steps based on the updated current observation status of each UAV: ​​inputting the current observation status of the UAV into the trained multi-agent reinforcement learning model, determining the optimal patrol node for each UAV in the UAV swarm at the next moment, until all patrol nodes in the patrol topology are covered.

[0126] The device also includes a model training module.

[0127] The model training module specifically includes:

[0128] The learning model building submodule is used to build multi-agent reinforcement learning models.

[0129] The parameter initialization submodule is used to initialize the actor network parameters, critic network parameters, and experience replay pool in the multi-agent reinforcement learning model, and to set the learning rate.

[0130] The action calculation submodule is used to calculate the drone's actions by taking the drone's current observation state as input to the actor network in the MAPPO algorithm for each drone.

[0131] The reward calculation submodule is used to interact the drone's actions with the environment based on the surprise minimization reinforcement learning method, so as to obtain the reward from the environment feedback and minimize the surprise reward.

[0132] The action execution submodule is used to execute drone actions. When the drone moves to the next location, it obtains the experience tuple for the current time step.

[0133] The dominance function and complete trajectory determination submodule is used to determine the dominance function and complete trajectory of the UAV swarm based on the empirical tuples at each time step.

[0134] The trajectory splitting submodule is used to split the complete trajectory into trajectory sequences of length L and store the trajectory sequences in the experience replay pool.

[0135] The sample extraction submodule is used to extract a set number of samples from the experience replay pool when the storage capacity of the experience replay pool reaches its maximum.

[0136] The model training submodule is used to update the actor network in the multi-agent reinforcement learning model based on the samples. c riti c The weights of the network, up to the actor network and the critique c The network converges, resulting in a well-trained multi-agent reinforcement learning model.

[0137] Specifically, the advantage function and the complete trajectory determination submodule include:

[0138] The dominant function determination unit is used to determine the dominant function based on the formula. Determine the dominance function of the drone swarm; among which, γ is the discount factor, λ is the GAE parameter, and T is the time step. This is the output of the critic network.

[0139] The complete trajectory determination unit is used to obtain the complete trajectory based on the large number of experience tuples generated during the interaction between the UAV and the environment.

[0140] In some other embodiments, a smart forest fire monitoring and early warning device based on a drone swarm is provided, comprising:

[0141] Map module: Used to obtain the current environment patrol topology map and real-time map information. The drone can obtain patrol node information within the observable range in real time.

[0142] Patrol Module: During training, the UAV uses local observation information obtained from the map module as input and iteratively trains until it can autonomously select the next globally optimal patrol node. During execution, the UAV performs node selection operations based on the local observation information and the trained patrol algorithm.

[0143] Control module: The UAV autonomously navigates to the next node planned by the patrol module at a constant flight speed.

[0144] In summary, this application has the following technical effects:

[0145] First, this application draws on the idea of ​​minimizing surprise, allowing drones to experience familiar states as much as possible and maintaining a relatively dispersed distribution in space. This method introduces an auxiliary reward that minimizes unexpected events, aiming to improve the problem of conflict node access conflicts under poor communication conditions, i.e., to disperse drones as much as possible and avoid local conflicts. The principle behind the above scheme in this application is as follows: Organisms can shape niches that maintain relative predictability in an environment with increasing entropy. In a more realistic world environment, natural forces beyond the control of drones have provided an excellent source of novelty: from other drones to unexpected natural forces, drones must cope with constantly emerging unexpected events. In this situation, how drones reject interference and maintain a stable balance is more challenging than seeking novelty. Based on this observation, an auxiliary reward is introduced, which aims to reduce the entropy of the drone's access state. Let there be a historical trajectory of the global state s1, ..., s2. t , Where, μ t The mean of historical states σ t The variance of historical states After calculating the reward that minimizes the global state entropy, update μ and σ. The update method is as follows: Therefore, the entire process alternates between two steps: refining the strategy to seek more predictable stimuli (i.e., minimizing rewards through surprise) and learning a density model to assess the unexpectedness of the stimulus (i.e., updating μ and σ). This allows the drone to receive greater rewards when accessing more familiar states. Through these two steps, the drone learns a strategy to change the environment from high entropy to low entropy, making the states it experiences more familiar and predictable. This further improves the robustness of the drone's strategy and reduces the impact of poor communication conditions.

[0146] Second, due to the inconsistency between local and global information, the agent needs to strike a balance between local and global perspectives. Therefore, a method is proposed to control the exploration of a single agent by utilizing the difference between the individual's local observation access state entropy and global access state entropy. Global state entropy and individual state entropy respectively measure the degree to which the global and individual entities are in a steady state, and the difference between the two entropies measures the individual's exploration progress at that point. The approach involves... Add a regularization coefficient λ = Tanh(Norm(H(s)) i ))-Norm(H(o i To regulate the degree of individual exploration, individual entropy is greater than global entropy (λ < 0). Maximizing L(θ) means... Decreasing L(θ) means reducing the degree of individual exploration; individual entropy is lower than global entropy (λ > 0), maximizing L(θ) means Increasing the size of the area enhances the level of individual exploration. This balances individual exploration with global steady state, minimizing the impact of inconsistencies between local and global information.

[0147] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0148] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A smart forest fire monitoring and early warning method based on unmanned aerial vehicle (UAV) swarms, characterized in that the method... The method comprises the following steps: acquiring a patrol topology graph, real-time map information and unmanned aerial vehicle (UAV) group information of a current environment; the UAV group information comprises starting positions of each UAV and a current observation state of the UAVs; the current observation state of the UAVs comprises patrol nodes within an observation range of the UAVs and states of other UAVs within the observation range of the UAVs; the patrol topology graph comprises a plurality of patrol nodes; an UAV environment space model is established based on the patrol topology graph, the real-time map information and the UAV group information; the current observation state of the UAVs is input into a trained multi-agent reinforcement learning model to determine optimal patrol nodes of each UAV in the UAV group at a next time step; the multi-agent reinforcement learning model is a model based on a MAPPO algorithm; an action of the UAV is executed, and an experience tuple of a current time step is obtained when the UAV moves to a next position; an advantage function of the UAV group and a complete trajectory are determined according to the experience tuple of each time step; the complete trajectory is split into blocks with a length of L, and the blocks are stored in an experience replay pool; when a storage amount of the experience replay pool reaches a maximum, a preset number of samples are extracted from the experience replay pool; weights of an actor network and a critic network in the multi-agent reinforcement learning model are updated based on the samples until the actor network and the critic network converge, so that the trained multi-agent reinforcement learning model is obtained; the UAVs are controlled to perform a patrol task according to the optimal patrol nodes of each UAV in the UAV group at the next time step, and positions and current observation states of the UAVs in the UAV environment space model are updated; based on the updated current observation states of the UAVs, the step of inputting the current observation states of the UAVs into the trained multi-agent reinforcement learning model to determine the optimal patrol nodes of each UAV in the UAV group at the next time step is performed again until all the patrol nodes in the patrol topology graph are covered. 2.The wisdom forest fire prevention monitoring and early warning method based on the UAV group according to claim 1, characterized in that, The training process of the multi-agent reinforcement learning model comprises the following steps: a multi-agent reinforcement learning model is constructed; actor network parameters, critic network parameters and an experience replay pool in the multi-agent reinforcement learning model are initialized, and a learning rate is set; for each UAV, a current observation state of the UAV is taken as an input of an actor network in a MAPPO algorithm, and an action of the UAV is calculated through the actor network; the UAV action is interacted with an environment based on an unexpected minimization reinforcement learning method to obtain a reward of environmental feedback and a minimized unexpected reward. 3.The wisdom forest fire prevention monitoring and early warning method based on the UAV group according to claim 1, characterized in that, The experience tuple is specifically: e t = (s t ,o t ,h t,π ,h t,V ,u t ,r t ,s t+1 ,o t+1 ); where s t is the global state at time t; o t is the state perceived by the UAV at time t; h t,π is the hidden state of the actor network, h t,v is the hidden state of the critic network, u t is the action of the UAV at time t, r t is the immediate reward consisting of the environment feedback reward and the minimized unexpected reward; s t+1 is the global state at time t+1; o t+1 is the state perceived by the UAV at time t+1. 4.The unmanned aerial vehicle swarm-based intelligent forest fire monitoring and early warning method according to claim 1, characterized in that, the advantage function of the UAV group and the complete trajectory are determined according to the experience tuple of each time step, and the complete trajectory is specifically obtained according to a large number of experience tuples generated in an interaction process between the UAV and the environment. According to the formula determining an advantage function of the UAV group; wherein, γ is a discount factor, λ is a GAE parameter, T is a time step, is the output of the critic network; The block with the length of L is specifically: 5.The unmanned aerial vehicle swarm-based intelligent forest fire monitoring and early warning method according to claim 1, characterized in that, the weights of the actor network and the critic network in the multi-agent reinforcement learning model are updated, and the updating specifically comprises the following steps: where τ[l:l+L] is a sequence of length L of trajectories, is a sequence of length L of advantage values, is a sequence of length L of discounted returns. 6.The unmanned aerial vehicle swarm-based intelligent forest fire monitoring and early warning method according to claim 1, characterized in that, the actor network is updated according to a maximization target function L(θ); the critic network is updated according to a minimization value function L(φ). The maximization target function L(θ) is specifically: 7.The unmanned aerial vehicle swarm-based intelligent forest fire monitoring and early warning method according to claim 6, characterized in that, ​ where B is batch size, n is the number of agents, is the parameter update ratio, is the observation of agent i, is the action taken by agent i at present, is the policy of the current network, is the policy of the network before update, ∈ is a hyperparameter for controlling the range of clipping, clip() is a parameter clipping method, and is limited to [1-∈, 1+∈], is the advantage function obtained, is the policy taken according to the observation, H is the entropy of the policy, and λ is a coefficient for controlling the entropy. The minimization value function L(φ) is specifically: wherein, is the discount reward, is the state value output by the critic network, is the state value output by the critic network before the update.

8. A wisdom forest fire prevention monitoring and early warning device based on a UAV group, characterized in that, The device comprises: An information acquisition module is configured to acquire a patrol topology graph of a current environment, real-time map information, and unmanned aerial vehicle (UAV) group information; The UAV group information comprises starting positions of the UAVs and current observation states of the UAVs; the current observation state of a UAV comprises a patrol node within an observation range of the UAV and states of other UAVs within the observation range of the UAV; the patrol topology graph comprises a plurality of patrol nodes; An environment space model establishment module is configured to establish a UAV environment space model based on the patrol topology graph, the real-time map information, and the UAV group information; A calculation module is configured to input the current observation state of the UAV into a trained multi-agent reinforcement learning model to determine optimal patrol nodes of the UAVs in the UAV group at a next time point; the multi-agent reinforcement learning model is a model based on a MAPPO algorithm; An action execution submodule is configured to execute a UAV action and obtain an experience tuple of a current time step when the UAV moves to a next position; An advantage function and complete trajectory determination submodule is configured to determine an advantage function and a complete trajectory of the UAV group according to the experience tuple of each time step; A trajectory splitting submodule is configured to split the complete trajectory into a trajectory sequence with a length of L and store the trajectory sequence in an experience replay pool; A sample extraction submodule is configured to extract a set number of samples from the experience replay pool when a storage capacity of the experience replay pool reaches a maximum; A model training submodule is configured to update weights of an actor network and a critic network in the multi-agent reinforcement learning model based on the samples until the actor network and the critic network converge, to obtain the trained multi-agent reinforcement learning model; A state updating module is configured to control the UAVs to perform a patrol task according to the optimal patrol nodes of the UAVs at the next time point and update positions and current observation states of the UAVs in the UAV environment space model; An observation module is configured to re-perform the step of inputting the current observation state of the UAV into the trained multi-agent reinforcement learning model to determine the optimal patrol nodes of the UAVs in the UAV group at the next time point based on the updated current observation states of the UAVs, until all patrol nodes in the patrol topology graph are covered. 9.The wisdom forest fire prevention monitoring and early warning device based on the UAV group according to claim 8, characterized in that, Further comprising a model training module; The model training module specifically comprises: A learning model construction submodule is configured to construct a multi-agent reinforcement learning model; A parameter initialization submodule is configured to initialize actor network parameters, critic network parameters, and an experience replay pool in the multi-agent reinforcement learning model and set a learning rate; An action calculation submodule is configured to take the current observation state of a UAV as an input of an actor network in a MAPPO algorithm, and calculate a UAV action through the actor network for each UAV; A reward calculation submodule is configured to interact the UAV action with an environment based on an unexpected minimization reinforcement learning method to obtain an environment feedback reward and a minimized unexpected reward. 10.The wisdom forest fire prevention monitoring and early warning device based on the UAV group according to claim 9, characterized in that, The advantage function and the complete trajectory determination submodule specifically include: An advantage function determining unit is configured to determine an advantage function of the UAV group according to a formula wherein, γ is a discount factor, λ is a GAE parameter, T is a time step, is an output of the critic network. A complete trajectory determination unit is configured to obtain a complete trajectory according to a large number of experience tuples generated in the process of interaction between the unmanned aerial vehicle and the environment.

Images