A drone pursuit method and device based on perception-decision hierarchical reinforcement learning

Abstract

This invention discloses a drone pursuit method and apparatus based on perception-decision hierarchical reinforcement learning, belonging to the field of drone intelligent control technology. The method includes: acquiring local state estimation information of the target based on activated sensors in a sensor network, wherein the activated sensors are selected by a trained perception model based on the probability density function of sensor prediction accuracy; fusing the local state estimation information acquired by all activated sensors to obtain global state estimation information of the target; and inputting the drone's state information and the target's global state estimation information into a trained decision model to obtain the drone's flight strategy, wherein the reward function used by the decision model during training is determined based on the degree of consistency between the drone's trajectory and the target's trajectory. The flight strategy generated by this invention has good feasibility and high accuracy.

Description

Technical Field

[0001] This invention belongs to the field of unmanned aerial vehicle (UAV) intelligent control technology, specifically relating to a UAV pursuit method and device based on perception-decision hierarchical reinforcement learning. Background Technology

[0002] In recent years, reinforcement learning (RL) has developed rapidly, especially in fields such as games, robot control, and autonomous driving. The pursuit and escape game of high-speed aircraft is dynamic and adversarial, and the entire process must follow the theory of target perception, judgment, decision-making, and action loops—considered one of the most challenging scenarios.

[0003] Reinforcement algorithms (RL) have made significant progress in UAV control and game theory research. Researchers have successfully completed numerous UAV mission scenarios using different RL algorithms, assuming a fully observable Markov process. Furthermore, locally observable Markov decision processes have been incorporated into UAV path planning scenarios. Nevertheless, ensuring that high-speed aircraft autonomously formulate tactical plans in real time, and complete situational awareness, threat assessment, autonomous decision-making, and intelligent control—especially autonomy in pursuit missions—remains a critical challenge.

[0004] Traditional methods overly idealize the process of UAV target detection, often using fully observable information and neglecting the accuracy of actual sensor target estimation. This results in the generated UAV flight strategy failing to achieve the intended target. Summary of the Invention

[0005] This invention provides a drone chasing method and apparatus based on perception-decision hierarchical reinforcement learning, which can solve the above-mentioned technical problems.

[0006] In a first aspect, embodiments of the present invention provide a drone chasing method based on perception-decision hierarchical reinforcement learning, the method comprising: Based on the sensors activated in the sensor network, local state estimation information of the target is obtained, wherein the activated sensors are selected by the trained perception model based on the probability density function of the sensor prediction accuracy. By fusing the local state estimation information acquired by all activated sensors, the global state estimation information of the target is obtained. The state information of the UAV and the global state estimation information of the target are input into the trained decision model to obtain the flight strategy of the UAV. The perception model and the decision model are both trained by reinforcement learning. The reward function used by the decision model during training is determined based on the degree of consistency between the trajectory of the UAV and the target.

[0007] Secondly, embodiments of the present invention provide a drone chasing device based on perception-decision hierarchical reinforcement learning, including a perception module, an acquisition module, a fusion module and a decision module; The sensing module is used to determine the sensor to be activated based on the trained sensing model; The acquisition module is used to acquire local state estimation information of the target based on the sensors activated in the sensor network; The fusion module is used to fuse the local state estimation information acquired by all activated sensors to obtain the global state estimation information of the target. The decision module is used to input the UAV's state information and the global state estimation information into the trained decision model to obtain the UAV's flight strategy. The perception model and the decision model are both trained through reinforcement learning. The reward function used by the decision model during training is determined based on the degree of consistency between the UAV's trajectory and the target's trajectory.

[0008] The beneficial effects of this invention compared to existing technologies are as follows: This invention uses a perception model to select sensors with higher prediction accuracy based on the probability density of sensor prediction accuracy, thus prioritizing the use of observation data with better accuracy and improving the reliability of local state estimation. Subsequently, by fusing the estimation information acquired by the sensors, it integrates the advantages of multi-source perception data, reduces individual errors through complementary correction, and obtains a more accurate global state estimate of the target. Since the decision model determines the environmental reward for the decision model during training based on the consistency between the UAV and the target's flight trajectory, the decision model can achieve efficient coordination between the UAV and the target's movement. This achieves a closed loop from accurate perception to adaptive decision-making, overcoming the problem of traditional methods neglecting estimation accuracy due to idealized sensor information, leading to difficulties in achieving predetermined goals in flight strategies, and significantly improving the target detection and tracking performance of UAVs in real-world scenarios. Attached Figure Description

[0009] Figure 1 This is a schematic diagram of a model structure provided for an embodiment of the present invention; Figure 2 A schematic diagram of a training scenario for a perception model and a decision model provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of a sensor network structure provided in an embodiment of the present invention; Figure 4 A flowchart illustrating the implementation of a drone pursuit method based on perception-decision hierarchical reinforcement learning, provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the structure of a drone chasing device based on perception-decision hierarchical reinforcement learning, provided as an embodiment of the present invention. Detailed Implementation

[0010] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of the invention. However, those skilled in the art will understand that the invention can be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of the invention with unnecessary detail.

[0011] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0012] It should also be understood that the term “and / or” as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0013] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0014] Furthermore, in the description of this invention and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0015] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of the invention include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0016] The present invention will be further described in detail below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.

[0017] Example 1 Figure 1 The diagram shown is a schematic representation of a model structure provided in an embodiment of the present invention.

[0018] As an example, see Figure 1 Both the perception model and the decision model can adopt the actor-critic architecture, each including an action network and an evaluation network.

[0019] For example, during training, the action network of the perception model can select the sensor to be activated based on the state information of the input sensors and output an action vector. This action vector includes the binary decision value of each sensor. , Indicates sensor At time step Whether to activate it. Afterwards, the evaluation network outputs the value of the action network's decision based on the input sensor state information and action vectors. After training is complete, the action network continues to operate normally, while the evaluation network goes into sleep mode.

[0020] Similarly, during training, the action network of the decision model can output action vectors based on the global state estimate of the input target and the UAV's own state information. This motion vector is used to indicate the time step of the drone. Flight strategies at specific times, including drone time steps angular velocity at time Throttle control amount Afterwards, the evaluation network outputs the value of the action network's decision based on the input target's global state estimate, the UAV's own state information, and the action vector. After training is complete, the action network continues to operate normally, while the evaluation network goes into sleep mode.

[0021] Specifically, the throttle control of the drone corresponds to the acceleration and deceleration control of the speed.

[0022] Example 2 Figure 2 This is a schematic diagram of a training scenario for a perception model and a decision model provided in an embodiment of the present invention.

[0023] In some embodiments, see Figure 2The environment mainly includes three types of entities: the pursuer: a drone that performs an autonomous pursuit mission; the escapee: a highly maneuverable non-cooperative target that attempts to evade pursuit through maneuvering; and the sensor network, a perception network composed of multiple sensor nodes (see the intelligent agent represented by the robot in the figure), which is responsible for the cooperative detection and state estimation of the target, and transmits the fused information (i.e., the global state estimation information of the target) to the pursuer.

[0024] For example, the drone pursuit decision problem can be modeled as a partially observable Markov Decision Process (POMDP), and the decision-making capabilities of the perception model and the decision-making model can be trained through reinforcement learning.

[0025] For example, in the perception-decision hierarchical architecture, the perception model is designed for wireless sensor network scenarios with limited communication bandwidth and energy consumption, and realizes distributed collaborative perception and target state estimation. After the perception model completes the collaborative estimation of the target state, the purpose of the decision model is to transform these estimates into specific flight control commands to achieve trajectory following and pursuit of the target.

[0026] Specifically, during training, the perception model can sample the state information of each sensor from its own state space as input, outputting action vectors to determine which sensor is activated. The environment, based on the perception model's reward function, returns a reward value according to the action vectors output by the action network of the perception model. Simultaneously, the evaluation network of the perception model outputs the value of the action network's decision for this instance. Then, the local estimation information of all activated sensors is fused to obtain the global state estimation information of the target. This global state estimation information of the target, along with the UAV's state information sampled from the decision model's state space, is input into the decision model. The action network in this model generates the UAV's flight strategy, the evaluation network outputs the value of the action network's decision for this instance, and the reward for this decision is obtained from the environment. Finally, the network parameters of the perception model and the decision model are updated using the action vectors and rewards from both models, completing the training process.

[0027] For example, the training objectives for the perception model and the decision model can be expressed as: , in, This represents the optimization objective of the model. Represents the mathematical expectation. In time step The importance ratio represents the ratio of the strategy before and after the update. Indicates time step The action output by the action network, Indicates time step Status information sampled from the environment at that time. Indicates the current network parameters When the environment is Time-motion network selection The probability, Indicates the parameters in the old network When the environment is Time-motion network selection The probability, It is a time step Estimating the action advantage at a given point, based on time steps The reward given by the environment indicates the advantage of this action compared to the average behavior. This indicates a cut operation. It is a hyperparameter, usually set to 0.1 or 0.2, used to control the step size for each policy update.

[0028] This invention employs a shearing strategy to update model parameters, which can limit the magnitude of the strategy update and avoid the problem of excessive strategy changes, while ensuring an improvement in expected returns.

[0029] In one possible implementation, the state space of the perception model can be represented as follows: , Represents the sum of all state spaces. Indicates sensor The state space. Accordingly, the sensor time steps sampled from the state space. The state information at that time can be represented as: It can include local measurement information from the sensor. Local estimation information Neighbor node information , communication consumption KL divergence and posterior covariance .

[0030] For example, see Figure 3 In sensor networks, such as wireless sensor networks In this system, there is a one-to-one correspondence between sensors and nodes. Represents a set of nodes. The total number of sensors; nodes capable of communication are connected via directed edges. (i.e., sensor) ) to node Directed edge , Let it be the set of edges. Meaning node Can send to nodes Send message, yes The neighboring nodes are other sensors that can communicate with the sensor.

[0031] In one example, the KL divergence is calculated based on the probability density function of the sensor's history and the estimated prediction accuracy, and is used to measure the difference between the two probability densities.

[0032] For example, the KL divergence can be calculated using the following formula: , in, express and KL divergence between them , All represent sensors At time step The previous The probability density function of the historical prediction accuracy of the step. , All represent sensors At time step The probability density function of the estimated prediction accuracy. The separator is used to distinguish between two probability distributions involved in the calculation.

[0033] In one example, the reward function of the perception model satisfies the following formula: , in, For the reward of the perception model, The total number of sensors, Indicates sensor At time step Fusion estimation, Indicates time step Estimate the true state. Describing the L2 norm, Indicates sensor At time step Whether it is started, For time step Dynamic penalty factor at time.

[0034] For example, the dynamic penalty factor can be automatically adjusted based on the cumulative communication ratio, which can encourage the sensor to explore communication behavior in the early stages of training and focus more on communication conservation in the later stages.

[0035] Specifically, the dynamic penalty factor can satisfy the following formula: .

[0036] In one possible implementation, the local state estimation information acquired by each activated sensor and the information of its neighboring nodes can be sequentially subjected to Kalman filtering and flooding to obtain the preliminary fused state estimation information acquired by each activated sensor; then, based on the geometric mean fusion algorithm, the preliminary fused state estimation information acquired by all activated sensors is fused to obtain the global state estimation information of the target.

[0037] For example, the status information of the drone It can be represented as: ,in, The location of the drone. The target location estimated by the perception model. Let Euclidean distance be the distance between the drone and the target. For the drone's heading, The target azimuth angle, For heading error, For the scalar velocity of the drone, The estimated speed for the target.

[0038] In one example, information vectors can be used. and information matrix Alternate mean vector Covariance Matrix This is used to filter the local state estimation information obtained by the sensor and the information of its neighboring nodes.

[0039] Specifically, it can make , .

[0040] For example, the filtering prediction step in Kalman filtering can be represented as: , , in, For information vectors, Here is the state transition matrix. Indicates transpose. It is the identity matrix. For information matrix, Represents the information vector time step The posterior update value at that time, Indicates time step Time step The predicted prior value at that time, Indicates time step The posterior update value at that time.

[0041] For example, the measurement update steps of Kalman filtering can be represented as follows: , , in, For time step Time step The prediction information matrix at that time For the observation matrix, To measure the noise covariance matrix, For time step Time step Predictive information vector at time For sensors At time step The measured value, This indicates transpose.

[0042] In one example, efficient communication between nodes is crucial for achieving network consensus and accurate state estimation in distributed state estimation. Communication protocols define how information is shared and updated within the network, directly impacting the efficiency and reliability of the estimation process. Among various communication strategies, flooding protocols are particularly noteworthy due to their unique information propagation method. Flooding communication protocols avoid reusing any information and provide the highest convergence efficiency for network consensus; therefore, this invention employs flooding protocols for secondary local fusion of information.

[0043] Specifically, flooding can be divided into two stages: the initial stage and the flooding stage. In the initial stage, the sensor collects information from its neighboring nodes, and in the flooding stage, it collects new information received by its neighboring nodes during the previous information diffusion.

[0044] For example, the initial stage of flooding treatment can be represented as: , in, Indicates sensor Information received in the initial stage, Represents the union, Indicates sensor Belongs to sensors The set of neighboring nodes , Indicates sensor Information received in the initial phase.

[0045] For example, the flooding stage of flood control can be represented as: , in, Indicates sensor Time steps during the flooding phase The information received at that time , Sensors During the flooding phase sequence Information received during the next iteration express , The difference set. Specifically, when (i.e., the number of iterations is greater than the network diameter) At this point, the information has been fully transmitted through communication, and no new information needs to be transmitted in the next step. , This represents the empty set.

[0046] In one example, the information fusion process using the geometric mean fusion algorithm can be represented as: , , in, For the first The information vector at the next iteration For sensors At time step No. Information received in the next iteration For sensors With sensors The non-negative weights between them, whose values ​​are determined by the metropolitan area weights, are used to achieve rapid convergence of the sensor network. For the first The information vector at the next iteration express and The difference set, This is the information vector from the last trigger moment. For the first Information matrix at the next iteration For the first Information matrix at the next iteration This is the information matrix from the last triggering time.

[0047] Similarly, here we use Replaces posterior information.

[0048] In one possible implementation, unlike traditional point-based pursuit that only aims for "shortest-time interception," the decision model provided by this invention focuses on "trajectory-following pursuit": while ensuring rapid approach, it closely follows the target's trajectory and improves flight stability through motion smoothing constraints. Therefore, the reward function of the decision model of this invention includes geometric tracking reward, motion regularization reward, and spatiotemporal consistency reward.

[0049] For example, geometric tracking rewards can be used to constrain the drone's flight attitude to align with the target. For example, action regularization rewards can be used to smooth the flight path of a drone and prevent high-frequency oscillations.

[0050] For example, spatiotemporal consistency rewards are used to constrain drones to mimic the flight trajectory of a target within a preset time interval.

[0051] Specifically, both geometric tracking rewards and spatiotemporal consistency rewards can reflect the degree of consistency between the UAV and the target's flight trajectory.

[0052] In one example, since traditional Euclidean distance minimization methods struggle to capture flight attitude constraints, this invention introduces the Frenet coordinate system, decomposing the tracking error into longitudinal and lateral errors relative to the target velocity vector. Therefore, the geometric tracking reward can be a weighted sum of position approximation reward, heading velocity reward, forward velocity reward, lateral error reward, and along-track error reward.

[0053] For example, the location proximity reward can employ a hyperbolic tangent function to provide dense gradient guidance near the target, which satisfies the following formula: , in, For proximity rewards, It is the hyperbolic tangent function. The capture distance indicates that the drone is close enough to the target to complete the mission.

[0054] For example, the heading speed reward is used to constrain the UAV not only to reach the target point, but also to maintain a motion flow field consistent with the target, which can satisfy the following formula: , in, For heading speed bonus, This indicates a cut operation. The angle between the drone's speed and its heading.

[0055] in: , in, For the speed of the drone, Represents the velocity vector of the drone. Indicates the length of the modulus.

[0056] For example, the forward velocity reward can satisfy the following formula: , in, For forward speed bonus, , Let be the relative position vector between the UAV and the target. To limit the maximum speed of the drone.

[0057] For example, the horizontal track error reward can satisfy the following formula: , in, This is the bonus value for the horizontal rail error. For lateral error, The normal vector of the target trajectory. For trajectory scale parameters.

[0058] For example, the along-track error reward can satisfy the following formula: , in, As a reward for errors along the track, For longitudinal error, is the tangential vector of the target trajectory.

[0059] In one example, to prevent high-frequency oscillations and make the flight trajectory smoother, the amplitude of the flight policy and its first-order difference can be penalized. Therefore, the action regularization reward satisfies the following formula: , in, For action regularization rewards, , There are two weight parameters. , They are time steps Time step The amplitude of the drone flight strategy output by the decision-making model. It represents the square of the modulus.

[0060] In one example, to endow a UAV with a deep understanding of highly dynamic target trajectories, enabling the UAV not only to approach the target at a "position in a certain frame," but also to mimic the evolution pattern of the target trajectory in space and time within a preset time interval, this invention defines the alignment cost of two trajectory segments in the spatiotemporal consistency reward, and explicitly incorporates two types of information—spatial position and motion direction (time derivative)—into the cost. Therefore, the spatiotemporal consistency reward can be calculated using the following formula: , in, For spatiotemporal consistency rewards, , , , These are, respectively, drones and targets from time steps. Start to time step of Step position set, Indicates modulo; if ,but ,like but , This is the window width.

[0061] in: , , , in, For time step The drone trajectory at the time step The single-step alignment cost of the target trajectory at time, and, ,and ,in express and Spatial location difference, , The drone at time step Spatial location coordinates of the target in time step Spatial location coordinates; This indicates the difference in the direction of velocity. , Representing time steps Drone trajectory and time step The unit tangent of the target trajectory at time. Adjust the weights of both. Cumulative cost matrix. and . , These represent the time steps of the drone. The goal is to achieve the following within a certain timeframe. The trajectory increment. , The drone at time step Spatial location coordinates of the target in time step Spatial location coordinates.

[0062] Specifically, ,in, , The drone at time step Time step Location, , The target at time step Time step The position. When When the first frame is repeated, fill the left side with the first frame.

[0063] By treating rewards as a soft constraint, the decision-making model can be induced to spontaneously develop flight strategies that fit the target curve in the process of maximizing long-term returns.

[0064] Alternatively, since shape matching is meaningless when the drone is extremely far from the target, a gating factor can be used. Only when the drone enters the effective area near the target (measured by distance scale) When defined, trajectory-level rewards (such as spatiotemporal consistency rewards and geometric tracking rewards) are activated, thus achieving a smooth transition from "global convergence" to "fine-grained tracking".

[0065] The perception model provided by this invention selects sensors with higher prediction accuracy based on the probability density of sensor prediction accuracy for activation. This avoids the problem that traditional methods based on fixed threshold activation are difficult to adapt to heterogeneous target maneuvers and link quality in time and space. It prioritizes the use of observation data with higher accuracy, improving the reliability of local state estimation. Then, it fuses the estimation information acquired by the sensors, which can integrate the advantages of multi-source perception data and reduce individual errors through complementary correction to obtain a more accurate global state estimation of the target. Furthermore, it determines the reward of the decision model to the environment based on the consistency between the UAV and the target flight trajectory. This avoids the problems of poor training stability and interpretability of traditional methods that determine the reward of the decision model based solely on the distance between the target and the UAV. It drives the decision model to continuously optimize the strategy during training to achieve efficient coordination with the target's motion. Thus, it realizes a closed loop from accurate perception to adaptive decision-making, overcoming the problem that traditional methods ignore estimation accuracy due to idealizing sensor information, making it difficult for the flight strategy to achieve the predetermined goal. This significantly improves the target detection and tracking performance of UAVs in real-world scenarios.

[0066] Example 3 Figure 4 The diagram shown illustrates an implementation flowchart of a drone pursuit method based on perception-decision hierarchical reinforcement learning, provided by an embodiment of the present invention. As an example and not a limitation, this method can be applied to the aforementioned voice-enabled electronic devices. The method may include steps S401-S403, which are described below.

[0067] S401, based on the sensors activated in the sensor network, acquires local state estimation information of the target.

[0068] For example, the sensor that is activated is selected by the perception model trained in Embodiment 2 above based on the probability density function of the sensor prediction accuracy; S402, fuse the local state estimation information acquired by all activated sensors to obtain the global state estimation information of the target.

[0069] For example, the fusion method in step S402 is the same as that in Embodiment 2 above, and can be found in Embodiment 2 above for details, which will not be repeated here.

[0070] S403 inputs the UAV's state information and the target's global state estimation information into the trained decision model to obtain the UAV's flight strategy.

[0071] For example, the perception model and decision model here can both be the perception model and decision model trained based on the above embodiment 2.

[0072] For example, the reward function used by the decision model during training is determined based on the degree of consistency between the drone's trajectory and that of the target.

[0073] This invention selects sensors with higher prediction accuracy based on the probability density of sensor prediction accuracy using a perception model, prioritizing the use of more accurate observation data and improving the reliability of local state estimation. Then, it fuses the estimation information acquired by the sensors, integrating the advantages of multi-source perception data and reducing individual errors through complementary correction to obtain a more accurate global state estimate of the target. Since the decision model determines the environmental reward based on the consistency between the UAV and the target's flight trajectory during training, it enables efficient coordination between the UAV and the target's movement. This achieves a closed loop from accurate perception to adaptive decision-making, overcoming the problem of traditional methods neglecting estimation accuracy due to idealized sensor information, leading to flight strategies failing to achieve predetermined goals. This significantly improves the target detection and tracking performance of UAVs in real-world scenarios.

[0074] Example 4 Figure 5The diagram shown is a schematic representation of a drone chasing device based on perception-decision hierarchical reinforcement learning according to an embodiment of the present invention. It is an example and not a limitation. The device may include a perception module, an acquisition module, a fusion module and a decision module.

[0075] For example, the perception module is used to determine the activated sensors based on the trained perception model; the acquisition module is used to acquire the local state estimation information of the target based on the activated sensors in the sensor network; the fusion module is used to fuse the local state estimation information acquired by all activated sensors to obtain the global state estimation information of the target; and the decision module is used to input the state information of the UAV and the global state estimation information of the target into the trained decision model to obtain the flight strategy of the UAV. Here, both the perception model and the decision model are trained through reinforcement learning, and the reward function used by the decision model during training is determined based on the degree of consistency between the trajectory of the UAV and the target.

[0076] This invention selects sensors with higher prediction accuracy based on the probability density of sensor prediction accuracy using a perception model, prioritizing the use of more accurate observation data and improving the reliability of local state estimation. Then, it fuses the estimation information acquired by the sensors, integrating the advantages of multi-source perception data and reducing individual errors through complementary correction to obtain a more accurate global state estimate of the target. Since the decision model determines the environmental reward based on the consistency between the UAV and the target's flight trajectory during training, it enables efficient coordination between the UAV and the target's movement. This achieves a closed loop from accurate perception to adaptive decision-making, overcoming the problem of traditional methods neglecting estimation accuracy due to idealized sensor information, leading to flight strategies failing to achieve predetermined goals. This significantly improves the target detection and tracking performance of UAVs in real-world scenarios.

[0077] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

Claims

1. A drone pursuit method based on perception-decision hierarchical reinforcement learning, characterized in that, include: Based on the sensors activated in the sensor network, local state estimation information of the target is obtained, wherein the activated sensors are selected by the trained perception model based on the probability density function of the sensor prediction accuracy. By fusing the local state estimation information acquired by all activated sensors, the global state estimation information of the target is obtained. The state information of the UAV and the global state estimation information of the target are input into the trained decision model to obtain the flight strategy of the UAV. The perception model and the decision model are both trained by reinforcement learning. The reward function used by the decision model during training is determined based on the degree of consistency between the trajectory of the UAV and the target.

2. The method according to claim 1, characterized in that, The sensor state information input into the perception model during training includes: local measurement information, local estimation information, neighbor node information, communication cost, KL divergence, and posterior covariance; wherein, the neighbor nodes are other sensors that can communicate with the sensor, and the KL divergence is calculated based on the probability density function of the sensor's historical and estimated prediction accuracy.

3. The method according to claim 2, characterized in that, The local state estimation information and the local estimation information include the estimated position and estimated velocity of the target, and the local measurement information includes the actual position and actual velocity of the target.

4. The method according to claim 1, characterized in that, The reward function used by the perception model during training satisfies the following formula: ， in, The reward for the perception model. The total number of sensors, Indicates sensor At time step Fusion estimation, Indicates time step Estimate the true state at that time. Represents the L2 norm. Indicates sensor At time step Whether it is started, For time step Dynamic penalty factor at time.

5. The method according to claim 1, characterized in that, The process of fusing local state estimation information acquired by all activated sensors to obtain global state estimation information of the target includes: Kalman filtering and flooding are performed sequentially on the local state estimation information acquired by each activated sensor and the information of its neighboring nodes to obtain the preliminary fused state estimation information acquired by each activated sensor. Based on the geometric mean fusion algorithm, the preliminary fused state estimation information obtained by all activated sensors is fused to obtain the global state estimation information of the target.

6. The method according to claim 1, characterized in that, The reward functions used by the decision model during training include geometric tracking reward, action regularization reward, and spatiotemporal consistency reward. The geometric tracking reward is used to constrain the UAV's flight attitude to be consistent with the target. The action regularization reward is used to smooth the UAV's flight trajectory and prevent high-frequency oscillations. The spatiotemporal consistency reward is used to constrain the UAV to imitate the target's flight trajectory within a preset time interval.

7. The method according to claim 6, characterized in that, The geometric tracking reward is a weighted sum of the position approximation reward, the heading speed reward, the forward speed reward, the lateral error reward, and the along-track error reward. The position approximation reward, the heading speed reward, the forward speed reward, the lateral error reward, and the along-track error reward sequentially satisfy the following formula: ， in, Indicates the distance between the drone and the target. The reward is for approximating the position. It is the hyperbolic tangent function. For capture distance; ， in, For the aforementioned heading speed bonus, This indicates a cut operation. The angle between the drone's speed and its heading; ， in, For the forward velocity reward, For the scalar velocity of the drone, , Let be the relative position vector between the UAV and the target. To limit the maximum speed of the drone; ， in, The error of the horizontal rail is rewarded. For lateral error, For trajectory scale parameters; ， in, The reward for the track error is... This represents the longitudinal error.

8. The method according to claim 6, characterized in that, The action regularization reward satisfies the following formula: ， in, The reward is regularized for the action. , There are two weight parameters. , They are time steps Time step The amplitude of the drone flight strategy output by the decision-making model. It represents the square of the modulus.

9. The method according to claim 6, characterized in that, The spatiotemporal consistency reward satisfies the following formula: ， in, For the spatiotemporal consistency reward, , , , These are, respectively, drones and targets from time steps. Start to time step of Step position set; Represents the cumulative cost matrix. This indicates taking the modulus.

10. A drone pursuit device based on perception-decision hierarchical reinforcement learning, characterized in that, It includes a perception module, an acquisition module, a fusion module, and a decision-making module; The sensing module is used to determine the sensor to be activated based on the trained sensing model; The acquisition module is used to acquire local state estimation information of the target based on the sensors activated in the sensor network; The fusion module is used to fuse the local state estimation information acquired by all activated sensors to obtain the global state estimation information of the target. The decision module is used to input the UAV's state information and the global state estimation information into the trained decision model to obtain the UAV's flight strategy. The perception model and the decision model are both trained through reinforcement learning. The reward function used by the decision model during training is determined based on the degree of consistency between the UAV's trajectory and the target's trajectory.

Images