An unmanned system active target monitoring method based on two-party game
By dividing the target monitoring process of the unmanned system into monitoring and search stages using a two-party game-based approach, and by using a deep neural network optimization strategy, the problems of randomness and adversarial nature of target movement are solved, and more efficient target monitoring is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING INST OF TECH
- Filing Date
- 2022-10-28
- Publication Date
- 2026-05-15
AI Technical Summary
Existing technologies struggle to effectively address the randomness and adversarial nature of target movement during target monitoring in unmanned systems, leading to target loss and complex search processes. Furthermore, they fail to optimize the monitoring process to extend monitoring time and reduce the number of searches.
A two-party game-based approach is adopted to divide the target monitoring process into a monitoring phase and a search phase. A deep neural network is constructed using convolutional neural networks and long short-term memory networks. The strategies of the monitor and the target are optimized through reinforcement learning, and a corresponding reward function is set to extend the monitoring phase and shorten the search phase.
It improved the monitoring performance of unmanned systems, extended the duration of the monitoring process, reduced the number and time of the search process, and enhanced the monitoring effect on adversarial targets.
Smart Images
Figure CN115618738B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to an active target monitoring method for unmanned systems, specifically a target monitoring method that constructs a two-party zero-sum game based on the target and the monitor, uses reinforcement learning to solve for an approximate Nash equilibrium, and provides the monitor with an advantageous strategy. It belongs to the field of artificial intelligence and target monitoring technology. Background Technology
[0002] Currently, target monitoring technology is widely used in various fields, including civilian and military applications, primarily for acquiring target status and controlling the movement of monitors. With the rapid development of intelligent game theory, new approaches have been provided to address the nonlinearity and adversarial nature of moving targets in real-world scenarios. Target monitoring based on intelligent game theory has become one of the most popular research directions at the intersection of game theory and target monitoring.
[0003] Target monitoring technology, based on environmental perception results, continuously monitors targets in the environment to obtain real-time target status information, providing input for the motion trajectory planning of actuators and subsequent task decisions. Target monitoring is an important research direction in intelligent decision-making and control problems in unmanned systems such as autonomous vehicles and drones. It controls the movement of drones and autonomous vehicles according to the target's motion state, ensuring that the target remains within the detection range of onboard / airborne sensors.
[0004] For example, in autonomous driving, vehicles need to monitor targets in the environment to plan safe and reasonable driving routes. In drone target monitoring missions, for adversarial or random moving targets in the environment, drones must constantly monitor the target's status to ensure their own flight safety and complete specific tasks. Therefore, target monitoring based on unmanned systems is widely present in many aspects of society and has significant social benefits.
[0005] Currently, some progress has been made in game-theoretic target monitoring methods, but some problems and shortcomings still exist. For example, in non-adversarial scenarios, the target's movement purpose is unknown, resulting in high nonlinearity and randomness, making explicit modeling difficult. In adversarial environments, the target and the monitor have opposite movement purposes, leading to target loss during monitoring. However, current methods often neglect the search process after target loss; in reality, the target search and monitoring processes should alternate. Furthermore, it is necessary to consider how to obtain the monitor's dominant strategy, maximizing the monitoring process while minimizing the search process. Summary of the Invention
[0006] The purpose of this invention is to address the problems and deficiencies of existing technologies. In order to effectively solve a series of technical problems in the target monitoring process of unmanned systems, such as the failure to consider the search process when the target monitoring of the unmanned system fails due to the randomness, uncertainty and adversarial nature of the monitored target movement, and how to extend the monitoring process time and reduce the number and time of the search process, this invention creatively proposes an active target monitoring method for unmanned systems based on two-party game theory.
[0007] First, the relevant concepts will be explained.
[0008] 1. Monitoring phase based on Markov observation process
[0009] This refers to the target monitoring phase, where the monitor and the target can observe each other's state, and at this point, both are considered to be fully observable of the environmental state.
[0010] Markov processes use quintuples<S,A,P,R,γ> Let S be the state space, A be the action space, P be the state transition function, R be the immediate reward, and γ be the discount factor.
[0011] 2. Search Phase Based on Partial Markov Observation Process
[0012] This refers to a situation where, during the target search phase, both the monitor and the target are outside each other's detection range, meaning they cannot observe each other's state. In this case, the rewards obtained from their interactions with the environment are no longer zero-sum rewards.
[0013] Partially observable Markov processes are used<S,A,Z,P,O,R,γ> The septuplet is represented as follows: O is the observation space, where the monitor cannot observe the state but can observe it; Z is the observation probability function, representing the relationship between the observation and the state; S is the state space; A is the action space; P is the state transition function; R is the immediate reward; and γ is the discount factor.
[0014] 3. Enhance the diversity of target strategies
[0015] Based on the zero-sum game between the monitor and the target, and the idea of "the stronger the opponent, the stronger the target," a better monitor strategy can be trained by increasing the diversity of the target's strategies.
[0016] The present invention is achieved using the following technical solution.
[0017] An active target monitoring method for unmanned systems based on two-party game theory includes the following steps:
[0018] Step 1: Model the monitor and the target in the unmanned system as two independent intelligent agents. The target monitoring problem includes a target monitoring phase and a target search phase. The target monitoring phase is modeled as a zero-sum game based on Markov processes, and the target search phase is modeled as a non-zero-sum game based on partially observed Markov processes.
[0019] The independent intelligent agent refers to constructing the proactive monitoring problem as a multi-agent system, where each intelligent agent has an independent policy space.
[0020] The target monitoring phase and target search phase refer to the period when the target remains within the monitor's monitoring range. When the target disappears from the monitor's range, the monitor needs to locate and re-enter the target's monitoring range; this is considered the target search phase.
[0021] The monitoring range refers to the detection range of the sensing systems of both the monitor and the target. If the target's state cannot be detected outside this range, it is considered that the monitor has lost its monitoring capability and the target has successfully evaded monitoring.
[0022] The non-zero-sum game based on partially observed Markov processes is represented as follows: <S,A m， A t， P, O m， O t， r m， r t， , γ>, where the monitor's action is represented as a m ∈A m The reward is represented as r. m =r m (s t a m，t The strategy is represented as π. m (a m，t |o m，t The target's action is represented as a. t ∈A t The reward is represented as r. t =r t (s t a t，t The strategy is represented as π. t (a t，t |o t，t S is the state space, A is the state space. m A t Let O represent the action spaces of the monitor and the target, respectively. P is the state transition function, and O is the target's action space. m O t Let s represent the observation space of the monitor and the target, respectively. t This represents the state at time t, γ is the discount factor, and am，t a t，t Let π represent the actions of the monitor and the target at time t, respectively. m π t Represent the strategies of the monitor and the target, respectively. m，t o t，t Let represent the observation space of the monitor and the target at time t, respectively.
[0023] Step 2: Construct a deep neural network by combining a convolutional neural network with a long short-term memory network.
[0024] Convolutional neural networks are used to extract the state features of the monitor and the target, while long short-term memory networks are used to remember the actions of the monitor and the target during the monitoring phase. This allows them to provide better strategies for each during the search phase.
[0025] Step 3: Define the elements of deep reinforcement learning.
[0026] Considering the actual movement of unmanned systems, the states and actions of both the monitor and the target exist in a continuous space. The reward functions for both the monitor and the target include a monitoring phase and a search phase, with the goal of maximizing their own reward.
[0027] Furthermore, step 3 may include the following steps:
[0028] Step 3.1: The monitor's status includes the monitor's position, the distance between the monitor and the target, and the angle between their directions of movement. The target's status includes the target's position, the distance between the target and the monitor, and the angle between their directions of movement.
[0029] The action space of the monitor and the target includes the direction of movement and the distance traveled, as detailed below:
[0030] State m =[x m y m d m-t θ m-t Action m =[ρ m θ m ]
[0031] State t =[x t y t d m-t θ m-t Action t =[ρ t θ t ]
[0032] Where, State m Statet Representing the states of the monitor and the target respectively, Action m Action t These represent the actions of the monitor and the target, respectively. m and y m Indicates the location of the monitor, d m-t θ represents the distance between the monitor and the target. m-t ρ represents the directional angle between the monitor and the target. m θ represents the stride length of the monitor. m Indicates the direction of movement of the monitor; x t and y t Indicates the position of the target, ρ t θ represents the target's step size. t Indicates the direction of the target's movement.
[0033] Step 3.2: Reward Setting in the Monitoring Phase. The monitor and the target constitute a zero-sum game. Therefore, their rewards are opposite; that is, the monitor's gain is the target's loss. Specifically, the monitor's gain includes higher rewards the closer they are to the target, and higher rewards for changes in the monitor's movement angle within a reasonable range, while the opposite results in gains for the target. Details are as follows:
[0034] Reward m，m =w d ·min(d m-t / d max -1,0)+w θ1 ·[min(θ m-t -θ max θ max )] -1 +w θ2 ·[max(θ m-t -θ min θ min )] -1 +w t ·(θ m，t -θ m，t-1 ) -1
[0035] Reward t =-Reward m
[0036] Among them, Reward m，m Reward represents the monitor's reward during the monitoring phase. t d represents the reward for the goal. m-t d represents the distance between the monitor and the target. max min(d) represents the maximum monitoring range of the monitor. m-t / d max -1, 0) indicates that the value of this item is 0 when the target is outside the monitoring range. d >0 indicates a coefficient related to distance. θ max θ min θ represents the maximum and minimum angles between the movement directions of the monitor and the target, respectively. The monitor's movement direction should remain between the maximum and minimum angles. m，t θ m，t-1 These represent the movement directions of the monitor at two adjacent moments. Due to energy consumption considerations for unmanned systems, the smaller the angle, the lower the reward. θ1 w θ2 This is a coefficient that constrains the direction of motion.
[0037] Step 3.3: Reward Setting for the Search Phase. To address the issue of monitors losing the target during the target search phase, the rewards for monitors are increased to include penalties, while also encouraging them to continuously approach the target, thus minimizing the duration of the search phase.
[0038] Specifically as follows:
[0039] Reward m，s =Reward m，m +w s ·max((d m-t -d max ) -1 ,0)-w p ·d m-t Among them, Reward m，s w represents the reward for the monitor during the search phase. s w is a control factor related to the target distance. s >0; w p w is a control factor related to the penalty item. p >0.
[0040] Step 4: Strategies to enhance the target advantage.
[0041] Record the target's strategies during the search phase; these strategies enable the target to evade the monitor. Therefore, these are advantageous strategies relative to the monitor. By enhancing the target's advantageous strategies, the monitor's strategy can be optimized.
[0042] Step 5: Optimize the target monitoring process.
[0043] In the target monitoring phase, an approximate solution to the Nash equilibrium is obtained through reinforcement learning algorithms, thus realizing the target monitoring process. In long-term monitoring tasks, the monitoring and search processes alternate. By increasing the diversity of target strategies, the monitoring phase is gradually extended while the search phase is gradually shortened, thereby achieving better monitoring results.
[0044] From step 1 to step 5, based on the multi-agent monitoring process modeling and phased reward setting of the two-party game, the duration of the target monitoring process is optimized by increasing the diversity of target strategies.
[0045] Step 6: Based on the monitoring task, monitor the targets, especially those with randomness and adversarial nature.
[0046] Beneficial effects
[0047] The method of the present invention has the following advantages compared with the prior art:
[0048] This method fully considers the uncertainty and adversarial nature of target movement in target monitoring tasks, as well as monitoring capabilities during long-term monitoring. Based on a two-party game, it constructs an interaction process between two independent intelligent agents, dividing the entire monitoring process into a monitoring phase and a search phase, each with its own reward function. By increasing the diversity of target strategies, the monitor's reward is enhanced, thereby extending the duration of the monitoring process, reducing the search time, and improving the monitor's performance in long-term monitoring tasks. Attached Figure Description
[0049] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation
[0050] The method of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments.
[0051] like Figure 1 As shown, an active target monitoring method for unmanned systems based on two-party game theory includes the following steps:
[0052] Step A: Modeling the monitoring process of independent intelligent agents based on two-party game theory;
[0053] In this specific embodiment, the steps are the same as those in step 1 of the invention.
[0054] Step B: Construct a neural network model by combining a long short-term memory network;
[0055] Specifically, in this embodiment, the procedure is the same as step 2 of the invention.
[0056] Step C: Setting the status and action space of the monitor and the target;
[0057] Specifically, in this embodiment, the steps are the same as in step 3.1 of the invention.
[0058] Step D: Setting rewards for the monitoring phase;
[0059] Specifically, in this embodiment, the steps are the same as in step 3.2 of the invention.
[0060] Step E: Search Phase Reward Settings:
[0061] Specifically, in this embodiment, step 3.2 of the invention is the same;
[0062] Step F: Enhance the diversity of target strategies;
[0063] Specifically, in this embodiment, it is the same as step 4 in the invention.
[0064] Step G: Optimization of the target monitoring process;
[0065] Specifically, in this embodiment, step 5 is the same as in the invention.
[0066] Step H: Based on the monitoring task, monitor the targets, especially those with randomness and adversarial characteristics.
[0067] Example
[0068] The example described is "In an adversarial environment, UAV A (the monitor) monitors another UAV B (the target) for an extended period, ensuring the target remains within the monitor's detection range." This example will illustrate the active target monitoring method for unmanned systems based on two-party game theory described in this invention. The specific operational steps of the method will be explained in detail below.
[0069] A method for active target monitoring in unmanned systems based on two-party game theory, such as... Figure 1 As shown, it includes the following steps:
[0070] Step A: Modeling the monitoring process;
[0071] Specifically, in this embodiment, drone A and drone B are modeled as independent intelligent agents, and during the monitoring phase, the two constitute a zero-sum game.
[0072] Step B: Construct a neural network model;
[0073] In this specific embodiment, a model is constructed using a long short-term memory network for the search phase. Drone A can estimate the position of Drone B, and Drone B expands its strategy by using its actions when it is not being monitored.
[0074] Step C: Setting the status and action space of the monitor and the target;
[0075] Specifically, in this embodiment, the state and action space of drone A and drone B are included;
[0076] Step D: Setting rewards for the monitoring phase;
[0077] Specifically, in this embodiment, the rewards for drone A and drone B are constructed based on distance and angle constraints, and the rewards for the two are opposite.
[0078] Step E: Search Phase Reward Settings:
[0079] Specifically, in this embodiment, a penalty is added to drone A, and drone A is encouraged to continuously explore and approach drone B;
[0080] Step F: Enhance the diversity of target strategies;
[0081] Specifically, in this embodiment, by increasing the strategy diversity of drone B, the monitoring capability of drone A is indirectly improved in a zero-sum game.
[0082] Step G: Optimization of the target monitoring process;
[0083] Specifically, in this embodiment, reinforcement learning training is used to increase the time spent on the monitoring phase of UAV A, thereby reducing the number of searches and the time spent on the search.
[0084] Step H: Based on the monitoring task, monitor targets that are random and adversarial.
Claims
1. A method for active target monitoring in an unmanned system based on two-party game theory, characterized in that, Includes the following steps: Step 1: Model the monitor and the target in the unmanned system as two independent intelligent agents. The target monitoring problem includes a target monitoring phase and a target search phase. The target monitoring phase is modeled as a zero-sum game based on Markov processes, and the target search phase is modeled as a non-zero-sum game based on partially observed Markov processes. Among them, independent intelligent agents construct the proactive monitoring problem as a multi-agent system, where each intelligent agent has an independent policy space; The target monitoring phase and the target search phase refer to the period when the target remains within the monitor's monitoring range. This is considered the target monitoring phase. When the target disappears from the monitor's monitoring range, the monitor needs to find the target and bring it back into the monitoring range. This is considered the target search phase. The monitoring range refers to the detection range of the sensing systems of both the monitor and the target. If the other party's state cannot be detected beyond this range, it is considered that the monitor has lost its monitoring capability and the target has successfully evaded monitoring. Step 2: Construct a deep neural network by combining a convolutional neural network with a long short-term memory network; Among them, the convolutional neural network is used to extract the state features of the monitor and the target, and the long short-term memory network is used to remember the actions of the monitor and the target during the monitoring phase. Step 3: Define the elements of deep reinforcement learning; The states and actions of both the monitor and the target are in a continuous space; the reward functions of both the monitor and the target include a monitoring phase and a search phase, and the goal of both is to maximize their own reward. Step 3.1: The monitor's status includes the monitor's position, the distance between the monitor and the target, and the angle between their directions of movement; the target's status includes the target's position, the distance between the target and the monitor, and the angle between their directions of movement. The action space of the monitor and the target includes the direction of movement and the distance traveled, as detailed below: , , , in, , These represent the states of the monitor and the target, respectively. These represent the actions of the monitor and the target, respectively. Indicates the location of the monitor. Indicates the distance between the monitor and the target. Indicates the directional angle between the monitor and the target. Indicates the stride length of the monitor. Indicates the direction of movement of the monitor; Indicates the location of the target. Indicates the target's step length. Indicates the direction of the target's movement; Step 3.2: Setting rewards during the monitoring phase. The monitor and the target constitute a zero-sum game; their rewards are opposite, that is, the monitor's gain is the target's loss. The monitor's gains include: the closer the monitor is to the target, the higher the reward; the more reasonable the monitor's angle of movement changes, the higher the reward, and vice versa, it becomes the target's gain. Specifically as follows: , in, This indicates the reward for the monitor during the monitoring phase. Rewards that represent the goal Indicates the distance between the monitor and the target. Indicates the maximum monitoring range of the monitor. This indicates that the value of this item is 0 when the target is outside the monitoring range; This represents a coefficient related to distance; These represent the maximum and minimum angles between the movement directions of the monitor and the target, respectively; the movement direction of the monitor should be maintained between the maximum and minimum angles. These represent the movement directions of the monitor at two adjacent moments; considering the energy consumption of unmanned systems, the smaller the angle, the lower the reward. This is a coefficient that constrains the direction of motion; Step 3.3: Reward setting for the search phase; In the target search phase, to address the issue of the monitor losing the target, the reward for the monitor is increased to include penalties, and the monitor is encouraged to keep getting closer to the target, so that the search phase lasts as short as possible; Specifically as follows: in, This indicates the reward for the monitor during the search phase. For control factors related to target distance, ; For control factors related to penalty items, ; Step 4: Strategies to enhance target advantages; Record the target's strategies during the search phase; these strategies enable the target to escape the monitor. Therefore, these strategies are advantageous relative to the monitor. By enhancing the target's advantageous strategies, the monitor's strategies can be optimized. Step 5: Optimize the target monitoring process; During the target monitoring phase, an approximate solution to the Nash equilibrium is obtained through reinforcement learning algorithms to realize the target monitoring process. In long-term monitoring tasks, the monitoring and search processes will alternate. By increasing the diversity of target strategies, the monitoring phase will be gradually extended and the search phase will be gradually shortened, thereby achieving better monitoring results. Step 6: Monitor the target according to the monitoring task.
2. The method for active target monitoring of an unmanned system based on two-party game theory as described in claim 1, characterized in that, In step 1, based on a partially observed Markov process non-zero-sum game, the game is represented as follows: The actions of the monitor are represented as The reward is represented as The strategy is represented as The target's action is represented as The reward is represented as The strategy is represented as It is a state space. Represent the action space of the monitor and the target, respectively. It is a state transition function. These represent the observation spaces of the monitor and the target, respectively. express The state at any given moment, It is a discount factor. They represent the monitor and the target respectively. Momentary actions These represent the strategies of the monitor and the target, respectively. They represent the monitor and the target respectively. The space of observation at any given moment.