Intelligent agent path real-time decision-making method facing dynamic threat and local perception

Abstract

The invention provides a dynamic threat and local perception-oriented agent path real-time decision-making method, and belongs to the technical field of agent path planning, and the method comprises the steps: collecting own state data and local obstacle perception data in an environment, carrying out the preprocessing, splicing the preprocessed data to obtain overall feature information, and carrying out the real-time decision-making of an agent path. And after being converted into a 27-dimensional Q value, the 27-dimensional Q value is input into the current Q network to obtain an optimal action, and then the optimal action is executed. The current Q network is obtained by updating a historical Q network by using historical experience data, and in the updating process, the Q network is guided to select actions in the direction of reward increase by adopting a multi-dimensional dense reward function. Based on a path planning thought of multi-source data of local situation awareness, an intelligent agent is allowed to dynamically adjust a path according to a real-time environment, the adaptive capacity of the intelligent agent to a dynamic complex scene is enhanced, a dense reward mechanism can stimulate the intelligent agent to avoid obstacles in a finer mode, and the intelligent agent rewarding efficiency is improved. And the flexibility and the accuracy of intelligent agent obstacle avoidance are obviously improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent agent path planning, and specifically relates to a real-time decision-making method for intelligent agent paths oriented to dynamic threats and local perception. Background Art

[0002] Traditional agent path planning methods mostly rely on classic path search and optimization algorithms, such as Algorithms such as the Dijkstra algorithm and other graph-search-based strategies perform well in known, static environments, but have limitations when dealing with complex, dynamic ones. Traditional intelligent agent path planning, typically based on global environmental information, cannot adapt to dynamic changes in the real-time environment during flight and lacks the ability to self-adjust and learn to optimize based on the actual environment. To alleviate this problem, researchers have introduced deep reinforcement learning methods.

[0003] Deep reinforcement learning is a machine learning method that learns optimal strategies by interacting with the environment. The intelligent agent learns to make behavioral decisions that maximize cumulative rewards through continuous trial and error. However, for complex intelligent agent path planning problems, the environmental space is large and complex. Therefore, researchers introduced deep reinforcement learning methods to improve the adaptability and robustness of intelligent agent path planning, applying DQN (deep Q network), PER-D3QN (oppositional dual deep Q network), and PSO-PRM (proximal policy optimization algorithm).

[0004] In document 1 (Li Yanru, Zuo Tiedong, Wang Jing. Research on intelligent path planning method of intelligent agent based on DQN deep reinforcement learning [J]. Electronic Technology and Software Engineering, 2022, (18): 5-8.), the authors applied a deep reinforcement learning network based on the PPO algorithm to construct a functional module, and trained DQN deep reinforcement learning with the support of the PPO algorithm. Combined with multiple iterative updates to optimize the strategy parameters, the optimal strategy was finally output to realize intelligent trajectory planning of the intelligent agent based on DQN deep reinforcement learning. In reference 2 (Bi Wenhao, Duan Xiaobo. Research on agent path planning algorithm based on deep reinforcement learning [J]. Aeronautical Science and Technology, 2023, 34(12): 118-124. DOI: 10.19452 / j.issn1007-5453.2023.12.014.), the authors introduced the competitive network model into the DQN algorithm to improve the convergence effect and stability of the algorithm while combining Double DQN (dual deep Q network) and the deep reinforcement learning algorithm of the priority reproduction strategy to construct PER-D3QN, so that the agent can effectively avoid obstacles according to the current battlefield status and realize global path planning. In document 3 (Zhao Dongmei. Research on Intelligent Agent Path Planning Technology [D]. University of Electronic Science and Technology of China, 2012), the authors proposed an improved PRM algorithm for heuristic node enhancement to make the trajectory route smoother; for online trajectory planning, they proposed a PSO-PRM algorithm, using the PSO algorithm for global trajectory planning and the PRM algorithm for local trajectory planning. At the same time, they proposed a trajectory optimization method based on the general graph method to obtain the shortest smooth path from the starting point to the end point while satisfying the performance constraints of the intelligent agent.

[0005] Many traditional methods for intelligent agent path planning assume a static environment and rely on pre-known global environmental information for planning. This limits the agent's ability to adapt to complex and dynamic environments, making it difficult to effectively adjust its path in real time and achieve robust path planning. Second, existing classic path planning algorithms are computationally intensive in complex environments and often fail to fully utilize local perception information in dynamic environments, resulting in a lack of comprehensive understanding of the real-time situation. Path selection based on global perception information is inefficient and time-consuming, making it difficult to meet real-time requirements. Third, a single data source often only captures specific types of information, resulting in an incomplete perception of the environment, low accuracy and robustness, and prone to blind spots in path planning systems, significantly reducing reliability. Fourth, most path planning algorithms employ sparse obstacle avoidance rewards, requiring extensive exploration to find an effective obstacle avoidance strategy. They are prone to becoming stuck in a suboptimal strategy and struggling to learn effective and complex obstacle avoidance strategies. Summary of the Invention

[0006] To address the aforementioned issues in the prior art, the present invention provides a real-time agent path decision-making method for dynamic threats and local perception. The technical problem to be solved by the present invention is achieved through the following technical solutions: A real-time agent path decision method for dynamic threats and local perception is applied to an agent. The real-time agent path decision method for dynamic threats and local perception includes: S100, collecting the current state data of the vehicle and the local obstacle perception data in the environment; S200, preprocessing the local obstacle perception data and the self-state data, and then splicing the preprocessed data to obtain overall feature information at the current time; S300, converting the overall feature information of the current time into a 27-dimensional Q value, and inputting the 27-dimensional Q value into the current Q network, so that the current Q network selects the best action from the experience pool; the current Q network uses the historical experience data before the current time in the experience pool to update the historical Q network, and during the update process, a dense reward function of multiple dimensions is used to guide the Q network to select actions in the direction of increasing reward; S400: Execute the optimal action.

[0007] Beneficial effects: The present invention provides a real-time decision-making method for intelligent agent paths oriented to dynamic threats and local perception. The method is applied to an intelligent agent and includes: collecting the agent's own state data at the current time and local obstacle perception data in the environment; preprocessing the local obstacle perception data and the agent's own state data, and then splicing the preprocessed data to obtain overall feature information at the current time; converting the overall feature information at the current time into a 27-dimensional Q value, and inputting the 27-dimensional Q value into a current Q network so that the current Q network selects the best action from an experience pool; the current Q network uses the experience data before the current time in the experience pool to update a historical Q network, and during the update process, uses a multi-dimensional dense reward function to guide the Q network to select actions in the direction of increasing rewards, and then executes the best action. The present invention is based on the path planning idea of ​​multi-source data of local situational awareness, allowing the intelligent agent to dynamically adjust the path at any time according to the real-time environment, thereby enhancing the intelligent agent's adaptability to dynamic and complex scenes; and the intensive reward mechanism designed by the present invention can motivate the intelligent agent to avoid obstacles in a more refined way, provide more feedback signals for the intelligent agent, accelerate the intelligent agent's learning process, significantly improve the flexibility and accuracy of the intelligent agent's obstacle avoidance, help it to collaboratively optimize multiple task objectives, and enable the intelligent agent to optimize and complete multiple task requirements at the same time.

[0008] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 This is a flow chart of a real-time decision-making method for intelligent agent paths oriented to dynamic threats and local perception provided by the present invention; Figure 2 Schematic diagram of the process of S200 pretreatment provided by the present invention; Figure 3 It is a schematic diagram of the structure of the model used by the two data sources provided by the present invention; Figure 4 It is an overall schematic diagram of the updated Q network provided by the present invention; Figure 5 It is a schematic diagram of the process of updating the Q network provided by the present invention; Figure 6 This is a visual schematic diagram of the UAV route setting provided by the present invention; Figure 7 It is a score curve graph of each batch of drones under training provided by the present invention; Figure 8 This is a curve chart of the average scores of the UAVs under training provided by the present invention. DETAILED DESCRIPTION

[0010] The present invention will be further described in detail below with reference to specific examples, but the embodiments of the present invention are not limited thereto.

[0011] This invention provides a real-time agent path decision-making method for dynamic threats and local perception. This method is applied to an agent, which can be a drone, a robot, or any other electronic device with agent functionality, without limitation. To facilitate the description of the specific details in this application, this application uses a drone as an example to describe the solution details.

[0012] Combine Figures 1 to 4 The present invention provides a real-time decision-making method for intelligent agent paths facing dynamic threats and local perception, including: S100, collecting the current state data of the vehicle and the local obstacle perception data in the environment; The self-state data includes: the three-dimensional position coordinates of the intelligent body, the destination coordinates, the Manhattan distance between the intelligent body and the destination at the current time, the Manhattan distance between the initial intelligent body and the destination, the step length of the intelligent body and the horizontal movement direction of the intelligent body; the local obstacle perception data includes: the grid where the obstacle is located and the three-dimensional coordinates of the corresponding grid.

[0013] Compared with the current more mainstream global path planning of intelligent agents, this application focuses on the local real-time environmental information perceived by the intelligent agent. The current traditional global path planning technology often assumes that the environment is static or changes little. When the environment changes, the traditional global planning needs to completely re-plan the path, which is less efficient. At the same time, global planning also requires environmental information of the entire flight area, which places high demands on computing resources and is not suitable for complex scenarios with frequent environmental changes. The path planning idea based on local situational awareness effectively avoids the disadvantages of low efficiency and high information dependence of global planning. The present application allows the intelligent agent to dynamically adjust the path at any time according to the real-time environment, effectively avoiding the situation of high computational complexity, slow path planning, and lack of flexibility in scenarios with dynamic high-density obstacle distribution.

[0014] S200, preprocessing the local obstacle perception data and the self-state data, and then splicing the preprocessed data to obtain overall feature information at the current time; In a specific embodiment of the present invention, S200 includes: S210, using an MLP (Multilayer Perceptron) to perform a primary process on the self-state data to obtain first intermediate data, and then performing a secondary process on the first intermediate data using the MLP to obtain self-state data features; The MLP consists of a 13-dimensional input layer, a fully connected layer, and a 13-dimensional output layer, with a PReLU function introduced between the fully connected layer and the output layer. Figure 3 As shown, Figure 3 The local representation model in is the MLP of this application, which includes an input layer, a fully connected layer, and an output layer.

[0015] refer to Figure 2 As shown in the figure, the self-state data (uav_state) includes the three-dimensional coordinate position of the drone (x, y, z), the coordinates of the destination (target[0], target[1], target[2]), the three-dimensional directional distance difference between the drone and the destination (dx, dy, dz), the Manhattan distance between the current drone and the destination (distance), the Manhattan distance between the initial drone and the destination (d_origin), the 13-dimensional data vector x1 of the drone's step length (step) and the drone's horizontal movement direction (dir). It passes through the 13-dimensional input layer, then passes through the fully connected layer and introduces the PReLU function to process the self-state data. In this way, the self-state data features can be obtained twice, which is expressed as x1_end.

[0016] S220, reshaping the local obstacle perception data into a vector of a predetermined shape, performing a primary processing on the vector of the predetermined shape using a CNN (Convolutional Neural Network) to obtain second intermediate data; and performing a secondary processing on the second intermediate data using the CNN to obtain local obstacle information features; Combine Figure 2 and Figure 3 The present invention can collect the 125-dimensional grid obstacle information list (ob_space) of the agent's surrounding environment data, reshape it into a 5×5×5 vector x2, pass it through a convolution layer with a convolution kernel size of 3×3 and a step size of 1, and then introduce the PRelu activation function. After two processes, the local obstacle information feature can be obtained, which is expressed as x2_end. Figure 3 The obstacle representation model in

[15] is a CNN model, which includes input layer, convolution layer, activation function, pooling layer, fully connected layer and output layer.

[0017] This application collects multi-source data. Path planning based on multi-source data not only increases the agent's perception dimension of the surrounding environment, but also provides richer decision-making basis. Different from the current feature extraction ideas used to process target coordinates and agent coordinate information, this application combines the Manhattan distance (distance) between targets to accurately adjust the flight direction, combines the step length (step) and flight direction (dir) to help optimize the flight path, improve flight efficiency and avoid invalid flight, and combines grid obstacle information (ob_space) to provide local environmental information, timely feedback on environmental changes, enhance the agent's adaptability to dynamic and complex scenes, and ensure the safe and efficient execution of flight missions.

[0018] S230: Flatten the local obstacle information features, and then combine the flattened local obstacle information features with the self-state data features to obtain overall feature information.

[0019] In this step, x1_end and the flattened x2_end are concatenated along the first channel dimension to form the overall feature information out, which is then converted into a 27-dimensional Q estimate, as shown in Figure 2 out[0]-out

[26] shown.

[0020] It is worth noting that in the initial training phase, the state information of the agent itself and the training environment need to be initialized. By randomly initializing the agent parameters and the environment state parameters and setting the training difficulty at the same time, the agent can be prevented from converging to an undesirable strategy too early in the early stage.

[0021] The agent is a drone, and the virtual environment for drone training is 100 meters long, 100 meters wide, and 22 meters high. The number of drones in each training environment is set to 15, and the initial training difficulty level is 1. The drone training environment is randomly reset to an initial state, and multiple buildings are randomly generated. The building coordinates are randomly generated in the range (10-90, 10-90, 0), with a length of 1-10 meters, a width of 1-10 meters, and a height of 9-13 meters. The building locations are marked on the map. The destination is randomly generated in an obstacle-free area of ​​(60-90, 10-90, 3-15 meters), and 15 drones are generated in the range (60-90, 10-90, 3-15 meters).

[0022] Reset the drone status, including the drone's 3D coordinate position (x, y, z), the destination coordinates (target[0], target[1], target[2]), the distance difference between the drone and the destination in the 3D direction (dx, dy, dz), the Manhattan distance between the current drone and the destination (distance), the initial Manhattan distance between the drone and the destination (d_origin), the drone's step length (step) and the drone's horizontal movement direction (dir); at the same time, record the 5×5×5 grid obstacle information list centered on the drone (ob_space).

[0023] The present invention defines two Q networks: q_local and q_target. q_local represents the local network, which is used to estimate the Q value for the drone to select the appropriate action. q_target represents the target network, which is used to update parameters and generate target Q values ​​during the training process of the Q network.

[0024] Select mean square error as the loss function and define The difference between the current network Q value and the target Q value is calculated as the mean square error and used to optimize the local network and the target network. The loss function is expressed as: ; Where, is the loss function, is the Q value calculated by the target network, It is the Q value calculated using its own state data characteristics. yes or the total number of ; ; Where, is the discount factor, is the Q value at the next time point, Indicates whether the agent successfully reaches the destination. is 1, otherwise it is 0. Indicates reward and is also an intermediate parameter used for transfer. Indicates that the value ranges from 1 to .

[0025] The present invention uses the Adam optimizer as the optimizer for gradient descent optimization to optimize the parameters of the local network and specifies the learning rate lr as 0.001.

[0026] Initialize an experience replay pool with a capacity of 10,000, specify the dimension of the agent's state space in the environment to be 138, and the dimension of its action space to be 27.

[0027] In an optional embodiment of the present invention, combined with Figure 4 and Figure 5 The process of updating the historical Q network using the historical experience data in the experience pool includes: a. When the amount of historical experience data in the experience pool exceeds the threshold, the experience data at the earliest time point is removed and the experience data at the latest time point is stored; b. randomly sampling in the experience pool to obtain training data; c. For each agent, the training data is input into the historical local network carried by the agent, so that the historical local network uses the ε-greedy strategy to select an action from all actions stored in the experience pool, and the state of the agent at the next time point corresponding to the action is fed back to the historical target network; Combine Figure 4 and Figure 5 In this application, after the initial state of the agent and the environment is given, the agent selects an action based on the current strategy. The experiment uses the ε-greedy strategy with ε=0.1 combined with exploration and exploitation to select the action of the agent. After each step of the agent's action is completed, the current state of the drone is updated in real time. If the agent decides to exploit, the optimal action is selected based on the current state and the trained Q network. If the agent decides to explore, an action is randomly selected from the 27-dimensional action space. , the mathematical formula for selecting actions by the ε-greedy strategy is as follows: ; in, is the action dimension of the agent 27, The greedy probability is 0.01, is the best action chosen by the agent, Is the state-action value function, which means that in a given state Next, select Action The expected total reward reward, Is the policy function, which means that in state Select Action The probability of is the current state, which describes the agent's position or situation in the environment.

[0028] d. The historical target network uses the state of the agent at the next time point to calculate a Q value for updating the historical local network and the historical target network; wherein the Q value for updating the historical local network and the historical target network is calculated using the density reward function of the multiple dimensions; The design of the reward function is crucial in reinforcement learning. It will directly affect how the intelligent agent learns and makes decisions. Constructing a dense reward function from multiple dimensions, achieving a transition from sparse to dense reward functions, and avoiding the long-term lack of feedback caused by sparse rewards are important goals of this invention.

[0029] This application designs a multi-dimensional intensive reward function, including: giving the agent a climbing reward by calculating the distance between the agent and the nearest obstacle, giving the agent a target reward by calculating the agent's z-axis coordinate and the destination's z-axis coordinate, and giving the agent an obstacle avoidance reward by calculating the distance and distance change between the initial state and the current state. When training and updating the agent, the sum of the agent's climbing reward, goal reward, and obstacle avoidance reward is designed as the agent's intensive reward. Initially, the intensive reward is set to 0. The reward design is as follows: First, update the drone state and calculate the change in the drone coordinates. If the drone is stationary, give it a large penalty reward = reward-1000. Traverse the list of grid information (ob_space) around the drone and calculate and compare the distance between the drone and the nearest obstacle. , giving the drone obstacle avoidance rewards , the obstacle avoidance reward is expressed as: ; Where, is the obstacle avoidance reward, is the distance between the drone and the nearest obstacle; Comparing current drones Axis coordinates With the destination Axis coordinates , giving the drone a climbing bonus , the climbing reward is expressed as: ; Where, It's a climbing reward. It is an intelligent entity Axis coordinates, The destination is axis coordinates; Calculate the distance between the initial state drone and the destination , the current state of the drone is the distance from the destination Change in distance , giving drone target rewards , the target reward is expressed as: ; Where, is the target reward, is the distance change, is the distance between the agent and the destination in the initial state, is the distance between the agent and the destination in the current state.

[0030] Sparse rewards are goal-oriented, and rewards are also set for drones when they encounter certain important events. A collision results in a large penalty of reward = reward - 200. Upon reaching the destination, a large reward of reward = reward + 200 is given. If the maximum number of steps is exceeded, the drone is penalized with reward = reward - 20.

[0031] Common agent path planning methods using traditional algorithms, such as Algorithms, Dijkstra algorithms, and other graph-search-based strategies lack adaptive learning capabilities after initial planning and cannot be optimized based on flight experience. Current common path planning techniques that incorporate deep reinforcement learning methods, such as DQN (Deep Q Network), PER-D3QN (Persistent Dual Deep Q Network), and PSO-PRM (Proximal Policy Optimization), are often limited to providing rewards only for encountering specific significant events or for searching for target paths, resulting in poor adaptability to complex obstacle scenarios. The dense reward mechanism designed in this application can incentivize the agent to avoid obstacles in a more refined manner. Compared to traditional sparse obstacle avoidance strategies that only penalize collisions with obstacles, this reward mechanism can provide the agent with more feedback signals, accelerate the agent's learning process, significantly improve the agent's flexibility and accuracy in obstacle avoidance, and help it collaboratively optimize multiple mission objectives (obstacle avoidance, path optimization, flight stability, etc.), enabling the agent to simultaneously optimize and complete multiple mission requirements.

[0032] e. Reversely update the historical local network and the historical target network using the Q value in d; refer to Figure 4In this step, the difference between the Q value in d and the target Q value is used to calculate the mean square error; the mean square error is used as the loss function, and the loss function is used to reversely update and optimize the historical local network and the historical target network.

[0033] f. Repeat the process of ae until the probability of the agent's success is no less than 80%, where the agent's success refers to successfully reaching the destination within the specified number of steps.

[0034] In this application, each complete training episode of the drone uses an ε-greedy strategy with ε=0.1 to train 15 agents in the initialized environment. If the Q network has been trained before, the previously trained Q network parameters are loaded for training, and then the previous number of training cycles are trained, otherwise training starts from the beginning. Within the specified number of training cycles, the drone continuously selects actions according to the strategy Q value, changes its own state and obtains benefits, and stores its interaction experience. Each time the agent interacts with the environment, it generates an experience tuple (Transition) containing five fields: agent state (state), action (action), reward (reward), next state (next_state) and whether it is completed (done), and stores it in the experience pool. After every five drones complete training, a check is conducted. If the amount of experience data in the experience pool exceeds BATCH_SIZE, the Q network is trained. If the success rate of the drone exceeds 80%, the training difficulty (level) is increased to 10.

[0035] After every 100 episodes of training, the parameters of the q_target and q_local networks, the optimizer state, and the current training cycle number are saved locally for reading network parameters in the next training.

[0036] S300, converting the overall feature information of the current time into a 27-dimensional Q value, and inputting the 27-dimensional Q value into the current Q network, so that the current Q network selects the best action from the experience pool; the current Q network uses the historical experience data before the current time in the experience pool to update the historical Q network, and during the update process, a dense reward function of multiple dimensions is used to guide the Q network to select actions in the direction of increasing reward; The historical Q network is a Q network updated at a previous time point. The Q network includes a local network and a target network. The local network is used to input a 27-dimensional Q value and output a selected action. The target network is used to input a selected action and output a Q value.

[0037] The experience pool stores experience tuples including the agent state, action, reward, the agent state at the next time point, and whether the destination has been reached at each historical time point.

[0038] S400: Execute the optimal action.

[0039] In an optional embodiment of the present invention, S300 includes: S310, inputting the overall feature information of the current time into an adaptive MLP for processing to obtain a 27-dimensional Q value; In this step, the overall feature information out of the drone is input into a fully connected layer with a dimension of 338, and the PReLU function is referenced to output a 27-dimensional Q value estimate.

[0040] S320: Input the 27-dimensional Q value in S310 into the current local network, so that the current local network selects the action with the largest Q value from all actions stored in the experience pool as the best action.

[0041] In order to verify the advantages of the present invention, the present invention designed a visibility test verification project for UAV route planning, and intercepted the results of a certain UAV training such as Figure 6 As shown in the figure, the drone successfully reached its destination under the complex environment conditions with a training difficulty level of 10. At the same time, a verification algorithm for the drone’s route planning evaluation was designed to visually display the results of each scene’s training, including the drone’s score (Score) for each scene. Figure 7 The average score (Avg_Score) of the drone as of the current training batch is as follows Figure 8 .exist Figure 7 The horizontal axis is the training batch Epoch, and the vertical axis is the training score of each batch (EveryEpoch_Score). Figure 8 The horizontal axis is the training batch Epoch, and the vertical axis is the score. Due to the randomness of the environment, the randomness of the exploration strategy and the imbalance of the environment difficulty in reinforcement learning, according to Figure 7 It can be seen that the drone is not converging steadily, but Figure 8 It shows that with continuous training, the overall training effect of the drone is gradually optimized, and its average score (Avg_Score) is gradually increasing and stabilizing in an excellent training state.

[0042] It is worth noting that the terms "first" and "second" in this disclosure are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of the technical features indicated. Therefore, features defined as "first" or "second" may explicitly or implicitly include one or more of such features. In the description of this disclosure, "plurality" means two or more, unless otherwise specifically defined.

[0043] Although the present application is described herein with reference to various embodiments, those skilled in the art will be able to understand and implement other variations of the disclosed embodiments in practicing the claimed application by reviewing the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude a plurality.

[0044] The above is a further detailed description of the present invention in conjunction with specific preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. For those skilled in the art of the present invention, without departing from the concept of the present invention, several simple deductions or substitutions can be made, which should be considered to fall within the scope of protection of the present invention.

Claims

1. A real-time decision-making method for intelligent agent paths facing dynamic threats and local perception, characterized in that: Applied to an intelligent agent, the intelligent agent path real-time decision method for dynamic threats and local perception includes: S100, collecting the current state data of the vehicle and the local obstacle perception data in the environment; S200, preprocessing the local obstacle perception data and the self-state data, and then splicing the preprocessed data to obtain the overall feature information at the current time; S300, converting the overall feature information of the current time into a 27-dimensional Q value, and inputting the 27-dimensional Q value into the current Q network, so that the current Q network selects the best action from the experience pool; the current Q network updates the historical Q network using the historical experience data before the current time in the experience pool, and uses a dense reward function of multiple dimensions to guide the Q network to select an action in the direction of increasing the reward during the update process; S400: Execute the optimal action.

2. The real-time decision-making method for intelligent agent paths facing dynamic threats and local perception according to claim 1 is characterized in that: The self-state data includes: the three-dimensional position coordinates of the intelligent agent, the destination coordinates, the Manhattan distance between the intelligent agent and the destination at the current time, the Manhattan distance between the initial intelligent agent and the destination, the step length of the intelligent agent and the horizontal movement direction of the intelligent agent; the local obstacle perception data includes: the grid where the obstacle is located and the corresponding three-dimensional coordinates of the grid.

3. The real-time decision-making method for intelligent agent paths facing dynamic threats and local perception according to claim 2 is characterized in that: S200 includes: S210, using MLP to process the self-state data once to obtain first intermediate data, and then performing secondary processing on the first intermediate data through MLP to obtain self-state data features; Among them, the MLP includes a 13-dimensional input layer, a fully connected layer and a 13-dimensional output layer, and a PReLU function is introduced between the fully connected layer and the output layer; S220, reshaping the local obstacle perception data into a vector of a predetermined shape, and then using CNN to perform a primary processing on the vector of the predetermined shape to obtain second intermediate data; and then performing a secondary processing on the second intermediate data through CNN to obtain local obstacle information features; S230, flattening the local obstacle information features, and then concatenating the flattened local obstacle information features with the self-state data features to obtain overall feature information.

4. The real-time decision-making method for intelligent agent paths facing dynamic threats and local perception according to claim 1 is characterized in that: The historical Q network is a Q network updated at a previous time point, and the Q network includes a local network and a target network. The local network is used to input a 27-dimensional Q value and output a selected action; the target network is used to input a selected action and output a Q value.

5. The real-time decision-making method for intelligent agent paths facing dynamic threats and local perception according to claim 4 is characterized in that: S300 includes: S310, inputting the overall feature information of the current time into an adaptive MLP for processing to obtain a 27-dimensional Q value; S320, inputting the 27-dimensional Q value in S310 into the current local network, so that the current local network selects the action with the largest Q value from all actions stored in the experience pool as the best action.

6. The real-time decision-making method for intelligent agent paths facing dynamic threats and local perception according to claim 4 is characterized in that: The experience pool stores experience tuples including the agent state, action, reward at each historical time point, the agent state at the next time point, and whether the destination has been reached.

7. The real-time decision-making method for intelligent agent paths facing dynamic threats and local perception according to claim 4 is characterized in that: The process of updating the historical Q network using the historical experience data in the experience pool includes: a. When the amount of historical experience data in the experience pool exceeds the quantity threshold, the experience data at the earliest time point is removed and the experience data at the latest time point is stored; b. randomly sampling in the experience pool to obtain training data; c. For each agent, the training data is input into the historical local network carried by the agent, so that the historical local network selects an action from all actions stored in the experience pool using the ε-greedy strategy, and feeds back the state of the agent at the next time point corresponding to the action to the historical target network; d. The historical target network uses the state of the agent at the next time point to calculate the Q value for updating the historical local network and the historical target network; wherein the Q value for updating the historical local network and the historical target network is calculated using the density reward function of the multiple dimensions; e. Reversely update the historical local network and the historical target network using the Q value in d; f. Repeat the process of ae until the probability of success of the agent is not less than 80%, where success of the agent means successfully reaching the destination within the specified number of steps.

8. The real-time decision-making method for intelligent agent paths facing dynamic threats and local perception according to claim 7 is characterized in that: The intensive reward function of the multiple dimensions includes: giving an obstacle avoidance reward to the agent by calculating the distance between the agent and the nearest obstacle, giving a climbing reward to the agent by calculating the coordinate of the agent on the z-axis and the coordinate of the destination on the z-axis, and giving a target reward to the agent by calculating the distance and distance change between the agent and the destination in the initial state and the current state; Wherein, the obstacle avoidance reward is expressed as: ； In the formula, is the obstacle avoidance reward, is the distance between the drone and the nearest obstacle; The climbing reward is expressed as: ； In the formula, It's a reward for climbing. It is the intelligent agent in The coordinates of the axes, The destination is The coordinates of the axes; The target reward is expressed as: ； In the formula, is the target reward, is the distance change, is the distance from the agent to the destination in the initial state, is the distance between the agent and the destination in the current state.

9. The real-time decision-making method for intelligent agent paths facing dynamic threats and local perception according to claim 8 is characterized in that include: Using the difference between the Q value in d and the target Q value, calculate the mean square error; The mean square error is used as a loss function, and the loss function is used to reversely update and optimize the historical local network and the historical target network.

10. The real-time decision-making method for intelligent agent paths facing dynamic threats and local perception according to claim 9 is characterized in that: The loss function is expressed as: ； In the formula, is the loss function, is the Q value calculated by the target network, It is the Q value calculated using the characteristics of its own state data. yes or Total number of ； ； In the formula, is the discount factor, is the Q value at the next time point, Indicates whether the agent successfully reaches the destination. is 1, otherwise it is 0. Indicates reward and is also an intermediate parameter used for transfer. Indicates that the value ranges from 1 to .

Images