A real-time decision-making method for intelligent agent paths facing dynamic threats and local perception

Abstract

The present invention provides a real-time decision-making method for intelligent agent paths facing dynamic threats and local perception, and the technical field of intelligent agent path planning, including collecting self-state data and local obstacle perception data in the environment, and then preprocessing, and then splicing the preprocessed data to obtain overall feature information, converting it into a 27-dimensional Q value and inputting it into the current Q network to obtain the best action, and then executing the best action. The current Q network updates the historical Q network using historical experience data, and in the updating process, a dense reward function of multiple dimensions is used to guide the Q network to select actions in the direction of increasing rewards. The present invention is based on the path planning idea of ​​multi-source data of local situational awareness, allowing the intelligent agent to dynamically adjust the path according to the real-time environment, enhancing the adaptability of the intelligent agent to dynamic and complex scenes, and the dense reward mechanism can motivate the intelligent agent to avoid obstacles in a more sophisticated way, significantly improving the flexibility and accuracy of the intelligent agent's obstacle avoidance.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent agent path planning, and specifically relates to a real-time decision-making method for intelligent agent paths facing dynamic threats and local perception. Background Art

[0002] Traditional intelligent agent path planning methods mostly rely on classic path search and optimization algorithms, such as Algorithms, Dijkstra algorithms, and other graph search-based strategies, which perform well in known, static environments, but have certain limitations when dealing with complex, dynamic environments. Traditional intelligent agent path planning is usually based on global environmental information, which cannot adapt to the dynamic changes of the real-time environment during flight and lacks the ability to self-adjust and learn and optimize according to the actual environment. In order to alleviate this problem, researchers introduced deep reinforcement learning methods.

[0003] Deep reinforcement learning is a machine learning method that learns optimal strategies by interacting with the environment. The intelligent agent learns to make behavioral decisions that can maximize cumulative rewards through continuous trial and error. However, for complex intelligent agent path planning problems, the environmental space is large and complex. Therefore, researchers introduced deep reinforcement learning methods to improve the adaptability and robustness of the intelligent agent's path planning, and applied DQN (deep Q network), PER-D3QN (oppositional dual deep Q network), and PSO-PRM (proximal policy optimization algorithm).

[0004] In document 1 (Li Yanru, Zuo Tiedong, Wang Jing. Research on intelligent path planning method of intelligent agent based on DQN deep reinforcement learning [J]. Electronic Technology and Software Engineering, 2022, (18): 5-8.), the authors applied a deep reinforcement learning network based on the PPO algorithm to construct functional modules, and trained DQN deep reinforcement learning with the support of the PPO algorithm, combined with multiple iterative updates to optimize the strategy parameters, and finally output the optimal strategy to realize intelligent trajectory planning of intelligent agents based on DQN deep reinforcement learning. In reference 2 (Bi Wenhao, Duan Xiaobo. Research on agent path planning algorithm based on deep reinforcement learning [J]. Aeronautical Science and Technology, 2023, 34(12): 118-124. DOI: 10.19452 / j.issn1007-5453.2023.12.014.), the authors introduced the competitive network model into the DQN algorithm to improve the convergence effect and stability of the algorithm, while combining Double DQN (dual deep Q network) and the deep reinforcement learning algorithm of the priority reproduction strategy to construct PER-D3QN, so that the agent can effectively avoid obstacles according to the current battlefield status and realize global path planning. In document 3 (Zhao Dongmei. Research on Intelligent Agent Path Planning Technology [D]. University of Electronic Science and Technology of China, 2012), the authors proposed an improved PRM algorithm for heuristic node enhancement to make the trajectory route smoother; for online trajectory planning, they proposed a PSO-PRM algorithm, using the PSO algorithm for global trajectory planning and the PRM algorithm for local trajectory planning. At the same time, they proposed a trajectory optimization method based on the through-graph method to obtain the shortest smooth path from the starting point to the end point while satisfying the performance constraints of the intelligent agent.

[0005] Many traditional methods of current intelligent agent path planning assume that the environment is static and rely on pre-known global environmental information for planning, which makes the intelligent agent's ability to respond to situations limited when encountering complex and dynamic environments, and it is difficult to effectively adjust the path in real time to achieve robust path planning. Second, the existing classic path planning algorithms have a large amount of computation in complex environments, and often cannot fully utilize local perception information in dynamic environments, resulting in a lack of full understanding of the real-time situation in path planning. The global perception information is inefficient in selecting paths and takes a long time, making it difficult to meet real-time requirements. Third, a single data source can often only obtain specific types of information, resulting in incomplete environmental perception, low accuracy and robustness, and the path planning system is prone to perception blind spots, greatly reducing reliability. Fourth, most path planning algorithms use sparse obstacle avoidance rewards, which require a lot of exploration to find an effective obstacle avoidance strategy, and are prone to falling into a suboptimal obstacle avoidance strategy and staying, making it difficult to learn an effective and complex obstacle avoidance strategy. Summary of the invention

[0006] In order to solve the above problems existing in the prior art, the present invention provides a real-time decision-making method for intelligent agent paths facing dynamic threats and local perception. The technical problem to be solved by the present invention is achieved through the following technical solutions:

[0007] A real-time decision method for intelligent agent paths facing dynamic threats and local perception is applied to an intelligent agent. The real-time decision method for intelligent agent paths facing dynamic threats and local perception includes:

[0008] S100, collecting the current state data of the vehicle and the local obstacle perception data in the environment;

[0009] S200, preprocessing the local obstacle perception data and the self-state data, and then splicing the preprocessed data to obtain the overall feature information at the current time;

[0010] S300, converting the overall feature information of the current time into a 27-dimensional Q value, and inputting the 27-dimensional Q value into the current Q network, so that the current Q network selects the best action from the experience pool; the current Q network updates the historical Q network using the historical experience data before the current time in the experience pool, and uses a dense reward function of multiple dimensions to guide the Q network to select an action in the direction of increasing the reward during the update process;

[0011] S400: Execute the optimal action.

[0012] Beneficial effects:

[0013] The present invention provides a real-time decision-making method for intelligent agent paths facing dynamic threats and local perception, which is applied to an intelligent agent. The method comprises: collecting the current time's own state data and the local obstacle perception data in the environment; preprocessing the local obstacle perception data and the self-state data, and then splicing the preprocessed data to obtain the overall feature information at the current time; converting the overall feature information at the current time into a 27-dimensional Q value, and inputting the 27-dimensional Q value into the current Q network, so that the current Q network selects the best action from the experience pool; the current Q network updates the historical Q network using the experience data before the current time in the experience pool, and in the updating process, uses a multi-dimensional dense reward function to guide the Q network to select an action in the direction of increasing the reward, and then executes the best action. The path planning idea of ​​the present invention is based on multi-source data of local situational awareness, which allows the intelligent agent to dynamically adjust the path according to the real-time environment at any time, thereby enhancing the adaptability of the intelligent agent to dynamic and complex scenes; and the intensive reward mechanism designed by the present invention can motivate the intelligent agent to avoid obstacles in a more sophisticated way, provide more feedback signals for the intelligent agent, accelerate the learning process of the intelligent agent, significantly improve the flexibility and accuracy of the intelligent agent's obstacle avoidance, help it to coordinately optimize multiple task objectives, and enable the intelligent agent to optimize and complete multiple task requirements at the same time.

[0014] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a flow chart of a real-time decision-making method for intelligent agent paths facing dynamic threats and local perception provided by the present invention;

[0016] Figure 2 It is a schematic diagram of the process of S200 pretreatment provided by the present invention;

[0017] Figure 3 It is a schematic diagram of the structure of the model used by the two data sources provided by the present invention;

[0018] Figure 4 It is an overall schematic diagram of the updated Q network provided by the present invention;

[0019] Figure 5 It is a schematic diagram of the process of updating the Q network provided by the present invention;

[0020] Figure 6 It is a visualized schematic diagram of the UAV route setting provided by the present invention;

[0021] Figure 7 It is a score curve graph of each batch of drones under training provided by the present invention;

[0022] Figure 8 It is a curve chart of average scores of the UAV under training provided by the present invention. DETAILED DESCRIPTION

[0023] The present invention is further described in detail below with reference to specific embodiments, but the embodiments of the present invention are not limited thereto.

[0024] The present invention provides a real-time decision-making method for intelligent agent paths facing dynamic threats and local perception, which is applied to an intelligent agent, which can be a drone, a robot, or any electronic device with intelligent agent functions, and the present invention is not limited here. To facilitate the description of the specific details in this application, this application takes a drone as an example to describe the details of the solution.

[0025] Combination Figures 1 to 4 The present invention provides a real-time decision-making method for intelligent agent paths facing dynamic threats and local perception, including:

[0026] S100, collecting the current state data of the vehicle and the local obstacle perception data in the environment;

[0027] The self-state data includes: the three-dimensional position coordinates of the intelligent agent, the destination coordinates, the Manhattan distance between the intelligent agent and the destination at the current time, the Manhattan distance between the initial intelligent agent and the destination, the step length of the intelligent agent and the horizontal movement direction of the intelligent agent; the local obstacle perception data includes: the grid where the obstacle is located and the corresponding three-dimensional coordinates of the grid.

[0028] Compared with the current mainstream global path planning of intelligent agents, this application focuses on the local real-time environmental information perceived by the intelligent agent. The current traditional global path planning technology often assumes that the environment is static or changes little. When the environment changes, the traditional global planning needs to completely re-plan the path, which is inefficient. At the same time, global planning also requires environmental information of the entire flight area, which has high requirements for computing resources and is not suitable for complex scenarios with frequent environmental changes. The path planning idea based on local situational awareness effectively avoids the disadvantages of low efficiency and high information dependence of global planning. The present application allows the intelligent agent to dynamically adjust the path at any time according to the real-time environment, effectively avoiding the high computational complexity, slow path planning, and lack of flexibility in scenarios with dynamic high-density obstacle distribution.

[0029] S200, preprocessing the local obstacle perception data and the self-state data, and then splicing the preprocessed data to obtain the overall feature information at the current time;

[0030] In a specific implementation of the present invention, S200 includes:

[0031] S210, using MLP (Multilayer Perceptron) to process the self-state data once to obtain first intermediate data, and then performing secondary processing on the first intermediate data through MLP to obtain self-state data features;

[0032] The MLP includes a 13-dimensional input layer, a fully connected layer, and a 13-dimensional output layer, and a PReLU function is introduced between the fully connected layer and the output layer. Figure 3 As shown, Figure 3 The local representation model in is the MLP of this application, which includes an input layer, a fully connected layer and an output layer.

[0033] refer to Figure 2As shown in the figure, the self-state data (uav_state) includes the three-dimensional coordinate position of the drone (x, y, z), the coordinates of the destination (target[0], target[1], target[2]), the three-dimensional distance difference between the drone and the destination (dx, dy, dz), the Manhattan distance between the current drone and the destination (distance), the Manhattan distance between the initial drone and the destination (d_origin), the 13-dimensional data vector x1 of the drone's step length (step) and the drone's horizontal movement direction (dir). It passes through the 13-dimensional input layer, then passes through the fully connected layer and introduces the PReLU function to process the self-state data. In this way, the self-state data features can be obtained twice, which is expressed as x1_end.

[0034] S220, reshaping the local obstacle perception data into a vector of a predetermined shape, and then using CNN (Convolutional Neural Networks, convolutional neural network) to process the vector of the predetermined shape once to obtain second intermediate data; and then performing secondary processing on the second intermediate data through CNN to obtain local obstacle information features;

[0035] Combination Figure 2 and Figure 3 , the present invention can collect the 125-dimensional grid obstacle information list (ob_space) of the agent's surrounding environment data, reshape it into a 5×5×5 vector x2, pass it through a convolution layer with a convolution kernel size of 3×3 and a step size of 1, and then introduce the PRelu activation function. After two processes, the local obstacle information feature can be obtained, which is expressed as x2_end. Figure 3 The obstacle representation model in is a CNN model, which includes an input layer, a convolutional layer, an activation function, a pooling layer, a fully connected layer, and an output layer.

[0036] This application collects multi-source data. Path planning based on multi-source data not only increases the agent's perception dimension of the surrounding environment, but also provides richer decision-making basis. Different from the current feature extraction ideas used to process target coordinates and agent coordinate information, this application combines the Manhattan distance (distance) between targets to accurately adjust the flight direction, combines the step length (step) and flight direction (dir) to help optimize the flight path, improve flight efficiency and avoid invalid flight, and combines grid obstacle information (ob_space) to provide local environmental information, timely feedback on environmental changes, enhance the agent's adaptability to dynamic and complex scenes, and ensure the safe and efficient execution of flight missions.

[0037] S230, flattening the local obstacle information features, and then concatenating the flattened local obstacle information features with the self-state data features to obtain overall feature information.

[0038] In this step, x1_end and the flattened x2_end are concatenated along the first channel dimension into the overall feature information out, which is then converted into a 27-dimensional Q estimate, as Figure 2 out[0]-out

[26] as shown.

[0039] It is worth noting that in the initial training phase, the state information of the agent itself and the training environment need to be initialized. By randomly initializing the agent parameters and the environment state parameters and setting the training difficulty at the same time, the agent can be prevented from converging to an undesirable strategy too early in the early stage.

[0040] The agent is a drone, and the virtual environment for drone training is 100 long, 100 wide, and 22 high. The number of drones in each training environment is set to 15, and the initial training difficulty level is 1. The drone training environment is randomly reset to an initial state, and multiple buildings are randomly generated. The building coordinates are randomly generated in the range of (10~90, 10~90, 0), the building length is 1~10, the width is 1~10, and the height is 9~13. The location of the building in the map is marked. The destination is randomly generated in the barrier-free area of ​​(60~90, 10~90, 3~15), and 15 drones are generated in the range of (60~90, 10~90, 3~15).

[0041] Reset the drone status, including the drone's 3D coordinate position (x, y, z), the destination coordinates (target[0], target[1], target[2]), the distance difference between the drone and the destination in the 3D direction (dx, dy, dz), the Manhattan distance between the current drone and the destination (distance), the Manhattan distance between the initial drone and the destination (d_origin), the drone's step length (step) and the drone's horizontal movement direction (dir); at the same time, record the 5×5×5 grid obstacle information list centered on the drone (ob_space).

[0042] The present invention defines two Q networks: q_local and q_target, q_local represents the local network, which is used to estimate the Q value for the drone to select a suitable action, and q_target represents the target network, which is used to update parameters and generate target Q values ​​during the training of the Q network.

[0043] Select mean square error as the loss function and define The difference between the current network Q value and the target Q value is calculated as the mean square error, which is used to optimize the local network and the target network. The loss function is expressed as:

[0044] ;

[0045] In the formula, is the loss function, is the Q value calculated by the target network, It is the Q value calculated using the characteristics of its own state data. yes or Total number of

[0046] ;

[0047] ;

[0048] In the formula, is the discount factor, is the Q value at the next time point, Indicates whether the agent successfully reaches the destination. is 1, otherwise it is 0. Indicates reward and is also an intermediate parameter used for transfer. Indicates that the value ranges from 1 to .

[0049] The present invention adopts Adam optimizer as the optimizer of gradient descent optimization to optimize the parameters of the local network and specifies the learning rate lr as 0.001.

[0050] Initialize an experience replay pool with a capacity of 10,000, and specify that the dimension of the agent's state space in the environment is 138 and the dimension of its action space is 27.

[0051] In an optional embodiment of the present invention, in combination with Figure 4 and Figure 5 ,The process of updating the historical Q network using the historical experience data in the experience pool includes:

[0052] a. When the amount of historical experience data in the experience pool exceeds the quantity threshold, the experience data at the earliest time point is removed and the experience data at the latest time point is stored;

[0053] b. randomly sampling in the experience pool to obtain training data;

[0054] c. For each agent, the training data is input into the historical local network carried by the agent, so that the historical local network selects an action from all actions stored in the experience pool using the ε-greedy strategy, and feeds back the state of the agent at the next time point corresponding to the action to the historical target network;

[0055] Combination Figure 4 and Figure 5After the initial state of the agent and the environment is given, the agent selects an action according to the current strategy. The experiment uses the ε-greedy strategy with ε=0.1 combined with exploration and utilization to select the action of the agent. After the action of each step of the agent is completed, the current state of the drone is updated in real time. If the agent decides to exploit, the optimal action is selected based on the current state and the trained Q network. If the agent decides to explore, the action is randomly selected from the 27-dimensional action space. , the mathematical formula for selecting actions by the ε-greedy strategy is as follows:

[0056] ;

[0057] in, is the action dimension of the agent 27, The greedy probability is 0.01, is the best action chosen by the agent, is the state-action value function, which means that in a given state Next, select Action The expected total reward when is the policy function, which means that in the state Next select action The probability of is the current state, describing the agent's position or situation in the environment.

[0058] d. The historical target network uses the state of the agent at the next time point to calculate the Q value for updating the historical local network and the historical target network; wherein the Q value for updating the historical local network and the historical target network is calculated using the density reward function of the multiple dimensions;

[0059] The design of the reward function is crucial in reinforcement learning, as it will directly affect how the agent learns to make decisions. It is an important goal of this invention to construct a dense reward function from multiple dimensions, realize the transition from sparse to dense reward functions, and avoid the problem of long-term lack of feedback caused by sparse rewards.

[0060] The present application designs a multi-dimensional intensive reward function including: giving the agent a climbing reward by calculating the distance between the agent and the nearest obstacle, giving the agent a target reward by calculating the agent's coordinates on the z-axis and the destination's coordinates on the z-axis, and giving the agent an obstacle avoidance reward by calculating the distance and distance change between the agent and the destination in the initial state and the current state;

[0061] When training and updating the agent, the sum of the agent's climbing reward, target reward, and obstacle avoidance reward is designed as the agent's intensive reward. Initially, the intensive reward reward=0 is set, and the reward design is as follows:

[0062] First, update the drone status and calculate the change value of the drone coordinates. If the drone is stationary, give a large penalty reward = reward-1000. Traverse the list of grid information (ob_space) around the drone and calculate and compare the distance between the drone and the nearest obstacle. , giving the drone obstacle avoidance rewards , the obstacle avoidance reward is expressed as:

[0063] ;

[0064] In the formula, is the obstacle avoidance reward, is the distance between the drone and the nearest obstacle;

[0065] Computational comparison of current drones Axis coordinates With the destination Axis coordinates , giving the drone a climbing bonus , the climbing reward is expressed as:

[0066] ;

[0067] In the formula, It's a climbing reward. It is an intelligent Axis coordinates, The destination is The coordinates of the axes;

[0068] Calculate the distance between the initial state drone and the destination , the current state of the drone is the distance from the destination Variation of distance , giving drone target rewards , the target reward is expressed as:

[0069] ;

[0070] In the formula, is the target reward, is the distance change, is the distance from the agent to the destination in the initial state, is the distance between the agent and the destination in the current state.

[0071] Sparse rewards are goal-oriented. When the drone encounters certain important events, rewards need to be set for the drone. When the drone collides, a large amount of penalties are generated, reward = reward-200; when the drone reaches the destination, a large amount of rewards = reward+200 are given; if the maximum step length is exceeded, the drone is penalized, reward = reward-20.

[0072] Common agent path planning methods using traditional algorithms, such as Algorithms, Dijkstra algorithms and other graph search-based strategies lack adaptive learning capabilities after initial planning and cannot be optimized and adjusted based on flight experience. The current common path planning technologies that introduce deep reinforcement learning methods and apply DQN (deep Q network), PER-D3QN (oppositional dual deep Q network), and PSO-PRM (proximal policy optimization algorithm) are mostly unable to avoid only giving rewards when encountering specific major events or rewards for searching for target paths, and have poor adaptability to complex obstacle scenes. The dense reward mechanism designed in this application can motivate the agent to avoid obstacles in a more sophisticated way. Compared with the traditional sparse obstacle avoidance idea of ​​only giving penalties when hitting obstacles, this reward mechanism can provide more feedback signals for the agent, accelerate the learning process of the agent, and significantly improve the flexibility and accuracy of the agent's obstacle avoidance, helping it to coordinately optimize multiple task objectives (obstacle avoidance, path optimization, flight stability, etc.), so that the agent can optimize and complete multiple task requirements at the same time.

[0073] e. Reversely update the historical local network and the historical target network using the Q value in d;

[0074] refer to Figure 4 In this step, the difference between the Q value in d and the target Q value is used to calculate the mean square error; the mean square error is used as a loss function, and the loss function is used to reversely update and optimize the historical local network and the historical target network.

[0075] f. Repeat the process of ae until the probability of success of the agent is not less than 80%, where success of the agent means successfully reaching the destination within the specified number of steps.

[0076] In this application, each complete training episode of the drone adopts the ε-greedy strategy with ε=0.1 to train the agents in 15 initialized environments. If the Q network has been trained before, the previously trained Q network parameters are loaded for training, and then the previous number of training cycles is trained, otherwise the training starts from the beginning. Within the specified number of training cycles, the drone continuously selects actions according to the strategy Q value, changes its own state and obtains benefits, and stores its interaction experience. Each time the agent interacts with the environment, it will generate an experience tuple (Transition) containing five fields: agent state (state), action (action), reward (reward), next state (next_state) and whether it is completed (done), and store it in the experience pool. After every five drones complete training, check once. If the amount of experience data in the experience pool exceeds BATCH_SIZE, the Q network is trained. If the success rate of the drone exceeds 80%, the training difficulty (level) is increased to 10.

[0077] After every 100 episodes of training, the parameters of the two networks q_target and q_local, the optimizer status, and the current training cycle number are saved locally for reading the network parameters for the next training.

[0078] S300, converting the overall feature information of the current time into a 27-dimensional Q value, and inputting the 27-dimensional Q value into the current Q network, so that the current Q network selects the best action from the experience pool; the current Q network updates the historical Q network using the historical experience data before the current time in the experience pool, and uses a dense reward function of multiple dimensions to guide the Q network to select an action in the direction of increasing the reward during the update process;

[0079] The historical Q network is a Q network updated at a previous time point, and the Q network includes a local network and a target network. The local network is used to input a 27-dimensional Q value and output a selected action; the target network is used to input a selected action and output a Q value.

[0080] The experience pool stores experience tuples including the agent state, action, reward, the agent state at the next time point, and whether the destination has been reached at each historical time point.

[0081] S400: Execute the optimal action.

[0082] In an optional implementation manner of the present invention, S300 includes:

[0083] S310, inputting the overall feature information of the current time into an adaptive MLP for processing to obtain a 27-dimensional Q value;

[0084] In this step, the overall feature information out of the drone is input into a fully connected layer with a dimension of 338, and the PReLU function is referenced to output a 27-dimensional Q value estimate.

[0085] S320, inputting the 27-dimensional Q value in S310 into the current local network, so that the current local network selects the action with the largest Q value from all actions stored in the experience pool as the best action.

[0086] In order to verify the advantages of the present invention, the present invention designs a visibility test verification project for UAV route planning, and captures the results of a certain UAV training such as Figure 6 As shown in the figure, the drone successfully reached the destination under the complex environment with a training difficulty level of 10. At the same time, a verification algorithm for the drone route planning evaluation was designed to intuitively display the results of each scene of training, including the score of each scene of the drone. Figure 7 The average score (Avg_Score) of the drone as of the current training batch is as follows Figure 8 .exist Figure 7 The horizontal axis is the training batch Epoch, and the vertical axis is the training score of each batch (EveryEpoch_Score). Figure 8 The horizontal axis is the training batch Epoch, and the vertical axis is the score. Due to the randomness of the environment, the randomness of the exploration strategy and the imbalance of the environment difficulty in reinforcement learning, according to Figure 7 It can be seen that the drone is not converging steadily, but Figure 8 It shows that under continuous training, the overall training effect of the drone is gradually optimized, and its average score (Avg_Score) is gradually increasing and stabilizing in an excellent training state.

[0087] It is worth noting that the terms "first" and "second" in the present invention are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0088] Although the present application is described herein in conjunction with various embodiments, in the process of implementing the claimed application, those skilled in the art may understand and implement other variations of the disclosed embodiments by reviewing the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude a plurality of components or steps.

[0089] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A real-time decision-making method for intelligent agent paths facing dynamic threats and local perception, characterized in that: Applied to an intelligent agent, the intelligent agent path real-time decision method for dynamic threats and local perception includes: S100, collecting the current state data of the vehicle and the local obstacle perception data in the environment; S200, preprocessing the local obstacle perception data and the self-state data, and then splicing the preprocessed data to obtain the overall feature information at the current time; S300, converting the overall feature information of the current time into a 27-dimensional Q value, and inputting the 27-dimensional Q value into the current Q network, so that the current Q network selects the best action from the experience pool; the current Q network updates the historical Q network using the historical experience data before the current time in the experience pool, and uses a dense reward function of multiple dimensions to guide the Q network to select an action in the direction of increasing the reward during the update process; S400, executing the optimal action; The intensive reward function of the multiple dimensions includes: giving an obstacle avoidance reward to the agent by calculating the distance between the agent and the nearest obstacle, giving a climbing reward to the agent by calculating the coordinate of the agent on the z-axis and the coordinate of the destination on the z-axis, and giving a target reward to the agent by calculating the distance and distance change between the agent and the destination in the initial state and the current state; Wherein, the obstacle avoidance reward is expressed as: ； In the formula, is the obstacle avoidance reward, is the distance between the drone and the nearest obstacle; The climbing reward is expressed as: ； In the formula, It's a reward for climbing. It is the intelligent agent in The coordinates of the axes, The destination is The coordinates of the axes; The target reward is expressed as: ； In the formula, is the target reward, is the distance change, is the distance from the agent to the destination in the initial state, is the distance between the agent and the destination in the current state.

2. The real-time decision-making method for intelligent agent paths facing dynamic threats and local perception according to claim 1 is characterized in that: The self-state data includes: the three-dimensional position coordinates of the intelligent agent, the destination coordinates, the Manhattan distance between the intelligent agent and the destination at the current time, the Manhattan distance between the initial intelligent agent and the destination, the step length of the intelligent agent and the horizontal movement direction of the intelligent agent; the local obstacle perception data includes: the grid where the obstacle is located and the corresponding three-dimensional coordinates of the grid.

3. The real-time decision-making method for intelligent agent paths facing dynamic threats and local perception according to claim 2 is characterized in that: S200 includes: S210, using MLP to process the self-state data once to obtain first intermediate data, and then performing secondary processing on the first intermediate data through MLP to obtain self-state data features; Among them, the MLP includes a 13-dimensional input layer, a fully connected layer and a 13-dimensional output layer, and a PReLU function is introduced between the fully connected layer and the output layer; S220, reshaping the local obstacle perception data into a vector of a predetermined shape, and then using CNN to perform a primary processing on the vector of the predetermined shape to obtain second intermediate data; and then performing a secondary processing on the second intermediate data through CNN to obtain local obstacle information features; S230, flattening the local obstacle information features, and then concatenating the flattened local obstacle information features with the self-state data features to obtain overall feature information.

4. The real-time decision-making method for intelligent agent paths facing dynamic threats and local perception according to claim 1 is characterized in that: The historical Q network is a Q network updated at a previous time point, and the Q network includes a local network and a target network. The local network is used to input a 27-dimensional Q value and output a selected action; the target network is used to input a selected action and output a Q value.

5. The real-time decision-making method for intelligent agent paths facing dynamic threats and local perception according to claim 4 is characterized in that: S300 includes: S310, inputting the overall feature information of the current time into an adaptive MLP for processing to obtain a 27-dimensional Q value; S320, inputting the 27-dimensional Q value in S310 into the current local network, so that the current local network selects the action with the largest Q value from all actions stored in the experience pool as the best action.

6. The real-time decision-making method for intelligent agent paths facing dynamic threats and local perception according to claim 4 is characterized in that: The experience pool stores experience tuples including the agent state, action, reward at each historical time point, the agent state at the next time point, and whether the destination has been reached.

7. The real-time decision-making method for intelligent agent paths facing dynamic threats and local perception according to claim 4 is characterized in that: The process of updating the historical Q network using the historical experience data in the experience pool includes: a. When the amount of historical experience data in the experience pool exceeds the quantity threshold, the experience data at the earliest time point is removed and the experience data at the latest time point is stored; b. randomly sampling in the experience pool to obtain training data; c. For each agent, the training data is input into the historical local network carried by the agent, so that the historical local network selects an action from all actions stored in the experience pool using the ε-greedy strategy, and feeds back the state of the agent at the next time point corresponding to the action to the historical target network; d. The historical target network uses the state of the agent at the next time point to calculate the Q value for updating the historical local network and the historical target network; wherein the Q value for updating the historical local network and the historical target network is calculated using the density reward function of the multiple dimensions; e. Reversely update the historical local network and the historical target network using the Q value in d; f. Repeat the process of ae until the probability of success of the agent is not less than 80%, where success of the agent means successfully reaching the destination within the specified number of steps.

8. The real-time decision-making method for intelligent agent paths facing dynamic threats and local perception according to claim 7 is characterized in that include: Using the difference between the Q value in d and the target Q value, calculate the mean square error; The mean square error is used as a loss function, and the loss function is used to reversely update and optimize the historical local network and the historical target network.

9. The real-time decision-making method for intelligent agent paths facing dynamic threats and local perception according to claim 8 is characterized in that: The loss function is expressed as: ； In the formula, is the loss function, is the Q value calculated by the target network, It is the Q value calculated using the characteristics of its own state data. yes or Total number of ； ； In the formula, is the discount factor, is the Q value at the next time point, Indicates whether the agent successfully reaches the destination. is 1, otherwise it is 0. Indicates reward and is also an intermediate parameter used for transfer. Indicates that the value ranges from 1 to .

Images