A defensive path planning method, device and electronic device

Abstract

The present invention relates to the field of artificial intelligence technology, and provides a defensive path planning method, device and electronic device, including: determining a rasterized map, a starting node and a plurality of candidate target nodes according to a given path planning task; constructing and training a path planning model based on reinforcement learning based on the rasterized map to obtain a trained shortest path model; initializing an intelligent agent using the trained shortest path model to obtain an initialized intelligent agent; training the initialized intelligent agent based on the obtained observer prediction results to obtain a defensive path planning intelligent agent; and performing path planning based on the intelligent agent to obtain a target path. The present invention can obtain a defensive path planning intelligent agent that can adapt to a learning observer. When planning a path, the intelligent agent not only considers the shortest path, but also considers the defensiveness of the path to avoid the prediction of external observers. The inventive method can be extended to other planning tasks involving privacy protection and adversarial strategies.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a defensive path planning method, device and electronic equipment. Background Art

[0002] Path planning refers to the process of finding the optimal path from the starting point to the destination by optimizing the length, time or other specific objective functions of the path in a given environment while satisfying constraints.

[0003] In some scenarios, we need to protect the path target from being discovered. For example, when armored trucks transport cash in a city, they may be exposed to potential threat observers. At this time, it is necessary to adjust the route and plan a defensive path between multiple false candidate targets to prevent the real destination from being identified, thereby enhancing the security of planning.

[0004] Most of the existing solutions are based on planning paths based on manually designed rules. However, these methods only consider the case of single planning, that is, they usually assume that the external observer is fixed and the planned path can only confuse the observer once. However, in reality, multiple planning is more common, and potential observers may have a certain learning ability and learn from the historical trajectories of multiple planning to see through the fixed confusing behavior. In this case, the performance of existing methods is significantly reduced, so that the confusing effect is completely lost. Therefore, there is an urgent need for an adaptive planning method that can effectively adapt to learning observers during multiple interactions. Summary of the invention

[0005] The present invention provides a defensive path planning method, device and electronic device, which can obtain a defensive path planning intelligent agent capable of adaptively adapting to a learning observer. The technical solution proposed by the present invention is as follows:

[0006] In a first aspect, the present invention provides a defensive path planning method, comprising:

[0007] Determine a rasterized map, a starting node, and multiple candidate target nodes according to a given path planning task;

[0008] Based on the rasterized map, a path planning model based on reinforcement learning is constructed, and the path planning model based on reinforcement learning is trained to obtain a trained shortest path model, wherein the trained shortest path model can plan the shortest path from the starting node to any node including the multiple candidate target nodes;

[0009] Initializing a pre-established intelligent agent using the trained shortest path model to obtain an initialized intelligent agent;

[0010] Obtaining observer prediction results, and training the initialized agent based on the observer prediction results to obtain a defensive path planning agent;

[0011] Path planning is performed based on the defensive path planning agent to obtain a target path.

[0012] Optionally, constructing a path planning model based on reinforcement learning based on the rasterized map, and training the path planning model based on reinforcement learning to obtain a trained shortest path model includes:

[0013] Input the current position and each candidate target node into the pre-established strategy network to obtain the expected cumulative reward for each action;

[0014] Use the epsilon-greedy strategy to select actions, obtain the next state and corresponding rewards based on the actions, and record them as an experience pool;

[0015] The expected cumulative reward is updated using the past states, actions, and rewards in the experience pool, and the network parameters are optimized using gradient descent until the agent converges to obtain the trained shortest path model.

[0016] Optionally, the obtaining of the observer prediction result and training the initialized agent based on the observer prediction result to obtain the defensive path planning agent includes:

[0017] Saving the initial parameters of the initialized agent, recorded as meta-parameters;

[0018] In each task, a candidate target node is selected as the real target node;

[0019] Inputting the real target node of the current task into the initialized agent to obtain a path generated by the agent;

[0020] Based on the obtained observer prediction results and the path generated by the agent, the value function is calculated using the reinforcement learning algorithm and the agent parameters are updated through back propagation;

[0021] Record the gradient of back propagation until the preset number of back propagation gradients is obtained;

[0022] On the basis of the meta-parameters, the meta-parameters of the agent are updated based on the preset number of back-propagation gradients until all tasks are completed.

[0023] Optionally, the method of calculating the value function using the reinforcement learning algorithm and updating the agent parameters by back propagation is:

[0024]

[0025] in, represents the agent parameters for the kth task, Indicates The agent parameters for the second task are: represents the learning rate, represents the value function, The agent parameters are The agent output at time Express Find the gradient.

[0026] Optionally, the value function is calculated as follows:

[0027]

[0028] in, represents the value function, represents the expected value, Indicates the first Steps, total step, Indicates The discount rate of the step, Indicates is the agent reward obtained when inputting, Respectively represent The state and action of the agent at this step, Indicates The state of the agent at step , Represents the additional reward obtained by the agent.

[0029] Optionally, performing path planning based on the defensive path planning agent to obtain a target path includes:

[0030] The target node position is input, and the defensive path planning agent selects an action according to the maximum expected cumulative reward and takes one step, and records the agent position until it reaches the target node position, and the recorded agent position trajectory is recorded as the target path.

[0031] In a second aspect, the present invention further provides a defensive path planning device, comprising the following modules:

[0032] A task determination module, used to determine a rasterized map, a starting node and a plurality of candidate target nodes according to a given path planning task;

[0033] A model training module, used to construct a path planning model based on reinforcement learning based on the rasterized map, and train the path planning model based on reinforcement learning to obtain a trained shortest path model, wherein the trained shortest path model can plan the shortest path from the starting node to any node including the multiple candidate target nodes;

[0034] An initialization module, used to initialize a pre-established intelligent agent using the trained shortest path model to obtain an initialized intelligent agent;

[0035] An agent training module, used to obtain the observer prediction results, and train the initialized agent based on the observer prediction results to obtain a defensive path planning agent;

[0036] The path planning module is used to perform path planning based on the defensive path planning agent to obtain a target path.

[0037] In a third aspect, the present invention further provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the defensive path planning method as described in the first aspect above is implemented.

[0038] In a fourth aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the defensive path planning method as described in the first aspect above.

[0039] In a fifth aspect, the present invention further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the defensive path planning method as described in the first aspect above.

[0040] Based on the above technical solution, the beneficial effects of the present invention compared with the prior art are as follows:

[0041] The defensive path planning method, device and electronic device provided by the present invention construct a path planning model based on reinforcement learning, train the path planning model based on reinforcement learning to obtain a trained shortest path model, and initialize an intelligent agent based on the trained shortest path model. The intelligent agent can quickly plan the shortest path from the starting node to the target node based on a rasterized map. After initialization, the initialized intelligent agent is trained based on the observer's prediction results, and the intelligent agent is iteratively optimized through inner layer reinforcement learning, so that when planning the path, it not only considers the shortest path, but also considers the defensiveness of the path to avoid the prediction of external observers and confuse the observer. At the same time, through the outer layer meta-learning iteration, the historical experience across multiple rounds of tasks is used to enable the intelligent agent to adapt to the changes of external observers, and continuously improve the robustness and defensiveness of its path planning, thereby obtaining a defensive path planning intelligent agent that can adapt to learning observers. The method of the invention is simple and robust, and can be easily extended to other planning tasks involving privacy protection and adversarial strategies.

[0042] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0043] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0045] Figure 1 It is one of the structural schematic diagrams of the defensive path planning method provided by the present invention.

[0046] Figure 2 This is the second structural diagram of the defensive path planning method provided by the present invention.

[0047] Figure 3 It is a structural schematic diagram of the defensive path planning device provided by the present invention.

[0048] Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] Combine the following Figure 1-Figure 4 The defensive path planning method, device and electronic device of the present invention are described.

[0051] Reference Figure 1 As shown, the defensive path planning method includes the following:

[0052] Step S110: determining a rasterized map, a starting node, and a plurality of candidate target nodes according to a given path planning task.

[0053] For a given path planning task, we first need to determine the basic structure of the environment. To facilitate calculation and path search, the map is rasterized and divided into several equally spaced grids through unit segmentation. Each grid represents a location in the environment, and obstacles in the environment are converted into unreachable grids in the corresponding location of the map. Each grid is assigned two different spatial states, with obstacle space marked as 1 (black) and free space marked as 0 (white).

[0054] Next, the initial position of the agent, i.e. the starting node, is determined. Then several candidate target nodes are selected on the map. The selection of candidate target nodes is generally the end point or key position according to the task requirements.

[0055] Step S120: Based on the rasterized map, a path planning model based on reinforcement learning is constructed, and the path planning model based on reinforcement learning is trained to obtain a trained shortest path model.

[0056] This trained shortest path model can plan the shortest path from the starting node to any node including multiple candidate target nodes. During the training process, the model will learn the structure of the rasterized map and the connection relationship between nodes so as to quickly find the shortest path in subsequent path planning.

[0057] The above reinforcement learning-based path planning model is constructed as follows:

[0058] Define the agent. In the path planning problem, the agent represents the entity that needs to plan the path, such as a robot or a vehicle. The agent moves in the grid map and adjusts its movement strategy based on the feedback from the environment. Define the environment. The environment is a grid map, including obstacles, traversable areas, starting points and end points. The environment provides the agent with state information, such as the current position, the distribution of surrounding obstacles, etc. Define the action space. The action space contains all the actions that the agent can perform in the grid map, such as moving up, down, left, and right. The agent selects actions based on the current state and strategy. Define the reward function. The reward function is used to evaluate the effect of the agent after performing the action. In the path planning problem, the reward function can be designed as follows: a high reward is obtained when reaching the target position, a negative reward is obtained when colliding with an obstacle, and a certain negative reward can also be obtained for each step of movement (to encourage the agent to reach the target position as soon as possible). Select a suitable reinforcement learning algorithm, such as Q-learning, SARSA, Deep Q-Network (DQN), etc. Initialize the state value function or action value function according to the algorithm requirements, and set parameters such as learning rate and discount factor.

[0059] The process of training a reinforcement learning-based path planning model is:

[0060] Place the agent at the starting point and initialize its state-value function or action-value function. In each training iteration, the agent selects an action based on the current state and strategy. After executing the action, the agent observes the new state and obtains a reward based on the reward function. Update the state-value function or action-value function to reflect new knowledge and experience. Repeat the above process until the agent learns the shortest path from the starting point to the end point or reaches the predetermined number of training times. After the training is completed, use the test set or validation set to evaluate the performance of the model. The model can be evaluated by calculating the average number of steps, success rate, and other indicators of the agent from the starting point to the end point. According to the evaluation results, adjust the parameters of the reinforcement learning algorithm (such as learning rate, discount factor, etc.) or improve the model structure (such as using a more complex neural network to approximate the action-value function). Repeat the training process until satisfactory performance is obtained and the above-mentioned trained shortest path model is obtained.

[0061] Step S130: Initialize the pre-established intelligent agent using the trained shortest path model to obtain an initialized intelligent agent.

[0062] Using the trained shortest path model, a pre-established agent is initialized. This agent will be used for subsequent path planning tasks. The initialization process includes loading the parameters of the trained shortest path model and rasterized map information into the agent, so that the agent has basic environmental perception and path planning capabilities.

[0063] Step S140: Obtain observer prediction results, and train the initialized agent based on the observer prediction results to obtain a defensive path planning agent.

[0064] In practical applications, there may be one or more observers who can make predictions or provide additional information about the agent's path planning process. The prediction results of these observers are obtained, and the initialized agent is further trained based on these results. During the training process, the agent will learn how to adjust its path planning strategy based on the observer's predictions to improve the accuracy and robustness of path planning.

[0065] The above observer can be a person or a model with learning ability. The observer can collect data by directly observing the behavior of the intelligent agent. For example, the parameters such as the moving path, speed, acceleration, etc. of the intelligent agent can be recorded. In addition to direct observation, the observer can also use indirect data, such as the historical path data of the intelligent agent, environmental maps, obstacle information, etc., to infer the possible behavior of the intelligent agent. The observer can also use rule-based methods to predict the path planning of the intelligent agent. These rules can be based on experience, expert knowledge, or domain-specific logic. Observers can also use machine learning techniques to train predictive models. These models can be supervised learning (using labeled data) or unsupervised learning (using unlabeled data to discover patterns).

[0066] The observer can observe the behavior of the agent in real time while it is performing a task, and dynamically adjust the prediction model based on the observation results. This online learning method allows the observer to capture changes in the agent's behavior in a timely manner and make corresponding predictions. The observer outputs the prediction results indirectly (such as providing additional environmental information), and the agent uses this information as feedback to improve its path planning strategy, thereby enhancing its defensiveness and adaptability.

[0067] By introducing the observer's prediction results to train the agent, the agent can learn how to adjust its path planning strategy based on external information. This helps to enhance the robustness of path planning and enables the agent to maintain stable performance in complex and changing environments. This method encourages the agent to learn and adapt autonomously based on the observer's prediction results. This not only helps to improve the agent's path planning ability, but also promotes its learning and adaptation ability in other tasks. Through continuous learning and adjustment, the agent can gradually adapt to more complex environments and task requirements.

[0068] Step S150: Perform path planning based on the defensive path planning agent to obtain a target path.

[0069] Path planning is performed based on the trained defensive path planning agent. The location of the target node is input into the agent, and the agent will plan an optimal path from the starting node to the target node based on the rasterized map and its own path planning strategy. This path is the target path.

[0070] This paper proposes an innovative defensive path planning method, which is not only based on the shortest path planning agent, but also integrates reinforcement learning optimization and meta-learning framework. Through the inner layer reinforcement learning iterative optimization, the path's defensiveness is enhanced; at the same time, the outer layer meta-learning iteratively utilizes historical experience across multiple rounds of tasks, enabling the agent to adapt to the changing external observer.

[0071] Specifically, by constructing a path planning model based on reinforcement learning, training the path planning model based on reinforcement learning to obtain a trained shortest path model, and initializing an intelligent agent based on the trained shortest path model, the intelligent agent can quickly plan the shortest path from the starting node to the target node based on a rasterized map. After initialization, the initialized intelligent agent is trained based on the observer's prediction results, and the intelligent agent is iteratively optimized through inner-layer reinforcement learning, so that when planning a path, it not only considers the shortest path, but also the defensiveness of the path to avoid the prediction of external observers. At the same time, through the outer-layer meta-learning iteration, using historical experience across multiple rounds of tasks, the intelligent agent can adapt to the changes of external observers, and continuously improve the robustness and defensiveness of its path planning, thereby obtaining a defensive path planning intelligent agent that can adapt to learning observers. The invention method is simple and robust, and can be easily extended to other planning tasks involving privacy protection and adversarial strategies.

[0072] The trained shortest path model is obtained by reinforcement learning training and is used to initialize the intelligent agent. The specific training method is not limited. In an optional embodiment, the step S110 described above constructs a path planning model based on reinforcement learning based on the rasterized map, and trains the path planning model based on reinforcement learning to obtain the trained shortest path model, including:

[0073] S1201: Input the current position and each candidate target node into a pre-established strategy network to obtain the expected cumulative reward of each action.

[0074] Define a reward function, such as giving a -1 reward for each action step, a +1 reward for each action step that shortens the distance to the target node, and a +100 reward for reaching the target node.

[0075] Establish a policy network, the present invention uses DQN as the agent policy network, which accepts the current position and each candidate target node as input. The output of the network is the expected cumulative reward of each possible action, which reflects the potential value from the current position to the target node.

[0076] S1202. Use the epsilon-greedy strategy to select an action, obtain the next state and corresponding reward based on the action, and record them as an experience pool.

[0077] This process is also the process of environmental interaction. At each decision moment, the epsilon-greedy strategy is used to select actions. Specifically, This helps discover new and potentially better paths. The probability of using (i.e. selecting the best action currently known) is used to maximize the immediate reward and path efficiency. After executing the selected action, the next state and the corresponding reward are obtained, and this experience (state, action, reward) is recorded as a record in the experience pool.

[0078] S1203. Use the past states, actions, and rewards in the experience pool to update the expected cumulative reward, and use gradient descent to optimize the network parameters until the agent converges to obtain the trained shortest path model.

[0079] This process is the process of policy updating. The expected cumulative reward of the policy network is updated using past experience (i.e., state, action, reward) in the experience pool. Optimization algorithms such as gradient descent are used to adjust the parameters of the network to minimize the error between the expected cumulative reward and the actual reward. Repeat the above-mentioned environmental interaction process of S1202 and the policy update process of the above-mentioned step S1203 until the agent converges (i.e., the performance of the network no longer improves significantly or reaches the preset number of iterations). After training, the policy network will be able to output the shortest path from any starting point to any target node. This model can be used for subsequent path planning, navigation, and decision-making tasks.

[0080] The present invention combines the process of obtaining the trained shortest path model based on raster map training with raster map representation. The raster map representation simplifies the environment model and reduces the complexity of path search. By adjusting the size of the grid and the epsilon-greedy strategy, the shortest path model is obtained. The value can balance the trade-off between exploration and exploitation to adapt to different environments and task requirements. Through reinforcement learning algorithms (such as epsilon-greedy strategy) and gradient descent optimization, it can quickly converge to the optimal or approximately optimal path. It can provide the agent with a shortest or approximately shortest path from the starting point to the target node, while taking into account obstacles and environmental constraints.

[0081] In an optional embodiment, the obtaining of the observer prediction result in step S140, and training the initialized agent based on the observer prediction result to obtain the defensive path planning agent, include:

[0082] S1401. Save the initial parameters of the initialized agent as meta-parameters.

[0083] In this step, the parameters of the initialized agent are saved, which will be used as meta-parameters for subsequent parameter updates in the meta-learning framework. Meta-parameters represent the initial state or memory of the agent, which will be continuously adjusted and optimized in the subsequent training process.

[0084] S1402. In each task, select a candidate target node as the real target node.

[0085] In order to simulate the path planning task in the real environment, a candidate target node is randomly selected as the real target node in each task. This ensures that the agent can perform effective path planning when facing different target nodes.

[0086] S1403: Input the real target node of the current task into the initialized agent to obtain a path generated by the agent.

[0087] After the real target node is input into the agent, the agent will generate a path from the starting node to the target node based on the current rasterized map and its own position.

[0088] S1404. Based on the obtained observer prediction results and the path generated by the agent, the value function is calculated using the reinforcement learning algorithm and the agent parameters are updated through back propagation.

[0089] In this step, the value function is calculated using the observer's predictions and the path generated by the agent. The value function measures the quality of the agent's current strategy, that is, whether the path it generates successfully circumvents the observer's predictions. In order to encourage the agent to generate a more defensive path, the present invention designs an additional positive and negative reward mechanism:

[0090] The calculation method for the additional positive and negative rewards obtained by the agent is:

[0091]

[0092] in, represents the additional reward obtained by the agent, Represents the observer's predicted probability that a certain node is the true target node, represents the observer parameter, represents the path of the agent output during k tasks, , represents the number of iterations, represents the real target node, is the reward weight.

[0093] When the observer predicts correctly, the agent receives a negative reward, with a reward value of , which encourages the agent to generate paths that are difficult for the observer to predict. When the observer predicts incorrectly, the agent receives a positive reward with a reward value of It should be noted that the above positive reward formula should actually be Multiply by a positive number, but in order to maintain consistency with the negative reward form mentioned above, the negative sign is used here to indicate that the opposite of the probability of the observer's prediction error is used as the basis for the reward. The absolute value or appropriate adjustment form should be taken in actual calculation.

[0094] Subsequently, reinforcement learning algorithms (such as Q-learning, policy gradient, etc.) and back-propagation techniques are used to update the parameters of the agent to maximize the total reward it obtains.

[0095] S1405, recording the gradient of the back propagation until a preset number of back propagation gradients are obtained;

[0096] In each parameter update process, the gradient information of back propagation is recorded. This gradient information reflects the degree of influence of the agent parameters on the value function. In order to obtain a more stable and accurate parameter update, multiple back propagations are performed and gradient information is accumulated, that is, the above steps S1403 and S1404 are iteratively executed until the preset number of times is reached.

[0097] S1406. Based on the meta-parameters, update the meta-parameters of the agent based on the preset number of back-propagation gradients until all tasks are completed.

[0098] After accumulating enough gradient information, the parameters of the agent are updated based on the meta-parameters. Specifically, the back propagation gradients of the preset number of times are obtained, and the sum of the back propagation gradients of the preset number of times is taken as the parameter update amount. The new agent parameters are calculated based on the meta-parameters, which is recorded as a meta-update.

[0099] This updating method not only considers the experience information in the current task, but also integrates the historical experience in previous tasks (reflected by meta parameters). In this way, the agent can gradually learn how to better adapt to the changes of external observers and generate more defensive paths. This process will continue, iterating the above steps S1401-S1406 until all tasks are completed.

[0100] By introducing observer prediction results and additional positive and negative reward mechanisms, the present invention encourages the agent to generate paths that are difficult for observers to predict. This improves the defensiveness of path planning and helps to protect the action intentions of the agent in scenarios such as privacy protection and adversarial strategies. The parameter update method under the meta-learning framework enables the agent to gradually learn how to better adapt to changes in external observers. This adaptive ability enables the agent to maintain stable performance in the face of a changing external environment. By recording the gradient information of back propagation and accumulating multiple parameter updates, the parameter update process can be made more stable and accurate. This helps to improve training efficiency and reduce fluctuations and uncertainties during training. This method is not only applicable to defensive path planning tasks, but can also be extended to other planning tasks involving privacy protection and adversarial strategies. By adjusting components such as the observer prediction model and the reward function, it can adapt to the needs of different scenarios and tasks.

[0101] In an optional embodiment, the method of calculating the value function using the reinforcement learning algorithm and updating the agent parameters by back propagation described in step S1404 is:

[0102]

[0103] in, Indicates The agent parameters for the second task are: Indicates The agent parameters for the second task are: represents the learning rate, represents the value function, The agent parameters are The agent output at time Express Find the gradient.

[0104] In an optional embodiment, the value function is calculated as follows:

[0105]

[0106] in, represents the value function, represents the expected value, Indicates the first Steps, total step, Indicates The discount rate of the step, Indicates is the agent reward obtained when inputting, that is, in state Take Action After transfer to state The agent reward obtained when Respectively represent The state and action of the agent at this step, Indicates The state of the agent at step , Represents the additional reward obtained by the agent.

[0107] In an optional embodiment, the step S150 described above of performing path planning based on the defensive path planning agent to obtain a target path includes:

[0108] S1501. Input the target node position. The defensive path planning agent selects an action according to the maximum expected cumulative reward and takes one step. The agent position is recorded until the target node position is reached. The recorded agent position trajectory is recorded as the target path.

[0109] First, the location of the target node needs to be clear, and the agent will plan its path based on this location information. After receiving the location of the target node, the defensive path planning agent will start planning its path. The agent will use the reinforcement learning algorithm to calculate the maximum expected cumulative reward for each possible action based on the current rasterized map, its own position, and the location of the target node. The agent will then select the action with the maximum expected cumulative reward and execute it, that is, move forward one step. Each time the agent performs an action, its new position is recorded. This is to construct the action trajectory of the agent for subsequent analysis and optimization.

[0110] The agent will continue to repeat the above process of selecting actions and recording positions until it reaches the target node. In this process, the agent will adjust its action strategy based on the real-time updated map information and the observer's prediction results to ensure that it can reach the target node safely and efficiently.

[0111] When the agent reaches the target node, the recorded position trajectory of the agent is defined as the target path. This path is the optimal path learned by the agent during multiple interactions and adaptations, which takes into account multiple factors such as the length, safety, and defensiveness of the path.

[0112] The path planning method based on the defensive path planning agent of the present invention can quickly plan the optimal path from the starting node to the target node. By continuously learning and adapting to changes in the external environment, the agent can gradually optimize its action strategy, thereby improving the efficiency of path planning. Specifically, by introducing the observer prediction results and additional positive and negative reward mechanisms, the defensive path planning agent takes into account the prediction results of the external observer when planning the path, and the agent is encouraged to generate a path that is difficult for the observer to predict, that is, a more defensive path. The agent in this method has a strong adaptive ability. It can continuously learn and adapt to changes in the external environment during multiple interactions, including the prediction strategy of the learning observer. This adaptive ability enables the agent to maintain stable performance and generate a more defensive path when facing a constantly changing external environment. This helps to improve the defensiveness of path planning in scenarios where it is necessary to hide action intentions or counter observer predictions, thereby protecting the privacy and security of the agent.

[0113] In order to more clearly illustrate the defensive path planning method of the present invention, the following is combined with the attached Figure 2 Each step in an embodiment of the method of the present invention is described in detail.

[0114] Step S210, determining a rasterized map, a starting node and a plurality of candidate target nodes according to a given path planning task.

[0115] Step S220, based on a given rasterized map, construct a path planning model based on reinforcement learning, train the path planning model based on reinforcement learning to obtain a trained shortest path model, which can plan the shortest path from the starting point to any node, and use the trained shortest path model to initialize the defensive path planning agent.

[0116] Step S230, save the agent parameters, recorded as meta-parameters.

[0117] Step S240: In each task, a candidate target node is selected as the real target node. The real target node of the current task is input, the path generated by the agent is output, and the observer prediction result is obtained.

[0118] In this step, the agent inputs the current position and the real target node position. It selects an action according to the maximum expected cumulative reward value and takes one step. The agent position is recorded and the above steps are repeated until the real target node is reached. The recorded agent position trajectory is recorded as the path output. The path is output as the solution to the task for application in real scenarios, and then the prediction results of external observers are obtained.

[0119] Step S250: If the observer correctly predicts the true target, the agent receives an additional negative reward, otherwise, the agent receives an additional positive reward. The value function is calculated and back-propagated according to the reinforcement learning algorithm, the back-propagation gradient is recorded, and then the agent parameters are updated.

[0120] Step S260, iteratively execute steps S240 and S250 for a total of K times. K can be set according to actual needs, and the present invention does not make a specific limitation. Generally speaking, the more complex the path planning task is, the larger the K value is selected. In this embodiment, K is 4.

[0121] Step S270, obtain K back-propagation gradients, take the sum of the K gradients as the parameter update amount, calculate the new agent parameters based on the meta-parameters, and record it as a meta-update.

[0122] Step S280, iteratively execute steps S230, S240, S250, S260, and S270 until the task is completed.

[0123] The defensive path planning device provided by the present invention is described below. The defensive path planning device described below and the defensive path planning method described above can be referenced to each other.

[0124] The defensive path planning device provided by the present invention refers to Figure 3 As shown, including:

[0125] A task determination module 310 is used to determine a rasterized map, a starting node and a plurality of candidate target nodes according to a given path planning task;

[0126] A model training module 320 is used to construct a path planning model based on reinforcement learning based on the rasterized map, and train the path planning model based on reinforcement learning to obtain a trained shortest path model, wherein the trained shortest path model can plan the shortest path from the starting node to any node including the multiple candidate target nodes;

[0127] An initialization module 330 is used to initialize a pre-established intelligent agent using the trained shortest path model to obtain an initialized intelligent agent;

[0128] An agent training module 340 is used to obtain the observer prediction result, and train the initialized agent based on the observer prediction result to obtain a defensive path planning agent;

[0129] The path planning module 350 is used to perform path planning based on the defensive path planning agent to obtain a target path.

[0130] Figure 4An example of a physical structure diagram of an electronic device is shown in FIG. Figure 4 As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430 and a communication bus 440, wherein the processor 410, the communication interface 420 and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call the logic instructions in the memory 430 to execute the defensive path planning method.

[0131] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0132] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the defensive path planning methods provided by the above methods.

[0133] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to execute the defensive path planning method provided by the above methods.

[0134] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0135] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0136] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A defensive path planning method, characterized in that: include: Determine a rasterized map, a starting node, and multiple candidate target nodes according to a given path planning task; Based on the rasterized map, a path planning model based on reinforcement learning is constructed, and the path planning model based on reinforcement learning is trained to obtain a trained shortest path model, wherein the trained shortest path model can plan the shortest path from the starting node to any node including the multiple candidate target nodes; Initializing a pre-established intelligent agent using the trained shortest path model to obtain an initialized intelligent agent; The initial parameters of the initialized agent are saved and recorded as meta-parameters; in each task, a candidate target node is selected as the real target node; the real target node of the current task is input into the initialized agent to obtain the path generated by the agent; according to the obtained observer prediction results and the path generated by the agent, the value function is calculated by the reinforcement learning algorithm and the agent parameters are updated by back propagation; an additional positive and negative reward mechanism is designed, and the calculation method of the additional positive and negative rewards obtained by the agent is: ； in, represents the additional reward obtained by the agent, Represents the observer's predicted probability that a certain node is the true target node, represents the observer parameter, represents the path of the agent output during k tasks, , represents the number of iterations, represents the real target node, is the reward weight; recording the gradient of back propagation until a preset number of back propagation gradients are obtained; based on the meta-parameters, updating the meta-parameters of the agent based on the preset number of back propagation gradients until all tasks are completed, thereby obtaining a defensive path planning agent; Path planning is performed based on the defensive path planning agent to obtain a target path.

2. The defensive path planning method according to claim 1, characterized in that: The step of constructing a path planning model based on reinforcement learning based on the rasterized map and training the path planning model based on reinforcement learning to obtain a trained shortest path model includes: Input the current position and each candidate target node into the pre-established strategy network to obtain the expected cumulative reward for each action; Use the epsilon-greedy strategy to select actions, obtain the next state and corresponding rewards based on the actions, and record them as an experience pool; The expected cumulative reward is updated using the past states, actions, and rewards in the experience pool, and the network parameters are optimized using gradient descent until the agent converges to obtain the trained shortest path model.

3. The defensive path planning method according to claim 1, characterized in that: The method of calculating the value function by using the reinforcement learning algorithm and updating the agent parameters by back propagation is: ； in, represents the agent parameters for the kth task, Indicates The agent parameters for the second task are: represents the learning rate, represents the value function, The agent parameters are The agent output at time Express Find the gradient.

4. The defensive path planning method according to claim 1, characterized in that: The value function is calculated as follows: ； in, represents the value function, represents the expected value, Indicates the first Steps, total step, Indicates The discount rate of the step, Indicates is the agent reward obtained when inputting, Respectively represent The state and action of the agent at this step, Indicates The state of the agent at step , Represents the additional reward obtained by the agent.

5. The defensive path planning method according to claim 1, characterized in that: The performing path planning based on the defensive path planning agent to obtain a target path includes: The target node position is input, and the defensive path planning agent selects an action according to the maximum expected cumulative reward and takes one step, and records the agent position until it reaches the target node position, and the recorded agent position trajectory is recorded as the target path.

6. A defensive path planning device, characterized in that: include: A task determination module, used to determine a rasterized map, a starting node and a plurality of candidate target nodes according to a given path planning task; A model training module, used to construct a path planning model based on reinforcement learning based on the rasterized map, and train the path planning model based on reinforcement learning to obtain a trained shortest path model, wherein the trained shortest path model can plan the shortest path from the starting node to any node including the multiple candidate target nodes; An initialization module, used to initialize a pre-established intelligent agent using the trained shortest path model to obtain an initialized intelligent agent; The agent training module is used to save the initial parameters of the initialized agent, recorded as meta-parameters; in each task, a candidate target node is selected as the real target node; the real target node of the current task is input into the initialized agent to obtain the path generated by the agent; according to the obtained observer prediction results and the path generated by the agent, the value function is calculated using the reinforcement learning algorithm and the agent parameters are updated through back propagation, and an additional positive and negative reward mechanism is designed: the calculation method of the additional positive and negative rewards obtained by the agent is: ； in, represents the additional reward obtained by the agent, Represents the observer's predicted probability that a certain node is the true target node, represents the observer parameter, represents the path of the agent output during k tasks, , represents the number of iterations, represents the real target node, is the reward weight; recording the gradient of back propagation until a preset number of back propagation gradients are obtained; based on the meta-parameters, updating the meta-parameters of the agent based on the preset number of back propagation gradients until all tasks are completed, thereby obtaining a defensive path planning agent; The path planning module is used to perform path planning based on the defensive path planning agent to obtain a target path.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the defensive path planning method according to any one of claims 1 to 5 is implemented.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the defensive path planning method according to any one of claims 1 to 5 is implemented.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the defensive path planning method according to any one of claims 1 to 5 is implemented.

Images