A path planning method based on combination of neural network and artificial potential field method

Abstract

The application provides a path planning method based on combination of a neural network and an artificial potential field method. In a target detection network, a Faster R-CNN containing a backbone network of GoogLeNet is used to identify target points and obstacles in an image. In the path planning method, a velocity component is obtained through gravity and repulsion of the artificial potential field method; meanwhile, the gravity and repulsion in the artificial potential field are used as state quantities of a strategy network of a confidence domain strategy optimization method, and the state quantities are input into the strategy network to obtain the velocity component. The velocity components obtained by the artificial potential field method and the confidence domain strategy optimization method are combined to obtain a final velocity, so that an intelligent agent can obtain a more intelligent velocity. The method can solve the problems of local minimum value of the artificial potential field method and long training time of the strategy network, and can improve the accuracy of target detection and the applicability of an unknown environment.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence, machine learning, and motion control, and relates to artificial intelligence and obstacle avoidance control methods, applicable to three-dimensional dynamic path planning for intelligent agents and drones. Background Technology

[0002] Existing path planning methods based on artificial potential fields have two drawbacks: 1. Goal unreachability: If there are obstacles near the destination, the agent will also be affected by the repulsive potential field as it approaches the destination, as the destination is within the range of the obstacle's repulsive potential field. At this point, the destination is no longer the lowest point in the entire potential field, preventing the agent from reaching the target point. 2. Local optima: During movement, the agent may reach a specific point where the attractive and repulsive forces acting on it are equal in magnitude and opposite in direction, resulting in a net force of zero. In this case, the agent will stop moving or remain stationary. These two problems arise because the agent's velocity determination depends only on the net force; when the net force is zero, the agent will encounter the aforementioned issues.

[0003] Furthermore, most existing deep reinforcement learning path planning methods employ deep Q-networks (DQNs). The input to the neural network is the entire map and the localization location, which is then processed internally to output the speed of the next step. Since most steps in the path planning process are handled within the deep Q-network, it has a high capacity for handling complex problems. However, this approach has two drawbacks: 1. It is not suitable for handling continuous actions; 2. During training, the neural network converges slowly, requiring precise parameter tuning.

[0004] Therefore, a path planning method combining neural networks and artificial potential fields is proposed. GoogLeNet extracts features more effectively than VGG and reduces the number of parameters. Replacing the VGG backbone with the GoogLeNet backbone yields Faster R-CNN based on the GoogLeNet backbone, which is a more accurate object detection method. Combining the policy network with the artificial potential field introduces an artificial potential field into the environment; the policy network then needs to determine the agent's next step speed based on potential energy, reward, attraction, and repulsion, reducing its workload. During training, the policy network employs a confidence region policy optimization method; this method is not only suitable for continuous actions but also makes the trained policy network more stable. Entropy loss is calculated when calculating the loss function; when the agent is uncertain about its next action, a higher entropy value increases uncertainty, encouraging exploration and helping the agent escape local optima. The final speed is no longer determined solely by the resultant force of the artificial potential field method, but also by the policy network in the confidence region policy optimization method. This enhances the agent's decision-making ability. Summary of the Invention

[0005] This invention proposes a path planning method based on a combination of neural networks and artificial potential field methods. In obstacle and target point identification, Faster R-CNN based on the GoogLeNet backbone network exhibits higher accuracy than Faster R-CNN based on the VGG backbone network. In path planning combining artificial potential field methods and confidence region policy optimization methods, the policy network in the confidence region policy optimization method can compensate for the agent's insufficient decision-making ability in the artificial potential field method, resulting in more reasonable decisions made by the agent under the same environment. Introducing the artificial potential field environment allows the agent to obtain more information, enhancing its interaction and perception capabilities with the environment. Training the policy network using the confidence region policy optimization method requires less parameter tuning and results in more stable training. It effectively solves the problems of local optima and target unreachability, making the agent suitable for path planning in unknown and complex environments. The steps in the usage process are as follows:

[0006] Step (1.1): The intelligent agent takes real-time photos of the environment using a camera to obtain images at each moment;

[0007] Step (1.2): For each time step, obstacles and target points are identified in the image using Faster R-CNN with a backbone network containing GoogLeNet;

[0008] The specific process of object detection using Faster R-CNN with a GoogLeNet backbone is as follows: Images captured in real-time by a camera are used to extract features from the GoogLeNet backbone, resulting in a feature map. Then, a region generation network (GRN) is used to generate candidate boxes on the feature map. These candidate boxes are projected onto the feature map to obtain the corresponding feature matrix. The feature matrix is ​​scaled to 7×7 size using a region of interest (ROI) pooling layer. It is then further scaled into feature vectors using a global average pooling layer. Finally, a fully connected layer is used to classify obstacles and target points and adjust the positions of the candidate boxes, thus detecting the locations of target points and obstacles.

[0009] The backbone network of GoogLeNet consists of an image input layer, a 7×7 convolutional layer with a stride of 2, a 3×3 max pooling layer with a stride of 2, a local response normalization layer, a 1×1 convolutional layer with a stride of 2, a 3×3 convolutional layer with a stride of 1, a local response normalization layer, a 3×3 max pooling layer with a stride of 2, two Inception modules, a 3×3 max pooling layer with a stride of 2, five Inception modules, a 3×3 max pooling layer with a stride of 2, and two Inception modules.

[0010] In the backbone network of GoogLeNet, the Inception module consists of four neural network layers. The first layer is the feature matrix extracted by the previous layer. The second layer consists of a 1×1 convolutional layer with a stride of 1 and a 3×3 max pooling layer with a stride of 1 to reduce the dimensionality of the feature matrix. The third layer consists of a 1×1 convolutional layer with a stride of 1, a 3×3 convolutional layer with a stride of 1, and a 5×5 convolutional layer with a stride of 1 to extract rich features through convolutional kernels of different sizes. The fourth layer connects the rich features extracted by the previous three layers to form an even richer feature matrix.

[0011] Step (1.3): The positions of the detected obstacles and target points are obtained by infrared ranging to determine the displacement of the obstacles and target points relative to the agent in the environment;

[0012] Step (1.4): The number of target points is The number of obstacles is ; Calculate the potential energy of the intelligent agent in the potential energy field;

[0013] No. The agent obtains the first time gravitational potential energy at each target point for:

[0014] (1)

[0015] in For the first Time-based intelligent agents and the first Displacement between target points; The constant of the gravitational potential field;

[0016] No. The agent obtains the first time The repulsive potential energy of the obstacle for:

[0017] (2)

[0018] in For the first Time-based intelligent agents and the first Displacement between obstacles; The radius of influence of the repulsive field of the obstacle in the environment; The constant of the repulsive potential field;

[0019] No. Total potential energy of intelligent agent at any moment for:

[0020] (3)

[0021] Step (1.5): Using the artificial potential field method, the displacement of the obstacle is converted into a repulsive force, and the displacement of the target point is converted into an attractive force;

[0022] No. Time-based intelligent agents and the first The repulsive force between the obstacles is projected onto Angles on a plane for:

[0023] (4)

[0024] No. Time-based intelligent agents and the first The repulsive force between the obstacles and Angle of axis for:

[0025] (5)

[0026] in For the first Time-based intelligent agents and the first The displacement between the obstacles The size of the projection onto the plane; For the first Time-based intelligent agents and the first The displacement between the obstacles The size of the projection on the axis; ( ) represents the inverse cosine function;

[0027] No. The agent is subject to the first The magnitude of the repulsive force of the obstacle for:

[0028] (6)

[0029] in For the first The mass of the obstacle; For the quality of the intelligent agent; It is the repulsion coefficient;

[0030] No. Time-based intelligent agents and the first The gravitational projection between the target points is Angles on a plane for:

[0031] (7)

[0032] No. Time-based intelligent agents and the first The gravitational force between the target points and Angle of axis for:

[0033] (8)

[0034] in For the first Time-based intelligent agents and the first The displacement between the target points The size of the projection onto the plane; For the first Time-based intelligent agents and the first The displacement between the target points The size of the projection on the axis;

[0035] No. The agent is subject to the first The gravitational force at each target point for:

[0036] (9)

[0037] in It is the gravitational coefficient;

[0038] No. Time-based intelligent agents and the first Gravity at each target point for:

[0039] (10)

[0040] in for In the Time projection on The magnitude of gravity on a plane; for Unit vector along the axial direction; for Unit vector along the axial direction; for Unit vector along the axial direction; ( ) represents the cosine function; ( ) represents the sine function;

[0041] No. Total Gravity of Intelligent Agents at All Times for:

[0042] (11)

[0043] No. Time-based intelligent agents and the first The repulsive force of the obstacle for:

[0044] (12)

[0045] in for In the Time projection on The magnitude of gravity on a plane;

[0046] No. Total repulsive force of intelligent agents at all times for:

[0047] (13)

[0048] Step (1.6): Based on the attraction and repulsion forces, calculate the first... The velocity components of the agent at any given moment ;

[0049] No. The velocity components of the agent at any given moment for:

[0050] (14)

[0051] in For the ( At time ), the velocity components calculated using the artificial potential field method; For time step;

[0052] For the first intelligent agent acceleration at any moment for:

[0053] (15)

[0054] Step (1.7): [The text appears to be incomplete and contains several grammatical errors. A more accurate translation would require Potential energy at time ( ), the ( ) Rewards at the moment, the first The gravitational and repulsive forces at each moment are arranged into a state vector, which is input into the policy network trained by the confidence region policy optimization method to calculate the velocity components. Projected on The angle between the planes, and The included angle and velocity magnitude of the axis are used to obtain the first value under the confidence region policy optimization method. The velocity components of the agent at any given moment ;

[0055] No. State vector at time step for:

[0056] (16)

[0057] in For the ( Rewards at certain moments; It is the first Total potential energy at any given moment; For the first The agent moves from the first target point to the second target point. The gravitational pull of a target point; For the first The agent moves from the first obstacle to the second obstacle. The repulsive force of an obstacle;

[0058] In the confidence region policy optimization method, in order to obtain the velocity component Capable of representing arbitrary directions, the agent's action space employs a continuous action space; within this continuous action space, the action probability distribution adopts an expected value of... variance is A one-dimensional Gaussian distribution; a complete representation of velocity components. To use a projection The angle between the planes, and a with The included angle and velocity components of the axis The velocity components are represented by three quantities: magnitude, size, and so on. Therefore, a three-policy network is used to distribute and represent the velocity components. The two included angles and their size;

[0059] The first State vector at time step Input three policy networks to obtain velocity components. The two included angles and the magnitude of the velocity, i.e., the complete velocity components. ;

[0060] Step (1.8): The velocity obtained using the traditional artificial potential field method The velocity obtained by the confidence region policy optimization method under the potential energy field Add them together to obtain the intelligent agent. Total velocity at time 1 for:

[0061] (17)

[0062] Step (1.9): In Within a given time period, the intelligent agent travels a certain distance.

[0063] No. Always Displacement within a time period for:

[0064] (18)

[0065] Step (1.10): As the agent's displacement relative to the target point and obstacles changes, its potential energy also changes, and the agent obtains the first... The reward at the moment, as the ( The state of the policy network at time () affects the state at time (). The policy network calculates the speed at time )

[0066] No. Moment reward value Designed as:

[0067] (19)

[0068] in This is the reward coefficient; As a reward constant; Is the ( Total potential energy at any given moment; It is the first Total potential energy at any given moment;

[0069] Step (1.11): Repeat steps (1.1) to (1.10) until the agent reaches the target point.

[0070] 2. The path planning method based on the combination of neural network and artificial potential field method according to claim 1, characterized in that the confidence region policy optimization method uses three policy networks, the training methods of the three policy networks are consistent, and the training steps are as follows:

[0071] Step (2.1): Using random parameters Initialize policy network Actor ;

[0072] Where parameters These are the weights and biases of the policy network; For parameters The policy network output in state Take each action below The conditional probability;

[0073] Step (2.2): Using random parameters Initialize Value Network Critic ;

[0074] Where parameters These are the weights and biases of the value network; For parameters Value network acquisition status And return the corresponding expected reward;

[0075] Step (2.3): Based on the current strategy, perform a complete path planning operation, generating N actions. The action sequence is as follows:

[0076] (20)

[0077] in, It is the first The state vector at time t, i.e., the state vector at time t, is derived from the state vector at time t Potential energy at time ( ), the ( ) Rewards at the moment, the first The state vector formed by the arrangement of gravitational and repulsive forces at each moment; It is the first The action at time state, i.e., the first Moment velocity components The two included angles and the magnitude of the velocity; Is the ( The state of observation at any given moment; yes Transform to The reward; Is the ( Actions at a given moment and state; Is the ( The state vector at time ) Is the ( Actions at a given moment and state; Is the ( The state vector at time ) yes Transform to The reward;

[0078] Step (2.4): Calculate the agent's reward value and advantage value;

[0079] No. The reward value of the agent at any given moment for:

[0080] (twenty one)

[0081] in It is the attenuation factor; For the first Momentary rewards;

[0082] No. Advantage value of the agent at any moment for:

[0083] (twenty two)

[0084] Step (2.5): Calculate the loss function of the value network Critic and update the value network parameters. ;

[0085] Value network loss function for:

[0086] (twenty three)

[0087] in For the first The reward value at any given moment; For the first The state vector at any given time; In the state of Expected return of the next item;

[0088] Use gradient descent to update parameters. for:

[0089] (twenty four)

[0090] (25)

[0091] in For learning rate, The parameters of the value network were updated before the parameters were updated. The parameters of the value network after the parameter update; In order to be in In the case of Perform parameter calculation The partial derivative;

[0092] Step (2.6): Calculate the entropy loss function in the policy network;

[0093] Entropy loss function for:

[0094] (26)

[0095] in In the first Moment State Execution of action number Moment Action variance over time; Pi; ln() is the natural constant; ln() is the natural constant. The natural logarithm function with base 0;

[0096] Step (2.7): Within the confidence region, the original complex old policy is approximated by the new policy, the new policy is adopted as the current policy, and the loss function of the policy network Actor is calculated.

[0097] Loss function of policy network for:

[0098] (27)

[0099] in These are the old strategy network parameters, which approximate the previous network parameters. These are the network parameters of the current policy, i.e., the network parameters after approximation. For the first Actions at any given moment; This is the current policy network. It is an old policy network; The weights for entropy loss; For the first The advantage value at any given moment;

[0100] The loss function is constrained by the confidence region, and can be expressed using the Kullback-Leibler divergence as follows:

[0101] (28)

[0102] in To limit the deviation between the current strategy and the old strategy to the greatest extent; Used to measure the difference between the probability distributions of the current policy and the old policy; for:

[0103] (29)

[0104] in and These are the current policy network's... The mean and variance of the output of each action; and These are the old policy network's first... The mean and variance of the output of each action;

[0105] Step (2.8): Calculate the policy gradient and update the parameters of the policy network. ;

[0106] Calculate the policy gradient for:

[0107] (30)

[0108] Update strategy network parameters :

[0109] (31)

[0110] in for The inverse matrix; for Transpose of a matrix; Let Hessian matrix be the Kullback-Leibler divergence between the current policy and the old policy. for:

[0111] (32)

[0112] To Find the partial derivative; To Find the two partial derivatives;

[0113] Step (2.9): Pass the current policy network parameters to the old policy network parameters; then process the old policy network parameters. Update to

[0114] (32)

[0115] Step (2.10): Repeat steps (2.3) to (2.9), within the confidence region constraints, continuously approximate the old policy using the current policy, and use the network parameters of the current policy. Changes come and go, updating old strategy network parameters Until the old strategy network parameters are adjusted. It stops changing and reaches its optimal state.

[0116] The present invention has the following advantages and effects compared with the prior art:

[0117] (1) In the identification of obstacles and target points, Faster R-CNN based on the GoogLeNet backbone network has higher accuracy than Faster R-CNN based on the VGG backbone network.

[0118] (2) In the artificial potential field method, the agent determines the speed only based on the resultant force, which is insufficient for decision-making. Therefore, the agent may encounter local optima and unreachable target points. However, in path planning that combines the artificial potential field method with the confidence region policy optimization method, the policy network can make up for the agent's insufficient decision-making ability in the artificial potential field method. Under the same environment, the agent makes more reasonable decisions and can solve the problems of local optima and unreachable target points.

[0119] (3) In the confidence region policy optimization method, the agent acquires information by interacting with the environment and then determines the next decision based on that information. Therefore, the key to judging the quality of the next decision is the amount of information the agent acquires. In path planning that combines the artificial potential field method with the policy network in the confidence region policy optimization method, the introduction of an artificial potential field environment allows the agent to acquire more information, enhancing its ability to interact with and perceive the environment. The more environmental information acquired, the more comprehensive and reasonable the agent's decision-making can be, making the agent suitable for path planning in unknown and complex environments.

[0120] (4) The confidence region policy optimization method is easier to converge and trains more smoothly. In the loss function of the policy network, the entropy loss is calculated. During training, when the agent is unsure of its next action, the entropy value is high, increasing the agent's uncertainty and encouraging exploration. This enhances the agent's decision-making ability when it doesn't know what action to take next. It helps the agent escape local optima and enter the global optimum. Attached Figure Description

[0121] Figure 1 This is an overall flowchart of the path planning method of the present invention.

[0122] Figure 2 This is a diagram of the Faster R-CNN framework containing the GoogLeNet backbone network of the present invention.

[0123] Figure 3 This is the backbone network diagram of the GoogLeNet method of this invention.

[0124] Figure 4 This is a diagram of the Inception module of the method of the present invention. Detailed Implementation

[0125] This invention proposes a path planning method based on a combination of neural networks and artificial potential field method, which is described in detail below with reference to the accompanying drawings:

[0126] Figure 1 This is an overall flowchart of the path planning method of the present invention. First, the agent takes real-time photos of the environment using a camera. Then, the obtained images are used to identify obstacles and target points in the images and determine their positions. Next, based on the obstacles and target points in the images and the obstacles and target points in the environment, the displacement of the agent relative to the obstacles and target points is measured using infrared ranging. Then, based on the displacement of the agent relative to the obstacles and target points, the potential energy of the agent, the attraction of the target point, and the repulsive force of the obstacle are calculated. Then, based on the attraction of the target point and the repulsive force of the obstacle, the net force and velocity component of the agent are calculated using the artificial potential field method. Simultaneously, the agent's potential energy, reward, attraction of the target point, and repulsive force of the obstacle are arranged into a state vector, and the velocity component of the agent is calculated using a policy network. Then, the velocity component obtained by the artificial potential field method is added to the velocity component vector obtained by the policy network to obtain the final velocity of the agent. Then, within a time step, the agent travels a certain displacement. This displacement then alters the environment, changing the agent's relative position to the target point and obstacles. The agent is then rewarded based on this environmental change. Finally, this reward is input into the state vector, influencing the determination of velocity.

[0127] Figure 2This is a diagram of the Faster R-CNN framework with a GoogLeNet backbone, based on the method of this invention. The Faster R-CNN with a GoogLeNet backbone consists of an image, a GoogLeNet backbone, a region generation network, a classifier, and a regressor. First, the image obtained by taking a picture is input into the GoogLeNet backbone. Then, the GoogLeNet backbone extracts features from the image to obtain a feature map. Next, the feature map is input into the region generation network. In the region generation network, the feature map undergoes a 3×3 convolution followed by two 1×1 convolutions to generate candidate boxes. Then, the candidate boxes are combined with the feature map to generate candidate boxes on the feature map. Next, the feature map with candidate boxes is input into the region of interest pooling layer. Then, max pooling is performed on the feature map within the candidate boxes to obtain several feature matrices. Then, global average pooling is performed on these feature matrices to obtain a feature vector. Finally, the feature vector is passed through different fully connected layers for activation functions and bounding box regression operations to classify objects within the candidate boxes and accurately locate the candidate boxes.

[0128] Figure 3 This is the backbone network diagram of GoogLeNet according to the method of this invention. The structure of the GoogLeNet backbone network is as follows: First, an image input layer is passed; then, a 7×7 convolutional layer with a stride of 2 is passed; then, a 3×3 max pooling layer with a stride of 2 is passed; then, a local response normalization layer is passed; then, a 1×1 convolutional layer with a stride of 2 is passed; then, a 3×3 convolutional layer with a stride of 1 is passed; then, a local response normalization layer is passed; then, a 3×3 max pooling layer with a stride of 2 is passed; then, two Inception modules are passed; then, a 3×3 max pooling layer with a stride of 2 is passed; then, five Inception modules are passed; then, a 3×3 max pooling layer with a stride of 2 is passed; finally, two Inception modules are passed to output the feature map.

[0129] Figure 4 This is a diagram of the Inception module of the method of this invention. First, the feature matrix extracted from the previous layer is input. Then, it is input into a 1×1 convolutional layer with a stride of 1 and a 3×3 max-pooling layer with a stride of 1, respectively, to reduce the dimensionality of the feature matrix. Next, the feature matrix is ​​input into a 1×1 convolutional layer with a stride of 1, a 3×3 convolutional layer with a stride of 1, and a 5×5 convolutional layer with a stride of 1; through convolutional kernels of different sizes, richer features are extracted. Finally, the rich feature layers are deeply concatenated.

[0130] The above description is only a preferred embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A path planning method based on a combination of neural networks and artificial potential field method; in obstacle and target point recognition, Faster R-CNN based on the GoogLeNet backbone network has higher accuracy than Faster R-CNN based on the VGG backbone network; in path planning combining artificial potential field method and confidence region policy optimization method, the policy network in the confidence region policy optimization method can compensate for the insufficient decision-making ability of the agent in the artificial potential field method, and the agent makes more reasonable decisions in the same environment; by introducing the environment of the artificial potential field, the agent can obtain more information and enhance the agent's interaction and perception capabilities with the environment; the policy network is trained using the confidence region policy optimization method, which does not require careful parameter tuning and the training is more stable; it can effectively solve the problems of local optima and target unreachability, making the agent suitable for path planning in unknown and complex environments; the steps in the process are as follows: Step (1.1): The intelligent agent takes real-time photos of the environment using a camera to obtain images at each moment; Step (1.2): For each time step, obstacles and target points are identified in the image using Faster R-CNN with a backbone network containing GoogLeNet; The specific process of object detection using Faster R-CNN with a GoogLeNet backbone network is as follows: The image obtained by the camera taking real-time pictures of the environment is used to extract features through the GoogLeNet backbone network to obtain a feature map; then, a region generation network is used on the feature map to generate candidate boxes. The candidate boxes generated by the region generation network are projected onto the feature map to obtain the corresponding feature matrix. The feature matrix is ​​scaled to 7×7 size by the region of interest pooling layer. It is then further scaled into feature vectors by the global average pooling layer. Finally, it is passed through a fully connected layer to classify obstacles and target points and adjust the position of candidate boxes; that is, to detect the position of target points and obstacles. The backbone network of GoogLeNet consists of an image input layer, a 7×7 convolutional layer with a stride of 2, a 3×3 max pooling layer with a stride of 2, a local response normalization layer, a 1×1 convolutional layer with a stride of 2, a 3×3 convolutional layer with a stride of 1, a local response normalization layer, a 3×3 max pooling layer with a stride of 2, two Inception modules, a 3×3 max pooling layer with a stride of 2, five Inception modules, a 3×3 max pooling layer with a stride of 2, and two Inception modules. In the backbone network of GoogLeNet, the Inception module consists of four neural network layers, with the first layer being the feature matrix extracted from the previous layer. The second layer consists of a 1×1 convolutional layer with a stride of 1 and a 3×3 max pooling layer with a stride of 1. The feature matrix is ​​reduced in dimensionality; the third layer consists of a 1×1 convolutional layer with a stride of 1, a 3×3 convolutional layer with a stride of 1, and a 5×5 convolutional layer with a stride of 1; rich features are extracted through convolutional kernels of different sizes; the fourth layer connects the rich features extracted by the first three layers to form a richer feature matrix. Step (1.3): The positions of the detected obstacles and target points are obtained by infrared ranging to determine the displacement of the obstacles and target points relative to the agent in the environment; Step (1.4): The number of target points is The number of obstacles is ; Calculate the potential energy of the intelligent agent in the potential energy field; No. The agent obtains the first time gravitational potential energy at each target point for: (1) in For the first Time-based intelligent agents and the first Displacement between target points; The constant of the gravitational potential field; No. The agent obtains the first time The repulsive potential energy of the obstacle for: (2) in For the first Time-based intelligent agents and the first Displacement between obstacles; The radius of influence of the repulsive field of the obstacle in the environment; The constant of the repulsive potential field; No. Total potential energy of intelligent agent at any moment for: (3) Step (1.5): Using the artificial potential field method, the displacement of the obstacle is converted into a repulsive force, and the displacement of the target point is converted into an attractive force; No. Time-based intelligent agents and the first The repulsive force between the obstacles is projected onto Angles on a plane for: (4) No. Time-based intelligent agents and the first The repulsive force between the obstacles and Angle of axis for: (5) in For the first Time-based intelligent agents and the first The displacement between the obstacles The size of the projection onto the plane; For the first Time-based intelligent agents and the first The displacement between the obstacles The size of the projection on the axis; ( ) represents the inverse cosine function; No. The agent is subject to the first The magnitude of the repulsive force of the obstacle for: (6) in For the first The mass of the obstacle; For the quality of the intelligent agent; It is the repulsion coefficient; No. Time-based intelligent agents and the first The gravitational projection between the target points is Angles on a plane for: (7) No. Time-based intelligent agents and the first The gravitational force between the target points and Angle of axis for: (8) in For the first Time-based intelligent agents and the first The displacement between the target points The size of the projection onto the plane; For the first Time-based intelligent agents and the first The displacement between the target points The size of the projection on the axis; No. The agent is subject to the first The gravitational force at each target point for: (9) in It is the gravitational coefficient; No. Time-based intelligent agents and the first Gravity at each target point for: (10) in for In the Time projection on The magnitude of gravity on a plane; for Unit vector along the axial direction; for The unit vector along the axial direction; for The unit vector along the axial direction; ( ) represents the cosine function; ( ) represents the sine function; No. Total Gravity of Intelligent Agents at All Times for: (11) No. Time-based intelligent agents and the first The repulsive force of the obstacle for: (12) in for In the Time projection on The magnitude of gravity on a plane; No. Total repulsive force of intelligent agents at all times for: (13) Step (1.6): Based on the attraction and repulsion forces, calculate the first... The velocity components of the agent at any given moment ; No. The velocity components of the agent at any given moment for: (14) in For the ( At time ), the velocity components calculated using the artificial potential field method; For time step; For the first intelligent agent acceleration at any moment for: (15) Step (1.7): [The text appears to be incomplete and contains several grammatical errors. A more accurate translation would require the full context.] Potential energy at time, the ( Rewards at the moment, the first The gravitational and repulsive forces at each moment are arranged into a state vector, which is input into the policy network trained by the confidence region policy optimization method to calculate the velocity components. Projected on The angle between the planes, and The included angle and velocity magnitude of the axis are used to obtain the first value under the confidence region policy optimization method. The velocity components of the agent at any given moment ; No. State vector at time step for: (16) in For the ( Rewards at certain moments; It is the first Total potential energy at any given moment; For the first The agent moves from the first target point to the second target point. The gravitational pull of a target point; For the first The agent moves from the first obstacle to the second obstacle. The repulsive force of an obstacle; In the confidence region policy optimization method, in order to obtain the velocity component Capable of representing arbitrary directions, the agent's action space employs a continuous action space; within this continuous action space, the action probability distribution adopts an expected value of... variance is A one-dimensional Gaussian distribution; a complete representation of velocity components. To use a projection The angle between the planes, and a with The included angle and velocity components of the axis The velocity components are represented by three quantities: magnitude, size, and so on. Therefore, a three-policy network is used to distribute and represent the velocity components. The two included angles and their size; The first State vector at time step Input three policy networks to obtain velocity components. The two included angles and the magnitude of the velocity, i.e., the complete velocity components. ; Step (1.8): The velocity obtained using the traditional artificial potential field method The velocity obtained by the confidence region policy optimization method under the potential energy field Add them together to obtain the intelligent agent. Total velocity at time 1 for: (17) Step (1.9): In Within a given time period, the intelligent agent travels a certain distance. No. Always Displacement within a time period for: (18) Step (1.10): As the agent's displacement relative to the target point and obstacles changes, its potential energy also changes, and the agent obtains the first... The reward at the moment, as the ( The state of the policy network at time () affects the state at time (). The policy network calculates the speed at time ) No. Moment reward value Designed as: (19) in This is the reward coefficient; As a reward constant; Is the ( Total potential energy at any given moment; It is the first Total potential energy at any given moment; Step (1.11): Repeat steps (1.1) to (1.10) until the agent reaches the target point.

2. The path planning method based on the combination of neural network and artificial potential field method according to claim 1, characterized in that, The confidence region policy optimization method described above uses three policy networks. The training methods for the three policy networks are consistent, and the training steps are as follows: Step (2.1): Using random parameters Initialize policy network Actor ; Where parameters These are the weights and biases of the policy network; For parameters The policy network output in state Take each action below The conditional probability; Step (2.2): Using random parameters Initialize Value Network Critic ; Where parameters These are the weights and biases of the value network; For parameters Value network acquisition status And return the corresponding expected reward; Step (2.3): Based on the current strategy, perform a complete path planning operation, generating N actions. The action sequence is as follows: (20) in, It is the first The state vector at time t, i.e., the state vector at time t, is derived from the state vector at time t Potential energy at time ( ), the ( ) Rewards at the moment, the first The state vector formed by the arrangement of gravitational and repulsive forces at each moment; It is the first The action at time state, i.e., the first Moment velocity components The two included angles and the magnitude of the velocity; Is the ( The state of observation at any given moment; yes Transform to The reward; Is the ( Actions at a given moment and state; Is the ( The state vector at time ) Is the ( Actions at a given moment and state; Is the ( The state vector at time ) yes Transform to The reward; Step (2.4): Calculate the agent's reward value and advantage value; No. The reward value of the agent at any given moment for: (21) in It is the attenuation factor; For the first Momentary rewards; No. Advantage value of the agent at any moment for: (22) Step (2.5): Calculate the loss function of the value network Critic and update the value network parameters. ; Value network loss function for: (23) in For the first The reward value at any given moment; For the first The state vector at any given time; In the state of The expected reward of the next return; Use gradient descent to update parameters. for: (24) (25) in For learning rate, The parameters of the value network were updated before the parameters were updated. The parameters of the value network after the parameter update; In order to be in In the case of Perform parameter calculation The partial derivative; Step (2.6): Calculate the entropy loss function in the policy network; Entropy loss function for: (26) in In the first Moment State Execution of action number Moment Action variance over time; Pi; ln() is the natural constant; ln() is the natural constant. The natural logarithm function with base 0; Step (2.7): Within the confidence region, the original complex old policy is approximated by the new policy, the new policy is adopted as the current policy, and the loss function of the policy network Actor is calculated. Loss function of policy network for: (27) in These are the old strategy network parameters, which approximate the previous network parameters. These are the network parameters of the current policy, i.e., the network parameters after approximation. For the first Actions at any given moment; This is the current policy network. It is an old policy network; The weights for entropy loss; For the first The advantage value at any given moment; The loss function is constrained by the confidence region, and can be expressed using the Kullback-Leibler divergence as follows: (28) in To limit the deviation between the current strategy and the old strategy to the greatest extent; Used to measure the difference between the probability distributions of the current policy and the old policy; for: (29) in and These are the current policy network's... The mean and variance of the output of each action; and These are the old policy network's first... The mean and variance of the output of each action; Step (2.8): Calculate the policy gradient and update the parameters of the policy network. ; Calculate policy gradient for: (30) Update strategy network parameters : (31) in for The inverse matrix; for Transpose of a matrix; Let Hessian matrix be the Kullback-Leibler divergence between the current policy and the old policy. for: (32) To Find the partial derivative; To Find the two partial derivatives; Step (2.9): Pass the current policy network parameters to the old policy network parameters; then process the old policy network parameters. Update to (32) Step (2.10): Repeat steps (2.3) to (2.9), within the confidence region constraints, continuously approximate the old policy using the current policy, and use the network parameters of the current policy. Changes come and go, updating old strategy network parameters Until the old strategy network parameters are adjusted. It stops changing and reaches its optimal state.

Images