A multi-target continuous control method for autonomous path planning and obstacle avoidance based on DDPG

Abstract

The application provides a DDPG-based autonomous path planning and obstacle avoidance multi-target continuous control method, which can improve model stability, improve feature expression capability, and is suitable for more application scenarios. The method does not treat the policy network and the value network as black boxes, but improves the structure of the policy network and the value network according to specific problems occurring in a complex environment, effectively improves the expression capability of multi-dimensional parameters, and makes the method especially suitable for application scenarios of path planning and obstacle avoidance in a complex environment.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent traffic control, in particular to a multi-target continuous control method for autonomous path planning and obstacle avoidance based on DDPG. BACKGROUND

[0002] Deep Deterministic Policy Gradient (DDPG) algorithm is a reinforcement learning algorithm based on actor-critic architecture, which uses two neural networks: policy network (Actor) and value network (Critic). It is applied to the environment with continuous action space, does not need to know the environment model in advance, and can learn directly from experience data. The policy network (Actor) is responsible for outputting the corresponding action according to the current state, and the value network (Critic) is responsible for evaluating the value of taking a specific action in the current state. DDPG algorithm updates the weights of the two neural networks continuously, so that the policy network can output the action close to the optimal action, that is, the Actor outputs an action A, which is input into the Critic to obtain the maximum Q value, and the value network can accurately evaluate the value of the action. The training process of DDPG algorithm is as follows: initialization: initialize the weights of the policy network and the value network; exploration: generate actions using random actions or exploration strategies, and execute the actions to observe the feedback of the environment; experience replay: store the state, action, reward, and next state into the experience replay pool; training: randomly sample from the experience replay pool, and update the weights of the policy network and the value network; repeat the steps of exploration, experience replay and training until the policy network Actor converges to complete the training. DDPG algorithm shows significant advantages in continuous action space control scenarios, so it is widely used in various types of agents, such as path planning of robots, intelligent cars, drones and other devices.

[0003] However, in the practical application of the DDPG algorithm in complex scenarios, such as path planning and obstacle avoidance scenarios of intelligent vehicles on actual urban roads, it is found that when the multi-dimensional state parameters are directly input into a single fully connected layer in the Actor network of the traditional DDPG algorithm, different types of features such as "self speed" and "obstacle distance" are prone to interfere with each other, such as the dimensional difference of speed suppressing the obstacle risk signal, which may cause insufficient feature expression capability in some scenarios; the single Critic of the traditional DDPG may overestimate the Q value in some cases, leading to the wrong selection of action biased towards "false high value", and thus affecting the accuracy of the final planning result; in the traditional DDPG algorithm, Ornstein-Uhlenbeck (OU) noise with fixed parameters is usually used to enhance exploration, which may cause the problem that the exploration behavior is irrelevant to the scene in some scenarios, such as still exploring a lot of steering away from obstacles, leading to path oscillation and invalid exploration; these problems may cause the planning result of the algorithm in complex scenarios to be unreliable, leading to the inability to provide the shortest and lowest carbon path for travelers, which does not meet the current concept of green travel of intelligent transportation. SUMMARY

[0004] In order to solve the problem of insufficient stability of the planning result of the path planning and obstacle avoidance method based on the traditional DDPG algorithm in complex scenarios, the present application provides a DDPG-based autonomous path planning and obstacle avoidance multi-objective continuous control method, which can improve the feature expression capability and improve the model stability, and can plan the shortest and lowest carbon environmentally friendly path on the basis of obstacle avoidance in complex scenarios.

[0005] The technical solution of the present application is as follows: a DDPG-based autonomous path planning and obstacle avoidance multi-objective continuous control method, characterized in that it comprises the following steps:

[0006] S1: mathematically modeling the path planning and obstacle avoidance problem based on the DDPG algorithm;

[0007] S2: constructing a strategy network supporting multi-modal fusion;

[0008] The structure of the strategy network comprises a strategy network input layer, a strategy network feature extraction layer, a strategy network attention fusion layer, a strategy network hidden layer and a strategy network output layer connected in sequence

[0009] The input of the strategy network is a multi-dimensional state parameter S;

[0010] The multi-dimensional state parameter S comprises self state parameters, target information parameters and obstacle information parameters;

[0011] The policy network feature extraction layer includes three parallel feature extraction branches: self-state feature extraction branch, target direction feature extraction branch and obstacle risk feature extraction branch;

[0012] The policy network input layer splits the parameters included in the multi-dimensional state parameter S according to types, and inputs the parameters into corresponding feature extraction branches respectively. cmd And ω cmd ;

[0013] S3: constructing a double Q-value value network;

[0014] The value network includes a value network input layer, a value network feature extraction layer, a value network fusion layer, a value network hidden layer and a value network output layer connected in sequence.

[0015] The value network feature extraction layer includes four parallel feature extraction branches: self-state feature extraction branch, target direction feature extraction branch, obstacle risk feature extraction branch and action feature extraction branch.

[0016] The value network hidden layer includes two parallel main value calculation branches with the same structure.

[0017] The input of the value network includes a multi-dimensional state parameter S and a 2-dimensional action v cmd And ω cmd output by the policy network.

[0018] The value network input layer splits the parameters included in the multi-dimensional state parameter S into self-state parameters, target information parameters and obstacle information parameters, and inputs the parameters into parallel self-state feature extraction branch, target direction feature extraction branch and obstacle risk feature extraction branch respectively.

[0019] The outputs of the four feature extraction branches are simultaneously input into the value network fusion layer for splicing of state features and action features. target

[0020] ​S4: constructing a path planning and obstacle avoidance network model based on the policy network supporting multi-modal fusion and the value network;

[0021] S5: constructing a training sample data set and a verification sample data set, training the path planning and obstacle avoidance network model, and obtaining a trained path planning and obstacle avoidance network model;

[0022] S6: based on the trained path planning and obstacle avoidance network model, realizing autonomous path planning and obstacle avoidance multi-objective continuous control of the agent.

[0023] It is further characterized in that:

[0024] In step S1, the following steps are specifically included:

[0025] S11: environment modeling;

[0026] The agent motion scene is defined as a continuous plane space, including a starting point A(x A ,y A ) and an ending point B(x B ,y B ) supporting configuration; the obstacles include static obstacles and dynamic obstacles; the minimum safety distance between the agent center and the obstacle boundary is defined as d safe ;

[0027] S12: constructing a state space;

[0028] Define a multi-dimensional state parameter S=[x curr ,y curr ,θ curr ,v curr ,ω curr ,△x AB ,△y AB ,d obs1 ,φ obs1 ,d obs2 ,φ obs2 ];

[0029] Wherein, (x curr ,y curr ) is the real-time position coordinate of the agent, θ curr is the heading angle of the agent, v curr is the real-time motion speed, ω curr is the real-time angular velocity; (△x AB ,△y AB ) is the relative coordinate of the current position and the ending point B;

[0030] △x AB =x B -x curr ,△y AB =yB -y curr ;

[0031] d obs1 and d obs2 is the straight-line distance of the closest two obstacles; φ obs1 and φ obs2 is the azimuth angle of the closest two obstacles relative to the agent's heading angle, φ ∈ [-π, π];

[0032] S13: the state update method is constructed as:

[0033] x curr = v cmd ·cosθ curr ; y curr = v cmd ·sinθ curr ; θ curr = ω cmd ;

[0034] x next = x curr + v cmd ·cosθ curr ·△t; y next = y curr + v cmd ·sinθ curr ·△t; θ next = θ curr + ω cmd ·△t;

[0035] wherein v cmd is the speed in the control action, ω cmd is the angular speed in the control action, △t is the interval time between the current state and the next state, x next is the x coordinate of the next state position, y next is the y coordinate of the next state position, and θ next is the heading angle of the next state;

[0036] Each state parameter s included in the multi-dimensional state parameter S is subjected to a state standardization operation before being input into the network:

[0037] s norm = (s-μ s ) / (σ s +ε);

[0038] wherein s norm is the characteristic value of each parameter s after the standardization operation, μ s is the mean value of the training set state corresponding to each parameter s, and σ sThe training set state standard deviation corresponding to the parameter s is ε=10 -6 ;

[0039] The operation in the attention fusion layer in the policy network includes:

[0040] The features output by the multiple branches in the policy network feature extraction layer are weighted and fused through the attention mechanism to obtain the fused features f fso ;

[0041] f fso =w s ·f s +w g ·f g +w o ·f o ;

[0042] Wherein, f s , f g , f o are the feature vectors of the state, target information and obstacle information respectively;

[0043] w s is the state weight, w g is the target information weight, and w o is the obstacle information weight, satisfying the condition w s +w g +w o =1;

[0044] ;

[0045] ;

[0046] The policy network hidden layer of the policy network is realized based on a full connection layer and a ReLU function;

[0047] The policy network output layer is a full connection layer, and the 64-dimensional features are linearly transformed through the full connection layer, mapped to [-1, 1] through tanh, and then scaled to the physical action range;

[0048] ;

[0049] In the formula, v min and ω min are the preset minimum speed and angular velocity for avoiding stagnation, v max and ω max are the preset maximum speed and maximum angular velocity; tanh() is the hyperbolic tangent function; o v and o ω are the original values of the policy network output layer;

[0050] In the hidden layer of the value network, the main value calculation branch is implemented based on a full connection layer cooperating with an activation function ReLU;

[0051] The output layer of the value network comprises two parallel single neuron full connection layers, and the output of each main value calculation branch is respectively input into a single neuron full connection layer for linear mapping, and a one-dimensional Q value is directly output, and then the minimum Q value is obtained as Q target ;

[0052] Q target =min (Q1’(s’,a’),Q2’(s’,a’));

[0053] Q1’(s’,a’) and Q2’(s’,a’) are two Q values output by the hidden layer of the value network, s’ is a state corresponding to the i-th action performed by the agent, and a’ is an action vector obtained by the state s’ based on the policy network Actor;

[0054] In step S5, the training method of the path planning and obstacle avoidance network model comprises:

[0055] a1: constructing a multi-objective reward function r;

[0056] r = r goal +r obstacle +r optimal +r terminal ;

[0057] In the formula, r goal is a target approach reward, r obstacle is an obstacle avoidance safety reward, r optimal is a path optimization reward, and r terminal is a terminal reward;

[0058] r goal =k1·(d prev -d curr ) / d max -k2·d curr ;

[0059] In the formula, d curr is the distance from the current position to the terminal point B; d curr 2 =△x AB 2 +△y AB 2 ;

[0060] d prev is the distance from the previous step to the terminal point; and d max is the initial distance from the starting point to the terminal point;

[0061] k1 and k2 are hyperparameters, obtained by training, k1 is a weight parameter for rewarding approaching the target, and k2 is a weight parameter for punishing long distance;

[0062] ;

[0063] wherein d obs,min is the distance to the nearest obstacle, d obs,min =min(d obs1 ,d obs2 );d obs1 and d obs2 are the straight-line distances to the nearest two obstacles;

[0064] k3 is a safety distance control parameter, k3 is a hyperparameter, obtained by training;

[0065] r optimal =-k4·v cmd 2 -k5·|ω cmd |-k6·△d detour ;

[0066] wherein k4 is an energy consumption penalty parameter, k5 is a steering penalty parameter, and k6 is a penalty for detouring parameter;

[0067] △d detour is the detour distance, △d detour =d curr +d path,accum -d AB ;

[0068] d path,accum is the cumulative moving distance of the agent from the starting point to the current position, d AB is the straight-line distance between the starting point A and the ending point B;

[0069] ;

[0070] wherein ε is the minimum value of the distance from the current position to the ending point B, used to mark the reaching of the ending point;

[0071] ;

[0072] wherein v cmd,k is the linear velocity of the k-th step, △t is the preset interval time between two adjacent states; t is the current training step number;

[0073] a2: the input of the policy network is: multi-dimensional state parameter S, and the output is ω cmd and v cmd ;

[0074] The input s of the policy network is jointly constructed with its corresponding output as a mapping function π(s);

[0075] When the agent perceives the current environment state as s, the output control instruction ω of the policy network corresponding to the mapping function π(s) is directly obtained cmd and v cmd ;

[0076] a3: divide each cycle of rounds into a hybrid guidance training phase and an adaptive exploration phase;

[0077] a4: in the hybrid guidance training phase, use A * algorithm to generate an obstacle avoidance path from the starting point A to the ending point B in the grid map, extract the key nodes on the path, and add the distance d node from the current position to the nearest key node in the multi-dimensional state parameter S, and increase the auxiliary reward r guide for moving to the key node in the reward function;

[0078] Total reward r' = r + r guide = r goal + r obstacle + r optimal + r terminal + r guide ;

[0079] r guide = -k7·d node ;

[0080] In the formula, k7 is the reward parameter for moving away from the key node;

[0081] a5: in the adaptive exploration phase, remove the A * key node guidance, delete the auxiliary reward r guide , keep the core reward function r, and adopt adaptive noise intensity σ(t);

[0082] Exploration action a explore = π(s) + N t ;

[0083] ;

[0084] ;

[0085] In the formula, σ0 is the initial noise intensity, T is the decay period, α is the obstacle distance coefficient, t is the current training step number, d obs,min is the distance to the nearest obstacle, d safe is the minimum safety distance between the agent center and the obstacle boundary; θ ou is the OU noise regression coefficient, μou is the noise mean, W t is a random distributed Gaussian white noise W t ~N(0, 1); N t is an adaptive OU noise;

[0086] In step S5, the training method of the path planning and obstacle avoidance network model further includes the following operations:

[0087] b1: define an error δ: δ = |y i -Q i (s i ,a i )|

[0088] wherein y i is the evaluation value for the i-th state and action;

[0089] Q i (s i ,a i ) is the target Q value output by the value network for the i-th state s i and action a i ;

[0090] y i =r i +γ·Q i ·(1-done i );

[0091] wherein r i is the reward obtained by the i-th action performed by the agent; γ is a discount factor, balancing immediate and future rewards; done i is a state marker of the action-performing agent, taking the value 1 to indicate reaching the end point or collision, and taking the value 0 to indicate continuing to run;

[0092] b2: optimize the priority experience replay PER, and integrate the optimized PER into the training process of the path planning and obstacle avoidance network model;

[0093] The adjustment weight method adopted by the PER includes:

[0094] Collision avoidance experience weight: if the distance d obs,min <the minimum safety distance d safe in the experience is greater than 0 and no collision done i = 0, then δ × 2;

[0095] Target fast approach experience weight: if d prev -d curr > single-step approach target threshold in the experience, then δ × 1.5;

[0096] Path optimization experience: if the experience distance △d detour If the distance exceeds the optimization threshold, then δ×1.2;

[0097] Wherein, d prev is the straight-line distance from the agent in the last control cycle to the end point B;

[0098] After the strategy network outputs the action, a collision prediction module is added, and collision prediction and action correction are performed based on the collision prediction module; the collision prediction module specifically includes the following operations:

[0099] c1: the collision prediction module predicts the position (x', y') of the agent at a future time based on the current action (v cmd , ω cmd );

[0100] ;

[0101] In the formula, θ curr is the heading angle state of the agent at the current time, △t pred is the prediction future time window, and △t pred > △t;

[0102] c2: calculate the distance between the predicted position (x', y') and all obstacles, find the closest obstacle, and record the distance between the closest obstacle and (x', y') as d';

[0103] ;

[0104] In the formula, (x obsk , y obsk ) is the center coordinate of the kth obstacle; r robot is the radius of the agent, and r obsk is the radius of the kth obstacle;

[0105] c3: read the preset minimum safety distance d safe between the center of the agent and the boundary of the obstacle, and compare d' and d safe ;

[0106] If d' < d safe , action correction logic is triggered, and step c4 is executed;

[0107] Otherwise, step c5 is executed;

[0108] c4: calculate the corrected steering angle ω cmd,corrected and the corrected speed v cmd,corrected , then assign ω cmd,corrected to ω cmd , and assign v cmd,corrected to v cmd, execute step c5;

[0109] ;

[0110] wherein sign (φ) is the sign of the obstacle azimuth, 1 when φ>0, and -1 when φ<0; k ω,corr is the angular velocity correction coefficient; k v,corr is the linear velocity correction coefficient; φ obsk，min is the azimuth of the nearest obstacle relative to the heading of the agent at the (x', y') position;

[0111] φ obsk =arctan2(y obsk –y',x obsk –x') - θ';

[0112] wherein θ' is the heading angle state of the agent at the (x', y') position;

[0113] c5: smoothing the velocity and the heading angle, output the smoothed velocity v cmd,smoothed and the smoothed heading angle ω cmd,smoothed ;

[0114] v cmd,smoothed =β*v cmd +(1-β)*v cmd , prev ;

[0115] ω cmd,smoothed =β*ω cmd +(1-β)*ω cmd , prev ;

[0116] wherein β is the filter coefficient, (v cmd , prev , ω cmd , prev ) is the smoothed action of the previous step;

[0117] In step S5, in the path planning and obstacle avoidance network model training process, the target network updating method is:

[0118] An adaptive parameter τ is designed to realize the optimization of the target network through adaptive soft updating,

[0119] The adaptive parameter τ is:

[0120] τ=τ min +(τ max -τ min )·exp (-epoch / epoch0);

[0121] wherein epoch is a training round, epoch0 is a baseline parameter, τ max and τ min are preset maximum and minimum values of τ.

[0122] The application provides a DDPG-based autonomous path planning and obstacle avoidance multi-target continuous control method, which expands the state dimension, uses a multi-dimensional state parameter S as a network input parameter, covers the self state, target information and obstacle perception information, ensures that the network comprehensively obtains decision basis, and improves the accuracy of the final output of the model; the application improves the policy network and the value network in the DDPG algorithm, sets multiple branch feature extraction branches in the policy network and the value network respectively for the multi-dimensional input state parameter S, respectively inputs different types of state parameters into the corresponding type of feature extraction branch, effectively avoids the mutual interference between different types of feature parameters, and improves the stability of the network model; an attention fusion network structure is arranged in the attention fusion layer of the policy network, which cooperates with the multi-branch feature extraction in the feature extraction layer, effectively improves the feature expression capability, and makes the network focus on the target and the dangerous obstacle; two main value calculation branches with the same structure but independent parameters are designed in the hidden layer of the value network, which respectively evaluate the Q value of the same group of “state and action”, and the minimum value output by the output layer is taken as the target Q value output, which effectively avoids the deviation caused by the overestimation of a single network, and further improves the stability of the network. The method divides the model training process into a hybrid guidance stage and an adaptive exploration stage. In the hybrid guidance stage, the A* algorithm is introduced, the key nodes on the path are set, the auxiliary reward of “moving to the key node” is added in the reward function, the intelligent agent quickly learns the basic behavior of “not colliding and moving to the target”, a large number of collisions caused by random exploration in the early stage are avoided, and the training efficiency is improved; in the adaptive exploration stage, the adaptive noise intensity σ(t) is introduced, so that the noise intensity can be dynamically adjusted according to the current training step t representing the training progress and the obstacle distance d, ensuring that the model can attenuate the noise in the later training stage, effectively reducing invalid exploration, and enabling the model to effectively focus on the target. Compared with other existing path planning and obstacle avoidance methods based on the DDPG algorithm, the method does not process the policy network and the value network as a black box, but improves the structure of the policy network and the value network according to specific problems occurring in a complex environment, effectively improves the expression capability of the multi-dimensional parameter, and makes the method especially suitable for application scenarios of path planning and obstacle avoidance in a complex environment, so that the method can plan the shortest and lowest-carbon environmentally friendly path on the basis of obstacle avoidance in a complex scenario. BRIEF DESCRIPTION OF DRAWINGS

[0123] Figure 1 FIG. 1 is a network structure diagram of the policy network and the value network in the prior art;

[0124] Figure 2This is a schematic diagram of the network structure of the strategy network in this application;

[0125] Figure 3 This is a schematic diagram of the network structure of the value network in this application;

[0126] Figure 4 This is a schematic diagram of the training process for the path planning and obstacle avoidance network model in this application. Detailed Implementation

[0127] like Figure 1 The diagram shows the network structure of the policy network (Actor) and value network (Critic) in the standard DDPG algorithm. The Actor and Critic networks are trained separately. The input to the Critic network is the state parameters and the action output from the Actor network. The action output from the Actor network is the input to the Critic network, and the output of the Critic network is used for backpropagation in the Actor network. However, the standard DDPG algorithm suffers from unreliable planning results when applied to complex environments, limiting its applicability in practical scenarios such as navigation in unknown environments and control of autonomous robots. This method improves the network structure of the policy network (Actor) and value network (Critic) to address the characteristics of complex environments.

[0128] This application includes a multi-objective continuous control method based on DDPG autonomous path planning and obstacle avoidance, which includes the following steps.

[0129] S1: Mathematical modeling of path planning and obstacle avoidance problems based on the DDPG algorithm.

[0130] Step S1 specifically includes the following steps:

[0131] S11: Environment Modeling;

[0132] The motion scene of the intelligent agent is defined as a continuous planar space, containing a configurable starting point A(x). A ,y A ) and endpoint B(x B ,y B Obstacles include static obstacles and dynamic obstacles; the minimum safe distance between the agent's center and the obstacle boundary is defined as d. safe Among them, dynamic obstacles carry a constant velocity (v). obsx , v obsy The state synchronously includes its speed information. Among them, (x) A ,y A ) and (x B ,y B These are all physical coordinates in planar space, directly reflecting the distance and direction between the agent and the target location.

[0133] In this embodiment, the agent is set to be a robot with a radius r robot =0.15m, and the minimum safety distance d safe between the robot center and the obstacle boundary is greater than or equal to 0.35m.

[0134] During training, static obstacles are randomly generated as circles with a radius r obs =0.2m or rectangles with a length l=0.4m and a width w=0.4m, with a number of 1-5, and the number is randomly changed during training to improve generalization.

[0135] S12: Constructing a state space;

[0136] Define multi-dimensional state parameters S=[x curr ,y curr ,θ curr ,v curr ,ω curr ,△x AB ,△y AB ,d obs1 ,φ obs1 ,d obs2 ,φ obs2 ].

[0137] This method expands the state dimension to 11 dimensions, covering self-state, target information, and obstacle perception information, ensuring that the network comprehensively obtains decision-making basis. Among them, (x curr ,y curr ) are the real-time position coordinates of the agent, θ curr is the heading angle of the agent, indicating the forward direction of the agent; v curr is the real-time motion speed, ω curr is the real-time angular velocity, indicating the turning speed of the agent.

[0138] (△x AB ,△y AB ) are the relative coordinates of the current position and the end point B, used to guide the agent to move towards the target;

[0139] △x AB =x B -x curr ,△x AB is the x-direction relative coordinate of the agent and the end point B;

[0140] △y AB =y B -y curr ,△y AB is the y-direction relative coordinate of the agent and the end point B;

[0141] d obs1 and dobs2 The straight-line distance of the closest two obstacles, used to cover the main obstacle avoidance risk;

[0142] φ obs1 and φ obs2 The azimuth angle of the closest two obstacles relative to the agent's heading, used to clarify the direction of the obstacle position, φ∈[-π,π].

[0143] In specific implementation, the collection of the parameter s in the multi-dimensional state parameter S is based on the multi-modal sensors installed on the agent, such as optical cameras, infrared sensors, radars, Beidou navigation modules, gyroscopes, accelerometers, vehicle speed sensors, compass sensors, etc. installed on intelligent vehicles. The specific collection method is realized based on existing technologies.

[0144] In this embodiment, only the collection of information about the closest two obstacles is designed. In actual application, more obstacles need to be supported, and the obstacle perception information d obs and φ obs can be expanded according to actual needs.

[0145] S13: The state update method is constructed as follows:

[0146] The agent updates from the current state (x curr , y curr, θ curr ) with the control action (v cmd , ω cmd ) as follows:

[0147] x curr =v cmd ·cosθ curr ; y curr =v cmd ·sinθ curr ; θ curr =ω cmd ;

[0148] x next =x curr +v cmd ·cosθ curr ·△t; y next =y curr +v cmd ·sinθ curr ·△t; θ next =θ curr +ω cmd ·△t;

[0149] In the formula, v cmd is the speed in the control action, and ω cmdTo control the angular velocity in the action, △t is the interval time between the current state and the next state, x next is the x coordinate of the next state position, y next is the y coordinate of the next state position, and θ next is the heading angle of the next state.

[0150] The specific calculation method of the distance and azimuth angle of the obstacle and the agent is as follows:

[0151] ;

[0152] wherein x obsk is the x coordinate of the center of the kth obstacle, y obsk is the y coordinate of the center of the kth obstacle, r robot is the radius of the agent itself, r obsk is the radius of the kth obstacle, d obsk is the distance between the robot and the kth obstacle, unit: m, and φ obsk is the azimuth angle of the kth obstacle relative to the heading direction of the robot, unit: rad, range [-π, π]. In this embodiment, the positive direction is the right side of the advancing direction of the robot.

[0153] In order to improve the network training convergence speed and avoid feature suppression caused by different dimensional differences, each state parameter s included in the multi-dimensional state parameter S in the application is subjected to a state standardization operation before being input into the network:

[0154] s norm = (s-μ s ) / (σ s +ε);

[0155] wherein s norm is the characteristic value of each parameter s after the standardization operation, μ s is the mean value of the training set state corresponding to each parameter s, σ s is the standard deviation of the training set state corresponding to the parameter s, and ε = 10 -6 to avoid the denominator being 0. μ s and σ s are obtained based on the statistics of the parameters s included in the training set.

[0156] In this embodiment, the scene is from A [0, 0] to B [10, 10], the x coordinate range is [0, 10], the mean value is taken as the midpoint of the range, and the standard deviation is taken as 1 / 3 of the range, that is, more than 95% of the sample distribution. x curr corresponding to μ s = 5.0 m, σ s = 3.0 m; y curr corresponding to μ s = 5.0 m, σs = 3.0 m. The heading angle is randomly distributed without bias, so the mean is 0 and the standard deviation is 1.0 rad, which covers most steering scenarios, such as ±1 rad corresponding to ±57°, so θ curr The corresponding μ s = 0 rad, σ s = 1.0 rad. The linear velocity is uniformly distributed within the constraint range, the mean is the midpoint of the range 0.3 m / s, and the standard deviation covering 95% of the samples is 1 / 3.3 of the range, so the current linear velocity v curr The corresponding μ s = 0.3 m / s, σ s = 0.12 m / s. The angular velocity is unbiased, the mean is 0, and the standard deviation 0.5 rad / s covers regular steering and avoids frequent large steering, so ω curr The corresponding μ s = 0 rad / s, σ s = 0.5 rad / s. The distance d obs1 The corresponding μ s = 2.0 m, σ s = 1.5 m, which is used to adapt to the scenario of “1-5 obstacles, safety distance d safe ≥ 0.35 m”. The azimuth angle is randomly distributed without fixed bias, and the standard deviation 1.2 rad covers most obstacle positions within 360°, such as ±1.2 rad corresponding to ±68.8°, so the azimuth angle φ obs The corresponding μ s = 0 rad, σ s = 1.2 rad. In this example, another obstacle is the second closest obstacle, which is usually farther than the closest obstacle, d obs2 The corresponding μ s = 2.5 m, σ s = 1.8 m.

[0157] The action space A is constructed to describe the continuous control output of the policy network. The action space is the continuous control quantity that the robot can execute, which is directly mapped to the actuator instruction, expressed as: A = [v cmd , ω cmd ]. According to the characteristics of the agent itself, set the constraint range for each agent, v cmd ∈ [v min , v max ], ω cmd ∈ [ω min , ω max ]. Among them, v min is the minimum speed to avoid stagnation, v max is the maximum speed limit energy consumption, or to ensure safety; ω min is the minimum heading angle, ω maxThe maximum heading angle is the control steering range, which is adapted to the running environment of the agent to achieve obstacle avoidance and detouring. In this embodiment, the constraints for the robot moving indoors are: v cmd ∈[0.1,0.5]m / s, ω cmd ∈[-1.5,1.5]rad / s. The action is output by the Actor network and scaled to ensure compliance with physical constraints.

[0158] The association of action and path: ω cmd decides the change of heading angle, indirectly adjusts the direction of the path, v cmd decides the moving speed, and together realizes moving towards the target and obstacle avoidance and detouring.

[0159] S2: construct a strategy network supporting multi-modal fusion;

[0160] As shown in Figure 2 , the structure of the strategy network includes: a strategy network input layer, a strategy network feature extraction layer, a strategy network attention fusion layer, a strategy network hidden layer and a strategy network output layer connected in turn

[0161] The input of the strategy network is a multi-dimensional state parameter S; the multi-dimensional state parameter S includes: self-state parameters, target information parameters and obstacle information parameters; the strategy network feature extraction layer includes three parallel feature extraction branches: self-state feature extraction branch, target direction feature extraction branch and obstacle risk feature extraction branch.

[0162] The strategy network input layer splits the parameters included in the multi-dimensional state parameter S according to the type, and obtains 3 independent input streams: the 5-dimensional self-state parameters are sent to the self-state feature extraction branch, the 2-dimensional target information parameters are sent to the target direction feature extraction branch, and the 4-dimensional obstacle information parameters are sent to the obstacle risk feature extraction branch.

[0163] The three feature extraction branches perform feature extraction, output 64-dimensional self-state features, 32-dimensional target direction features, and 64-dimensional obstacle risk features. The outputs of the three branches are simultaneously sent to the strategy network attention fusion layer for multi-branch feature weighted fusion through the attention mechanism, to obtain 128-dimensional fused features, which are sent to the strategy network hidden layer for global feature fusion. The strategy network hidden layer outputs 64-dimensional refined features, and finally the refined features are mapped to actions through the strategy network output layer, to output 2-dimensional control actions: v cmd and ω cmd .

[0164] The operations in the attention fusion layer in the strategy network include:

[0165] The features output by the multiple branches in the strategy network feature extraction layer are weighted and fused through the attention mechanism to obtain the fused features ffso ;

[0166] f fso =w s ·f s +w g ·f g +w o ·f o ;

[0167] wherein f s , f g , f o are feature vectors of self state, target information, obstacle information respectively, and the dimensions are 64, 32, 64 respectively.

[0168] w s is the self state weight, w s is the basic weight, which guarantees the smoothness of the action.

[0169] w g is the target information weight, the greater the distance d curr from the current position to the end point B, the greater w g , which realizes the priority attention to the target.

[0170] w o is the obstacle information weight, the smaller the distance d obs,min of the nearest obstacle, the greater w o , which realizes the priority attention to obstacle avoidance.

[0171] The conditions w s +w g +w o =1 are met.

[0172] ;

[0173] .

[0174] The policy network hidden layer of the policy network is realized based on a full connection layer with a nonlinear activation function ReLU function; the full connection layer adopts the classic structure of the weight matrix of the deep neural network plus the bias term.

[0175] The output layer of the policy network is a full connection layer, and after the 64-dimensional features are linearly transformed through the full connection layer, they are mapped to [-1, 1] through the tanh activation function, and then scaled to the physical action range; the number of neurons of the full connection layer is 2, corresponding to 2 control actions:

[0176] ;

[0177] In the formula, v min and ω minv max and ω max are preset minimum speed and minimum angular velocity; tanh() is hyperbolic tangent function; o v and o ω are original values of the output layer of the policy network;

[0178] In the method, a network structure of multi-branch feature extraction set attention fusion is designed in the structure of the policy network, so as to improve the feature expression capability and make the network focus on the target and dangerous obstacles.

[0179] S3: constructing a value network of double Q values;

[0180] As shown in the figure, the value network comprises, in sequence, a value network input layer, a value network feature extraction layer, a value network fusion layer, a value network hidden layer and a value network output layer. Figure 3

[0181] The value network hidden layer comprises two main value calculation branches which are arranged in parallel and have the same structure.

[0182] The value network hidden layer comprises two main value calculation branches which are arranged in parallel and have the same structure.

[0183] The input of the value network comprises 11-dimensional multi-dimensional state parameters S and 2-dimensional actions v cmd and ω cmd output by the policy network.

[0184] The value network input layer splits the parameters included in the multi-dimensional state parameters S into self-state parameters, target information parameters and obstacle information parameters, and then inputs them into the parallel self-state feature extraction branch, target direction feature extraction branch and obstacle risk feature extraction branch respectively, and inputs the 2-dimensional action parameters output by the policy network into the action feature extraction branch.

[0185] The four feature extraction branches respectively perform feature extraction, and output 64-dimensional self-state features, 32-dimensional target direction features, 64-dimensional obstacle risk features and 32-dimensional action features. The outputs of the four feature extraction branches are simultaneously input into the value network fusion layer to splice the state features and the action features, and obtain 192-dimensional spliced features. The spliced features are respectively input into the two main value calculation branches in the value network hidden layer.

[0186] In the hidden layer of the value network, the main value calculation branch is realized based on a full connection layer cooperating with an activation function ReLU; each main value calculation branch respectively outputs a 64-dimensional Q value, and the two Q values are output to the output layer. ​

[0187] The output layer of the value network comprises two parallel single neuron fully connected layers with linear mapping and without activation function design; without activation function, the original value scale needs to be preserved because the Q value is a continuous numerical value. The output of each main value calculation branch is respectively sent into a single neuron fully connected layer for linear mapping. Each set of 64-dimensional refined features is linearly transformed by a fully connected layer with 1 neuron to directly output a 1-dimensional Q value, and the minimum Q value is obtained as the output Q target .

[0188] Q target =min(Q1’(s’,a’),Q2’(s’,a’));

[0189] Q1’(s’,a’) and Q2’(s’,a’) are two Q values output by the hidden layer of the value network, s’ is a state corresponding to the i-th action performed by the agent, and a’ is an action vector obtained by the state s’ based on the policy network Actor;

[0190] The core function of the Critic network is to evaluate the value of the action output by the Actor and output the corresponding Q value. In this method, two main value calculation branches with the same structure but independent parameters are set in the hidden layer of the value network, which respectively evaluate the Q values corresponding to the same set of states s and actions a, and the smaller Q value is selected as the output, which can avoid the deviation caused by the overestimation of a single network, and thus avoid the problem of action biasing to “virtual high value”.

[0191] The standard DDPG adopts a soft update method with a fixed hyperparameter τ to control the parameter update amplitude, but in complex scenarios, it may cause the problem of slow updating of the target network. In this method, the adaptive soft update method is used to realize the optimization of the target network. An adaptive parameter τ is designed as follows:

[0192] τ=τ min +(τ max -τ min )·exp (-epoch / epoch0);

[0193] In the formula, epoch is the training round, in the training process, 1 epoch corresponds to the robot starting from the starting point A and trying to complete a complete task cycle of “autonomous path planning, obstacle avoidance, and reaching the end point B”; at the same time, a timeout time is also set to limit the rounds, that is, whether successful or not, if the timeout time is 15s or collision, the epoch is terminated.

[0194] epoch0 is a baseline parameter for controlling the rate of decay of the adaptive parameter τ with epoch, in this embodiment, epoch0=10 4 training rounds.

[0195] τ max and τ min are preset maximum and minimum values of τ, and specific values are set in advance based on the type of agent and the running environment of the agent according to experience values. In the embodiment, for a robot walking indoors, τ max = 0.01, and τ min = 0.001.

[0196] When epoch = 0, τ = 0.01, fast follow-up; when epoch = 10 4 , τ = 0.001 + 0.009 exp (-1) ≈ 0.0043, slow update.

[0197] Based on the adaptive parameter τ in the method, τ is larger at the beginning of training, the target network quickly follows the local network, and the convergence is accelerated; τ is reduced at the later stage of training, the target network is kept stable, and the final performance is improved.

[0198] S4: Construct a path planning and obstacle avoidance network model based on a policy network and a value network supporting multi-modal fusion.

[0199] Because the method is for a complex scene, the network structure of the policy network Actor and the value network Critic is improved, which can effectively improve the mapping precision of the model based on the mapping of the multi-dimensional state parameter S to the action. However, at the same time, a targeted training method must be designed for the improved network structure to ensure that the training process does not have problems such as gradient explosion and slow convergence. The specific training method is as follows.

[0200] S5: Construct a training sample data set and a verification sample data set, train the path planning and obstacle avoidance network model, and obtain a trained path planning and obstacle avoidance network model.

[0201] The application reconstructs the reward function as: target approach reward + obstacle avoidance safety reward + path optimization reward + terminal reward, balances multiple objectives through weights, guides the agent to learn the optimal path under the premise of obstacle avoidance, and realizes the planning of the shortest and lowest carbon environmental protection path on the basis of obstacle avoidance.

[0202] a1: Construct a multi-objective reward function r;

[0203] r = r goal + r obstacle + r optimal + r terminal ;

[0204] In the formula, r goal is a target approach reward, r obstacle is an obstacle avoidance safety reward, r optimal is a path optimization reward, and rterminal For terminal rewards.

[0205] Target proximity reward r goal The core orientation is to encourage movement toward the finish line and punish those who stray from it.

[0206] r goal =k1·(d prev -d curr ) / d max -k2·d curr ;

[0207] In the formula, d curr d represents the distance from the current position to the destination B. curr 2 =△x AB 2 +△y AB 2 ;

[0208] d prev d is the distance from the previous step to the destination. max The initial distance from the starting point to the finish line is used to normalize the reward.

[0209] k1 and k2 are hyperparameters obtained through training. k1 is the weight parameter for rewarding proximity to the target, and k2 is the weight parameter for penalizing distance. In this embodiment, k1=10 and k2=2, ensuring the core logic of continuously approaching the target.

[0210] Obstacle avoidance safety reward r obstacle This is used to ensure safety by punishing close contact with obstacles and rewarding safe distances. Specifically, dangerous distances are severely punished, while safe distances are lightly rewarded, thus preventing agents from risking getting too close to obstacles.

[0211] ;

[0212] In the formula, d obs,min d is the distance to the nearest obstacle. obs,min =min(d obs1 ,d obs2 );d obs1 and d obs2 The straight-line distance between the two nearest obstacles is d; obs,min ≤d safe / 2 is defined as an impending collision; d safe / 2<d obs,min ≤d safe Defined as the danger distance, d obs,min >d safe Defined as a safe distance.

[0213] k3 is a safety distance control parameter, a positive reward is given when the safety distance, encouraging to keep away from obstacles; k3 is a hyperparameter, obtained through training; in the embodiment, k3 = 3.

[0214] Path optimal reward r optimal to improve path quality, punish detour, energy consumption and frequent steering.

[0215] r optimal = -k4 · v cmd 2 -k5 · |ω cmd |-k6 · △d detour ;

[0216] In the formula, k4 is an energy consumption penalty parameter; k5 is a steering penalty parameter, avoiding frequent steering; k6 is a detour penalty parameter, guiding to approach a straight line path; k4, k5 and k6 are hyperparameters, obtained through training. In the embodiment, k4 = 0.1, k5 = 0.5 and k6 = 0.8.

[0217] △d detour is a detour distance, △d detour = d curr + d path,accum - d AB ;

[0218] d AB is the straight line distance between the starting point A and the ending point B. d curr is the straight line distance from the current agent to the ending point B.

[0219] d path,accum is the cumulative moving distance of the agent from the starting point to the current position, that is, the actual path length;

[0220] ;

[0221] In the formula, v cmd,k is the linear speed of the kth step, △t is the interval time of the adjacent two states, and t is the current training step number. In the embodiment, △t is 0.01s.

[0222] Terminal reward r terminal to strengthen the goal achievement;

[0223] ;

[0224] In the formula, ε is the minimum value of the distance from the current position to the ending point B, used to mark the reaching of the ending point, and is set to 0.1m in the embodiment; when d curr < 0.1m, it indicates that the ending point is reached. The specific timeout time is set according to the actual scene, and is set to 15s in the embodiment.

[0225] a2: the input of the policy network is a multi-dimensional state parameter S, and the output is ω cmd and v cmd ;

[0226] The input s of the policy network is jointly constructed with its corresponding output as a mapping function π(s);

[0227] When the agent perceives the current environment state as s, the output control instruction ω corresponding to the policy network is directly obtained through the mapping function π(s) cmd and v cmd .

[0228] DDPG is a model-free policy optimization algorithm based on reinforcement learning, which is used to solve the reinforcement learning problem in continuous action space. The biggest difference between reinforcement learning and supervised learning is that reinforcement learning does not have pre-prepared training data and corresponding output labels like supervised learning. Reinforcement learning only has reward values, but this reward value is different from the output value of supervised learning. It is not given in advance, but given later. In practical applications, it is found that pure reinforcement learning has low training efficiency and is easy to fall into local optimum in complex obstacle scenes, and problems such as too long detour may occur. Therefore, the A* algorithm is introduced in this method to guide the core logic of the agent to learn to move towards the target and avoid obstacles.

[0229] a3: divide each cycle into rounds, and divide it into a mixed guidance training phase and an adaptive exploration phase.

[0230] In this embodiment, the first 20% of training rounds are set as the guidance period, and the last 80% of training rounds are set as the autonomous exploration period.

[0231] a4: in the mixed guidance training phase, use the A * algorithm to generate an obstacle avoidance path from the starting point A to the ending point B in the grid map, extract the key nodes on the path, and add the distance d node from the current position to the nearest key node in the multi-dimensional state parameter S, and increase the auxiliary reward r guide for moving towards the key node in the reward function.

[0232] Total reward r' = r + r guide = r goal + r obstacle + r optimal + r terminal + r guide ;

[0233] r guide = -k7·d node ;

[0234] In the formula, k7 is the reward far from the key node parameter, k7 is a hyperparameter obtained through training, and in the embodiment, the value is 2.

[0235] By introducing A* algorithm prior guidance, combined with auxiliary reward r guide It can make the agent quickly learn the basic behavior of not colliding and moving towards the target, avoiding a large number of collisions caused by initial random exploration.

[0236] a5: In the adaptive exploration stage, remove A * key node guidance, delete auxiliary reward r guide , keep the core reward function r, and adopt adaptive noise intensity s(t), the agent autonomously explores a more optimal path that is shorter and lower in energy consumption than the A* path, improving path optimality.

[0237] The traditional DDPG adopts a fixed parameter OU noise (such as s = 0.3, 0 = 0.15), the exploration behavior is unrelated to the scene, and the problem of still a large number of explorations turning away when far away from the obstacle may occur, leading to path oscillation. In this method, obstacle perception type adaptive noise s(t) is designed, which dynamically adjusts the noise intensity with the training progress and obstacle distance, so that the exploration is realized when the exploration is needed, and the focus is realized when the focus is needed.

[0238] Exploration action a explore = p(s) + N t ;

[0239] ;

[0240] ;

[0241] In the formula, s0 is the initial noise intensity, which is used to quickly cover the basic obstacle avoidance action and avoid oscillation, s0 e [0.1~0.5]; in this embodiment, it is set to 0.3, which ensures that the initial exploration is sufficient but not out of control. T is the decay period, the value of T is aligned with the total training target and the training rhythm in stages, and in this embodiment, the value is 5x10 4 , so that the exploration intensity smoothly decays with the maturity of the strategy. a is the obstacle distance coefficient, which controls the influence amplitude of the obstacle distance on the noise intensity. The basic logic when setting a is to both strengthen the exploration demand in dangerous scenes and not suppress the target orientation in safe scenes, and in this example, a = 0.5. t is the current training step number, d obs,min is the distance to the nearest obstacle, d safe is the minimum safety distance between the center of the agent and the boundary of the obstacle; 0 ou is the OU noise regression coefficient, which represents the rate of noise regression to the mean, so that the noise remains continuous. Avoid action jumping, and have enough randomness to ensure exploration diversity, and in this embodiment, 0 ou = 0.15. m ouTo make the exploration action of the agent symmetric in the "positive and negative directions" for the expected noise mean, avoid policy freezing due to noise bias, W t is a random distribution of Gaussian white noise to introduce a random exploration component, W t ~N(0,1);N t is an adaptive OU noise. In practical applications, the specific values of parameters σ0, α, θ ou , μ ou may be obtained based on experience or simulation software simulation.

[0242] Based on the obstacle perception type adaptive noise σ(t), the noise attenuates in the later training period, reducing invalid exploration; when approaching the obstacle, d obs,min becomes smaller, and the noise intensity increases, encouraging the agent to explore different obstacle avoidance paths; when moving away from the obstacle, the noise intensity decreases, focusing on moving towards the target.

[0243] The traditional DDPG algorithm introduces a priority-based experience replay (PER) mechanism. Priority-based experience replay can significantly reduce the training time of the DDPG algorithm, improve the stability of the training process, and improve the robustness of the model. However, the traditional PER only assigns experience sampling weights based on the size of the TD error, and the larger the TD error, the higher the sampling probability, but it does not distinguish the value types of experience, such as obstacle avoidance experience and target achievement experience, and cannot learn with focus. When applied in complex environments, it can lead to slow learning speed. In this method, the PRE mechanism is optimized, and additional weights are added for key experiences in complex scenarios, such as collision avoidance and rapid target approach, to enable the agent to focus on learning core scenarios, making the training results more suitable for complex scenarios.

[0244] In step S5, the training method of the path planning and obstacle avoidance network model further includes the following operations:

[0245] PER based on multi-objective TD error is used to improve the sampling weight of high-value experience: TD.

[0246] b1: define error δ: δ = |y i -Q i (s i ,a i )|

[0247] In the formula, y i is the evaluation value for the i-th state and action; δ is used to measure the value uncertainty of the experience;

[0248] Q i (s i ,a i ) is the value network for the i-th state s i and action a iThe target Q value to be output;

[0249] y i =r i +γ·Q i ·(1-done i );

[0250] In the formula, r i The reward obtained by the agent for the i-th action is specifically the result of the reward function calculation; γ is a discount factor that balances immediate and future rewards, and in this embodiment, γ = 0.99; done i The state flag of the agent executing the action has a value of 1 indicating that the destination has been reached or a collision has occurred, and a value of 0 indicating that the action continues. By using y i The design ensures accurate evaluation of the value of each set of states s and actions a, taking into account goal achievement, obstacle avoidance, and optimality.

[0251] b2: Optimize the Priority Experience Replay (PER) and integrate the optimized PER into the training process of the path planning and obstacle avoidance network model;

[0252] Weight Adjustment: Based on experience with "collision avoidance," "rapid target approach," and "significant path optimization," additional weight coefficients are added; specifically, PER employs the following weight adjustment methods:

[0253] Collision avoidance experience weight: If the nearest obstacle distance d in the experience is... obs,min Minimum safe distance d safe This indicates a dangerous distance, and that no collision has occurred. i =0, then δ×2;

[0254] The goal is to quickly approach the experience weight: if d in the experience prev -d curr If the target threshold is approached in a single step, then δ×1.5;

[0255] The specific single-step approach threshold is preset according to the actual environment. In this embodiment, the single-step approach threshold is set to 0.5m.

[0256] Path optimization experience: If the detour distance △d in the experience is... detour If the reduction exceeds the optimization threshold, then δ×1.2; the specific optimization threshold is set according to the actual environment. In this embodiment, the optimization threshold is set to 0.3m, and the current detour distance Δd detour If the detour distance decreases by more than 0.3m compared to the previous moment, then δ×1.2.

[0257] Where, d prev This is the straight-line distance from the agent's previous control cycle to the endpoint B.

[0258] By improving the weight adjustment method of PRE, the agent focuses on learning experience in key scenarios such as obstacle avoidance and goal achievement, thereby improving training efficiency.

[0259] Traditional DDPG only uses a reward function, such as a collision penalty of -200, to softly guide obstacle avoidance, which has two major defects:

[0260] There is a security vulnerability: in the early stage of training and in complex scenarios, the network may output collision risk actions such as not turning at close range, and only relying on reward penalties cannot avoid physical collisions in real time;

[0261] There is also control jitter: the actions output by the actor network may have small high-frequency oscillations, such as frequent fluctuations in the steering angle after obstacle avoidance, which causes the actuators such as rudders and motors to frequently start and stop, increasing mechanical wear and affecting path stability.

[0262] To avoid the above problems, the present application adds a collision prediction module after the output of the action of the policy network, to avoid collisions caused by insufficient learning of the network; based on the collision prediction module, a hard constraint is ensured to realize collision prediction and action correction.

[0263] The collision prediction module specifically includes the following operations.

[0264] c1: the collision prediction module predicts the position (x', y') of the agent at a future time based on the current action (v cmd ,ω cmd );

[0265] ;

[0266] In the formula, θ curr is the heading angle state of the agent at the current time, △t pred is the prediction future time window, △t pred >△t, to ensure that the prediction can be made in advance, and △t pred needs to be selected as a suitable time to give the agent sufficient reaction time, but not too long to cause other problems, in the embodiment, △t pred is set to 0.1s.

[0267] c2: calculate the distance between the predicted position (x', y') and all obstacles, find the closest obstacle, and record the distance between the closest obstacle and (x', y') as d';

[0268] ;

[0269] In the formula, (x obsk ,y obsk ) is the center coordinate of the kth obstacle; r robotr k is the radius of the kth agent obsk r k is the radius of the kth obstacle

[0270] c3: read the preset minimum safety distance d between the center of the agent and the boundary of the obstacle safe , compare d' and d safe ;

[0271] If d' < d safe , trigger the action correction logic to execute step c4;

[0272] Otherwise, execute step c5.

[0273] c4: calculate the corrected steering angle ω cmd,corrected and the corrected speed v cmd,corrected After that, assign ω cmd,corrected to ω cmd , assign v cmd,corrected to v cmd , and execute step c5;

[0274] ;

[0275] In the formula, sign(φ) is the sign of the obstacle azimuth, which is 1 when φ > 0 and -1 when φ < 0. By using sign(), the agent is guided to turn away from the obstacle, ensuring that the turning direction is away from the obstacle. For example, if the obstacle is on the right side and φ = 30°, the right side of the agent's heading is positive in this method, and sign(φ) is 1 when φ > 0, so the agent is guided to turn right to avoid it.

[0276] k ω,corr is the angular velocity correction coefficient, and in this embodiment, k ω,corr = 1.2. When there is a risk of collision, the steering amplitude is moderately enlarged to ensure that the robot quickly deviates from the obstacle direction, while avoiding excessive steering that may cause equipment damage or loss of control of the path. k v,corr is the linear velocity correction coefficient. When there is a risk of collision, the moving speed is "moderately reduced" to reserve reaction time for steering and obstacle avoidance, while avoiding the impact of actuators or the loss of path efficiency caused by a sudden drop in speed. In this embodiment, k v,corr is set to 0.8.

[0277] φ obsk，min is the azimuth angle of the nearest obstacle relative to the heading of the agent at the (x', y') position.

[0278] φ obsk = arctan2(y obsk – y', x obsk – x') - θ';

[0279] where θ' is the heading angle state of the agent at the (x', y') position.

[0280] By correcting the steering angle ω cmd,corrected and the speed v cmd,corrected after correction, the agent increases the steering angle in the direction away from the obstacle, while reducing the speed, quickly escaping from the dangerous area.

[0281] c5: Smooth the speed and the heading angle, output the smoothed speed v cmd,smoothed and the smoothed heading angle ω cmd,smoothed ;

[0282] v cmd,smoothed = β * v cmd + (1-β) * v cmd , prev ;

[0283] ω cmd,smoothed = β * ω cmd + (1-β) * ω cmd , prev ;

[0284] In the formula, β is the filter coefficient, (v cmd , prev , ω cmd , prev ) is the smoothed action of the previous step.

[0285] The action output by the actor may have small oscillations in some cases, such as after obstacle avoidance. This method sets a first-order low-pass filter action smoothing formula in the collision prediction module to reduce action mutations, improves control feasibility through smoothing, effectively avoids frequent start and stop of actuators such as rudders or motors, and reduces mechanical wear. As shown in Figure 3 When the collision prediction module is set in the actor network, the steering angle ω cmd,corrected and the speed v cmd,corrected after correction by the collision prediction module are taken as the output (v, ω) of the actor network and sent to the critic network for subsequent calculation.

[0286] S6: Based on the trained path planning and obstacle avoidance network model, the agent realizes autonomous path planning and obstacle avoidance multi-objective continuous control.

[0287] In this embodiment, for an environment with a maximum distance of 10m from the starting point A to the terminal point B, the success condition is defined as: reaching the terminal point without collision, and the path length ≤ the straight-line distance between AB × 1.5.

[0288] Simulation environment: ROS Noetic (Robot Operating System Noetic Ninjemys), Gazebo 11 (Gazebo 11.11.0). Set the maximum training rounds: 5x10 4 ; Maximum number of steps per round: 800.

[0289] The numerical values corresponding to the core parameters are as follows:

[0290] TAU_MIN = 0.001;

[0291] TAU_MAX = 0.01;

[0292] Set epoch0 to 1e4: EPOCH0 = 10000;

[0293] Discount factor: GAMMA = 0.99;

[0294] Batch size: BATCH_SIZE = 128;

[0295] PER pool capacity: BUFFER_SIZE = 1000000;

[0296] Actor learning rate: LR_ACTOR = 1e-4;

[0297] Critic learning rate: LR_CRITIC = 1e-3;

[0298] Robot radius: ROBOT_RADIUS = 0.15;

[0299] Obstacle radius: OBS_RADIUS = 0.2;

[0300] Minimum safe distance: D_SAFE = 0.35.

[0301] This method generates a random dynamic scene, allowing the agent to interact with a variety of obstacle layouts, improving generalization ability. The generation rules and examples are as follows:

[0302] Randomly generate planar coordinates, satisfying the straight-line distance d AB ∈[5,15]m; covering near and far scenarios. d AB ≥5, to avoid too close and no path optimization space, d AB ≤15, to avoid too long training steps.

[0303] Randomly generate 1~5 static obstacles, shape as circle, r obs=0.2m or rectangle l=0.4m, w=0.4m; the distance between the center of the obstacle and A / B is greater than or equal to 1m to avoid initial collision, and the distance between the obstacles is greater than or equal to 0.5m to avoid overlap.

[0304] In the static obstacles, 1-2 are randomly selected to add a constant speed to become dynamic obstacles, and the speed v obs ∈[-0.2, 0.2]m / s. At the same time, the moving direction of the dynamic obstacle is not directly towards A to avoid danger at the initial moment.

[0305] After generating the scene, whether there is a feasible path is verified by using the A* algorithm, and if the A* algorithm cannot find a path, the scene is regenerated to avoid the agent falling into a "no solution scene".

[0306] The deployment optimization paradigm is as follows:

[0307] Network lightening: remove redundant parameters in the attention mechanism, INT8 quantization, and inference delay ≤5ms;

[0308] Obstacle perception acceleration: only the latest 2 obstacle information is extracted from the laser radar data to reduce the preprocessing time;

[0309] Fault tolerance: if the sensor fails (such as obstacle information loss), switch to "safe mode", move to the target at low speed, and stop when collision occurs.

[0310] In combination with the improvement of the structure of the strategy network Actor and the value network Critic in the method, and the optimization of the training method, the loss function used in the training process of the path planning and obstacle avoidance network model is designed, and the details are as follows.

[0311] The loss function L of the value network Critic is Q :

[0312] ;

[0313] In the formula, N is the batch size, λ is the L2 regularization coefficient to avoid overfitting; θ Q is all trainable network parameters of the value network; Q(s i ,a i ) is the target Q value of the output of the i-th experience of the value network, y i is the evaluation value of the path planning and obstacle avoidance network model for the i-th state and action. In this embodiment, N=128 and λ=10 -5 .

[0314] In the formula, is the MSE error term, which minimizes the deviation between the "current Q value" and the "target Q value" to ensure accurate value evaluation. λ|θ Q |2 2 represents the L2 norm squared, which is the sum of the squares of all parameters. The L2 regularization term suppresses excessively large parameters in the Critic network and avoids overfitting.

[0315] The loss function L of the policy network Actor π for:

[0316] ;

[0317] In the formula, Q min (s i ,π(s i The value network Critic outputs action π(s) to the policy network Actor. i Value assessment of ).

[0318] pass Maximizing Q is transformed into minimizing negative Q, and gradient ascent is transformed into gradient descent. A high Q value indicates a high action value. π If the value decreases, no response or adjustment is needed; however, if the Q value is low, meaning the action value is small, then L... π As the value increases, the action needs to be optimized to adapt to gradient descent optimization. Batch averaging is achieved using 1 / N*Σ to reduce the impact of random fluctuations in individual experience on action optimization. In practical applications, N=128, consistent with the batch size of the Critic network, to ensure synchronized training pace.

[0319] The optimizer update formula uses the Adam optimizer, and the parameter update logic uses gradient descent: θ t+1 =θ t -α·▽ θ_t L(θ t ), where θ t These are the model parameters for the t-th iteration, ▽ θ_t L(θ t Let α be the gradient at the t-th iteration, representing the direction and magnitude of the parameter update, and α be the learning rate.

[0320] Parameter values: Actor network learning rate α Actor =10 -4 ; Critic network learning rate αCritic=10 -3 Faster convergence to guide the Actor; ▽ θ L: The gradient of the loss function with respect to the network parameters, after gradient clipping, clipnorm=1.0.

[0321] Based on such Figure 4 The experimental results of this method are shown below:

[0322] Path planning capability: autonomously plan obstacle avoidance path without preset trajectory from point A to point B, success rate ≥ 95%;

[0323] Obstacle avoidance performance: static obstacle avoidance success rate 100%; facing dynamic obstacles with speed ≤ 0.2 m / s, obstacle avoidance success rate ≥ 90%;

[0324] Path optimality: path length ≤ AB between straight line distance × 1.3, energy consumption reduced by 15% compared with original tracking algorithm DDPG;

[0325] Real-time performance: inference delay ≤ 5 ms, control frequency 100 Hz, meeting the needs of dynamic environment;

[0326] Generalization: facing untrained obstacle layout, number ≤ 8, success rate ≥ 85%.

Claims

1. A DDPG-based autonomous path planning and obstacle avoidance multi-objective continuous control method, characterized in that, It comprises the following steps: S1: based on DDPG algorithm, the path planning and obstacle avoidance problem is mathematically modeled; S2: a strategy network supporting multi-modal fusion is constructed; The structure of the strategy network comprises: a strategy network input layer, a strategy network feature extraction layer, a strategy network attention fusion layer, a strategy network hidden layer and a strategy network output layer connected in turn The input of the strategy network is a multi-dimensional state parameter S; The multi-dimensional state parameter S comprises: a self-state parameter, a target information parameter and an obstacle information parameter; The strategy network feature extraction layer comprises three parallel feature extraction branches: a self-state feature extraction branch, a target direction feature extraction branch and an obstacle risk feature extraction branch; The policy network input layer splits the parameters included in the multi-dimensional state parameter S according to types, and respectively inputs into corresponding feature extraction branches. The outputs of the three feature extraction branches are simultaneously sent into the policy network attention fusion layer for multi-branch feature weighted fusion through an attention mechanism, to obtain fused features. The fused features are sent into the policy network hidden layer for global feature fusion. Finally, after action mapping through the policy network output layer, a 2-dimensional control action v is output: cmd and ω cmd ; S3: a value network of double Q values is constructed; The value network comprises: a value network input layer, a value network feature extraction layer, a value network fusion layer, a value network hidden layer and a value network output layer connected in turn; The value network feature extraction layer comprises four parallel feature extraction branches: a self-state feature extraction branch, a target direction feature extraction branch, an obstacle risk feature extraction branch and an action feature extraction branch; The value network hidden layer comprises two main value calculation branches which are parallel and have the same structure; The input of the value network includes: multi-dimensional state parameter S and 2-dimensional action v output by the policy network cmd and ω cmd ; After the value network input layer splits the parameters included in the multi-dimensional state parameter S into a self-state parameter, a target information parameter and an obstacle information parameter, the parameters are respectively input into the parallel self-state feature extraction branch, target direction feature extraction branch and obstacle risk feature extraction branch, and the 2-dimensional action parameter output by the strategy network is input into the action feature extraction branch; The outputs of the four feature extraction branches are simultaneously sent to the value network fusion layer for splicing of state features and action features; the spliced features are respectively sent to two main value calculation branches in the value network hidden layer, each of the main value calculation branches outputs a Q value to the output layer, and the value network output layer selects the smallest Q value as a target Q value Q target ​ S4: based on the strategy network supporting multi-modal fusion and the value network, a path planning and obstacle avoidance network model is constructed; S5: a training sample data set and a verification sample data set are constructed, the path planning and obstacle avoidance network model is trained, and a trained path planning and obstacle avoidance network model is obtained; S6: based on the trained path planning and obstacle avoidance network model, the agent is realized for autonomous path planning and obstacle avoidance multi-objective continuous control.

2. The DDPG-based autonomous path planning and multi-target continuous control method with obstacle avoidance according to claim 1, characterized in that: In step S1, the following steps are specifically included: S11: environment modeling; The intelligent agent motion scene is defined as a continuous plane space, including a starting point A(x A ,y A ) and an ending point B(x B ,y B ) of a support configuration; the obstacles include static obstacles and dynamic obstacles; the minimum safety distance of the intelligent agent center and the obstacle boundary is defined as d safe ; S12: constructing a state space; Definition of multi-dimensional state parameter S = [x curr ,y curr ,θ curr ,v curr ,ω curr ,△x AB ,△y AB ,d obs1 ,φ obs1 ,d obs2 ,φ obs2 ] wherein (x curr ,y curr ) is the real-time position coordinate of the agent, θ curr is the heading angle of the agent, v curr is the real-time motion speed, ω curr is the real-time angular speed; (△x AB ,△y AB ) is the relative coordinate of the current position and the terminal point B; Δx AB = x B - x curr , Δy AB = y B - y curr ; d obs1 and d obs2 is the straight-line distance to the closest two obstacles; φ obs1 and φ obs2 is the azimuth angle of the closest two obstacles relative to the agent's heading angle, φ ∈ [-π, π]. S13: constructing a state update method: x curr = v cmd · cos θ curr ; y curr = v cmd · sin θ curr ; θ curr = ω cmd ; x next =x curr +v cmd ·cosθ curr ·△t;y next =y curr +v cmd ·sinθ curr ·△t;θ next =θ curr +ω cmd ·△t; where v cmd is the speed in the control action, ω cmd is the angular speed in the control action, Δt is the interval time between the current state and the next state, x next is the x coordinate of the next state position, y next is the y coordinate of the next state position, θ next is the heading angle of the next state.

3. The DDPG-based autonomous path planning and obstacle avoidance multi-objective continuous control method according to claim 2, characterized in that: Each state parameter s included in the multi-dimensional state parameter S is subjected to a state standardization operation before being input into the network: s norm = (s - μ s ) / (σ s + ε) ; In the formula, s norm is the characteristic value of each parameter s after standardization operation, μ s is the state mean value of each parameter s corresponding to the training set, σ s is the state standard deviation of parameter s corresponding to the training set, and ε=10 -6 .

4. The DDPG-based autonomous path planning and multi-target continuous control method with obstacle avoidance according to claim 1, characterized in that: The operation in the attention fusion layer of the strategy network comprises: The features output by the multiple branches in the policy network feature extraction layer are weighted and fused through an attention mechanism to obtain fused features f fso ; f fso =w s ·f s +w g ·f g +w o ·f o ; wherein f s , f g , f o are the feature vectors of the own state, target information, and obstacle information, respectively. w s is a weight for the ego state g is a weight for the target information o is a weight for the obstacle information, satisfying the condition w s +w g +w o =1; ； ； where d safe is the minimum safety distance, d obs,min is the distance to the nearest obstacle, d AB is the straight-line distance between the start point A and the end point B; d curr is the distance from the current position to the end point B.

5. The DDPG-based autonomous path planning and multi-target continuous control method with obstacle avoidance according to claim 1, characterized in that: The strategy network hidden layer of the strategy network is realized based on a full connection layer and a ReLU function; The strategy network output layer is a full connection layer, 64-dimensional features are linearly transformed through the full connection layer, then mapped to [-1, 1] through a tanh function, and then scaled to a physical action range; ； where v min and ω min are preset minimum speed and angular speed to avoid stalling, v max and ω max are preset maximum speed and maximum angular speed; tanh() is the hyperbolic tangent function; o v and o ω are the original values of the policy network output layer.

6. The DDPG-based autonomous path planning and multi-target continuous control method with obstacle avoidance according to claim 1, characterized in that: In the hidden layer of the value network, the main value calculation branch is realized based on a full connection layer and an activation function ReLU; The value network output layer comprises: two parallel single neuron fully connected layers, the output of each main value calculation branch is respectively sent into a single neuron fully connected layer for linear mapping, and the minimum Q value is obtained as Q target ; Q target = min (Q1’(s’,a’),Q2’(s’,a’)) Q1'(s',a') and Q2'(s',a') are two Q values output by the value network hidden layer, s' is a state corresponding to the i th action performed by the agent, and a' is an action vector obtained by the agent based on the strategy network Actor.

7. The DDPG-based autonomous path planning and obstacle avoidance multi-objective continuous control method according to claim 2, characterized in that: In step S5, the training method of the path planning and obstacle avoidance network model comprises the following steps: a1: constructing a multi-objective reward function r; r = r goal +r obstacle +r optimal +r terminal ； In the formula, r goal is a target approach reward, r obstacle is an obstacle avoidance safety reward, r optimal is a path optimization reward, r terminal is a terminal reward; r goal = k1 · (d prev -d curr ) / d max -k2 · d curr ; where d curr is the distance from the current position to the end point B; d curr 2 =△x AB 2 +△y AB 2 ; d prev is the distance to the end point of the previous step; d max is the initial distance from the start point to the end point; k1 and k2 are hyperparameters, which are obtained through training, k1 is a weight parameter for rewarding approaching the target, and k2 is a weight parameter for punishing long distance; ； where d obs,min is the distance to the nearest obstacle, d obs,min = min(d obs1 , d obs2 ); d obs1 and d obs2 are the straight-line distances to the two nearest obstacles. k3 is a safety distance control parameter, k3 is a hyperparameter, which is obtained through training; r optimal = -k4 · v cmd 2 = -k5 · |ω cmd = -k6 · Δd detour ; In the formula, k4 is an energy consumption penalty parameter, k5 is a turning penalty parameter, and k6 is a penalty for detouring; △d detour for the detour distance, △d detour = d curr + d path,accum - d AB ; d path,accum is the cumulative distance of the agent from the start point to the current position, AB is the straight-line distance between the start point A and the end point B. ； In the formula, ε is the minimum value of the distance from the current position to the terminal point B, which is used to mark the arrival at the terminal point; ； wherein v cmd,k is the linear velocity of the k-th step, Δt is the preset interval time of the adjacent two states; t is the current training step number; a2: the input of the policy network is: multi-dimensional state parameter S, and the output is ω cmd and v cmd ; The input s of the policy network and its corresponding output are jointly constructed as a mapping function π(s); When the intelligent agent perceives that the current environment state is s, the output control instruction ω corresponding to the policy network is directly obtained through the mapping function π(s) cmd and v cmd ; a3: dividing each cycle into rounds, which are divided into a hybrid guidance training phase and an adaptive exploration phase; a4: in the phase of the mixed guidance training, use A * The algorithm generates an obstacle avoidance path from the starting point A to the ending point B in the grid map, extracts the key nodes on the path, and adds the distance d from the current position to the nearest key node in the multi-dimensional state parameter S node , increase the auxiliary reward r for moving to the key node in the reward function guide ; Total reward r' = r + r guide = r goal + r obstacle + r optimal + r terminal + r guide ; r guide = -k7·d node ; In the formula, k7 is a reward for moving away from the key node parameter; a5: remove A in adaptive exploration phase * Key node guidance, remove auxiliary reward r guide , keep core reward function r, and adopt adaptive noise intensity σ(t); Exploration action a explore = π(s) + N t ; ； ； where σ0is the initial noise intensity, T is the decay period, a is the obstacle distance coefficient, t is the current training step, d obs,min is the distance to the nearest obstacle, d safe is the minimum safety distance between the agent center and the obstacle boundary; θ ou is the OU noise regression coefficient, μ ou is the noise mean, W t is the random distributed Gaussian white noise W t ~N(0, 1); N t is the adaptive OU noise.

8. The DDPG-based autonomous path planning and obstacle avoidance multi-objective continuous control method according to claim 7, characterized in that: In step S5, the training method of the path planning and obstacle avoidance network model further comprises the following operations: b1 : define error delta: delta = |y i - Q i (s i , a i )|; where y i is the evaluation value for the i-th state and action; Q i (s i ,a i ) is the target Q-value output by the value network for the i-th state s i and action a i . y i = r i + γ · Q i · (1-done i ); where r i the reward obtained by the agent for performing the i-th action; γ is a discount factor that balances immediate and future rewards; done i a flag indicating the state of the agent performing the action, taking the value 1 if the end point is reached or a collision occurs, and the value 0 if it continues to run; b2: optimizing the priority experience replay PER, and integrating the optimized PER into the training process of the path planning and obstacle avoidance network model; The adjustment weight method adopted by the PER comprises: Collision avoidance experience weight: if the most recent obstacle distance d in the experience obs,min < Minimum safety distance d safe and no collision done i = 0, then δ x 2; Target fast approach experience weight: if d prev -d curr > single step approach target threshold, then δ x 1.5; Path optimization experience: if the experience of the distance around △d detour If the distance exceeds the optimization threshold, then δ x 1.2; where d prev is the straight-line distance from the agent's previous control period to the goal B.

9. The DDPG-based autonomous path planning and obstacle avoidance multi-objective continuous control method according to claim 2, characterized in that: After the policy network outputs an action, a collision prediction module is added, and collision prediction and action correction are performed based on the collision prediction module; The collision prediction module specifically comprises the following operations: c1 : the collision prediction module predicts the position (x', y') of the agent at a future time based on the current action (v cmd , ω cmd ). ； In the formula, θ curr is the current heading angle state of the agent, Δt pred is the predicted future time window, Δt pred > Δt; c2: calculate the distance between the predicted position (x', y') and all obstacles, find the closest obstacle, and record the distance between the closest obstacle and (x', y') as d'; ； where (x obsk ,y obsk ) is the center coordinate of the kth obstacle; r robot is the radius of the agent, and r obsk is the radius of the kth obstacle. c3: read the preset minimum safety distance d of the agent center from the obstacle boundary safe , compare d' and d safe ; if d' < d safe then trigger action correction logic, perform step c4; Otherwise, step c5 is performed; c4: calculate the corrected steering angle ω cmd,corrected and the corrected speed v cmd,corrected ω cmd,corrected is assigned to ω cmd v cmd,corrected is assigned to v cmd , step c5 is executed; ； In the formula, sign(φ) is the sign of the position of the obstacle, 1 is taken when φ>0, and -1 is taken when φ<0; k ω,corr is the angular velocity correction coefficient; k v,corr is the linear velocity correction coefficient; φ obsk，min is the azimuth angle of the nearest obstacle relative to the heading of the agent at the (x', y') position; φ obsk = arctan2(y obsk – y’, x obsk – x’) – Θ’; Where θ' is the heading angle state of the agent at (x', y'); c5: smoothing the speed and the heading angle, outputting the smoothed speed v cmd,smoothed and the smoothed heading angle ω cmd,smoothed ; v cmd,smoothed = β * v cmd + (1 - β) * v cmd , prev ; ω cmd,smoothed = β * ω cmd + (1 - β) * ω cmd , prev ; where β is a filter coefficient, (v cmd , prev , ω cmd , prev ) is the smoothed motion of the previous step.

10. The DDPG-based autonomous path planning and obstacle avoidance multi-objective continuous control method according to claim 1, characterized in that: In step S5, during the training process of the path planning and obstacle avoidance network model, the target network updating method is: An adaptive parameter τ is designed to realize the optimization of the target network through adaptive soft updating; The adaptive parameter τ: t = t min + (t max - t min ) · exp (-epoch / epoch0); where epoch is the training epoch, epoch0 is the baseline parameter, τ max and τ min are the maximum and minimum values of τ, respectively, as pre-set.

Images