DDDPG-based autonomous path planning and obstacle avoidance multi-target continuous control method

Abstract

The invention provides a DDPG-based autonomous path planning and obstacle avoidance multi-target continuous control method, which can improve the model stability, improve the feature expression capability and is suitable for more application scenes. According to the method, the strategy network and the value network are not processed as black boxes, the structures of the strategy network and the value network are improved for specific problems occurring in a complex environment, the expression ability of multi-dimensional parameters is effectively improved, and the method is particularly suitable for application scenes of path planning and obstacle avoidance in the complex environment.

Description

Technical Field

[0001] This invention relates to the field of intelligent traffic control technology, specifically to a multi-objective continuous control method based on DDPG autonomous path planning and obstacle avoidance. Background Technology

[0002] Deep Deterministic Policy Gradient (DDPG) is a reinforcement learning algorithm based on an actor-critic architecture. It utilizes two neural networks: an actor network and a critic network. Applied to environments with continuous action spaces, it learns directly from empirical data without requiring prior knowledge of the environment model. The actor network is responsible for outputting the appropriate action based on the current state, while the critic network evaluates the value of taking a specific action in the current state. DDPG continuously updates the weights of these two neural networks, enabling the actor network to output near-optimal actions. Specifically, an action A output by the actor, when input into the critic, yields the maximum Q-value, and the critic network accurately evaluates the value of the action. The training process of the DDPG algorithm is as follows: Initialization: Initialize the weights of the policy network and value network; Exploration: Generate actions using random actions or an exploration policy, and execute the actions while observing the feedback from the environment; Experience Replay: Store the state, action, reward, and next state in the experience replay pool; Training: Randomly sample from the experience replay pool and update the weights of the policy network and value network; Repeat the exploration, experience replay, and training steps until the policy network (Actor) converges and training is complete. The DDPG algorithm exhibits significant advantages in continuous action space control scenarios, and is therefore widely used in path planning for various types of intelligent agents, such as robots, intelligent vehicles, and drones.

[0003] However, when applying the DDPG algorithm to complex scenarios, such as path planning and obstacle avoidance in urban roads by intelligent vehicles, it has been found that in the Actor network of the traditional DDPG algorithm, when multi-dimensional state parameters are directly input into a single fully connected layer, different types of features, such as "self-vehicle speed" and "obstacle distance," easily interfere with each other. For example, the difference in the dimension of speed can suppress obstacle risk signals, leading to insufficient feature representation in some scenarios. The single Critic in traditional DDPG may overestimate the Q value in some cases, leading to erroneous choices of "artificially high value," which in turn affects the accuracy of the final planning result. In the traditional DDPG algorithm, Ornstein-Uhlenbeck (OU) noise with fixed parameters is usually used to enhance exploration. In some scenarios, the exploration behavior may become irrelevant to the scenario, such as continuing to explore and turn even when far away from obstacles, resulting in path oscillation and invalid exploration. These problems all lead to unreliable planning results in complex scenarios, making it impossible to provide travelers with the shortest and lowest-carbon path, which is inconsistent with the current concept of green travel in intelligent transportation. Summary of the Invention

[0004] To address the issue of insufficient stability in planning results of existing path planning and obstacle avoidance methods based on the traditional DDPG algorithm in complex scenarios, this invention provides a multi-objective continuous control method for autonomous path planning and obstacle avoidance based on DDPG. This method can enhance feature representation capabilities, improve model stability, and plan the shortest, lowest-carbon, and most environmentally friendly path while achieving obstacle avoidance in complex scenarios.

[0005] The technical solution of this invention is as follows: a multi-objective continuous control method based on DDPG autonomous path planning and obstacle avoidance, characterized by comprising the following steps: S1: Mathematical modeling of path planning and obstacle avoidance problems based on the DDPG algorithm; S2: Construct a policy network that supports multimodal fusion; The policy network structure includes: a policy network input layer, a policy network feature extraction layer, a policy network attention fusion layer, a policy network hidden layer, and a policy network output layer connected in sequence. The input to the policy network is a multi-dimensional state parameter S; The multi-dimensional state parameter S includes: self-state parameter, target information parameter, and obstacle information parameter; The policy network feature extraction layer includes three parallel feature extraction branches: self-state feature extraction branch, target direction feature extraction branch, and obstacle risk feature extraction branch; The policy network input layer splits the parameters included in the multi-dimensional state parameters S according to their types and inputs them into the corresponding feature extraction branches. The outputs of the three feature extraction branches are simultaneously fed into the policy network attention fusion layer for multi-branch feature weighting and fusion through an attention mechanism to obtain fused features. The fused features are then fed into the policy network hidden layer for global feature fusion. Finally, after action mapping through the policy network output layer, a 2D control action is output: v cmd and ω cmd ; S3: Constructing a value network with dual Q values; The value network comprises: a value network input layer, a value network feature extraction layer, a value network fusion layer, a value network hidden layer, and a value network output layer connected in sequence; The value network feature extraction layer includes four parallel feature extraction branches: self-state feature extraction branch, target direction feature extraction branch, obstacle risk feature extraction branch, and action feature extraction branch. The hidden layer of the value network includes two principal value calculation branches with identical structures, set in parallel. The inputs to the value network include: multi-dimensional state parameters S and the 2-dimensional action v output by the policy network. cmd and ω cmd ; The value network input layer decomposes the parameters included in the multi-dimensional state parameter S into self-state parameters, target information parameters, and obstacle information parameters, and then inputs them into the parallel self-state feature extraction branch, target direction feature extraction branch, and obstacle risk feature extraction branch, respectively. The 2D action parameters output by the policy network are sent into the action feature extraction branch. The outputs of the four feature extraction branches are simultaneously fed into the value network fusion layer for concatenation of state and action features. The concatenated features are then fed into two principal value calculation branches in the hidden layer of the value network. Each principal value calculation branch outputs a Q-value to the output layer. The value network output layer selects the smallest Q-value as the target Q-value. target ; S4: Construct a path planning and obstacle avoidance network model based on the policy network that supports multimodal fusion and the value network; S5: Construct a training sample dataset and a validation sample dataset, train the path planning and obstacle avoidance network model, and obtain the trained path planning and obstacle avoidance network model. S6: Based on the trained path planning and obstacle avoidance network model, the agent achieves autonomous path planning and obstacle avoidance multi-objective continuous control.

[0006] Its further features are: Step S1 specifically includes the following steps: S11: Environment Modeling; The motion scene of the intelligent agent is defined as a continuous planar space, containing a configurable starting point A(x). A ,y A ) and endpoint B(x B ,y B Obstacles include static obstacles and dynamic obstacles; the minimum safe distance between the agent's center and the obstacle boundary is defined as d. safe ; S12: Construct the state space; Define multidimensional state parameters S=[x curr ,y curr ,θ curr ,v curr ,ω curr ,△x AB ,△y AB ,d obs1 ,φ obs1 ,d obs2 ,φ obs2 ]; Among them, (x curr ,y curr θ represents the real-time position coordinates of the agent. curr v is the heading angle of the intelligent agent. curr For real-time motion speed, ω curr For real-time angular velocity; (△x) AB ,△y AB () represents the relative coordinates between the current position and the destination B; △x AB =x B -x curr , △y AB =y B -y curr ; d obs1 and d obs2 φ is the straight-line distance between the two nearest obstacles. obs1 and φ obs2 Let φ be the azimuth angle of the two nearest obstacles relative to the agent's heading angle, where φ∈[-π,π]. S13: The state update method is constructed as follows: x curr =v cmd ·cosθ curr ;y curr =v cmd ·sinθ curr ;θ curr =ω cmd ; xnext =x curr +v cmd ·cosθ curr ·△t;y next =y curr +v cmd ·sinθ curr ·△t;θ next =θ curr +ω cmd ·△t; In the formula, v cmd To control the speed during the action, ω cmd To control the angular velocity during the action, Δt is the time interval between the current state and the next state, x next The x-coordinate and y-coordinate of the next state position next Let θ be the y-coordinate of the next state position. next The heading angle for the next state; Each state parameter s included in the multi-dimensional state parameter S undergoes state normalization before being input into the network: s norm =(s-μ s ) / (σ s +ε); In the formula, s norm The eigenvalues ​​μ after standardization for each parameter s s Let σ be the mean of the training set states corresponding to each parameter s. s Let s be the standard deviation of the training set states corresponding to parameter s, and ε = 10. -6 ; The operations in the attention fusion layer of the policy network include: The features output from multiple branches in the feature extraction layer of the policy network are weighted and fused using an attention mechanism to obtain the fused feature f. fso ; f fso =w s ·f s +w g ·f g +w o ·f o ; Among them, f s f g f o These are feature vectors representing the user's own state, target information, and obstacle information, respectively. w s w is the weight of its own state. g For the target information weight, w o The obstacle information weights satisfy the condition w s +w g+w o =1; ; ; The hidden layer of the policy network is implemented based on a fully connected layer in conjunction with the ReLU function; The output layer of the policy network is a fully connected layer. The 64-dimensional features are linearly transformed through the fully connected layer, mapped to [-1,1] by tanh, and then scaled to the physical action range. ; In the formula, v min and ω min For the preset minimum speed and angular velocity to avoid stalling, v max and ω max The maximum speed and maximum angular velocity are preset; tanh() is the hyperbolic tangent function; o v and o ω The original values ​​are the output layer values ​​of the policy network; In the hidden layer of the value network, the principal value calculation branch is implemented based on a fully connected layer in conjunction with the ReLU activation function; The value network output layer comprises two parallel single-neuron fully connected layers. The output of each principal value calculation branch is fed into a single-neuron fully connected layer for linear mapping, directly outputting a 1D Q-value, and then the minimum Q-value is obtained as Q. target ; Q target =min (Q1'(s',a'),Q2'(s',a')); Q1'(s',a') and Q2'(s',a') are two Q values ​​output by the hidden layer of the value network, respectively. s' is the state corresponding to the i-th action executed by the agent, and a' is the action vector obtained by the state s' based on the policy network Actor. In step S5, the training method for the path planning and obstacle avoidance network model includes: a1: Construct a multi-objective reward function r; r = r goal +r obstacle +r optimal +r terminal ; In the formula, r goal To achieve the target reward, r obstacle As a safety reward for obstacle avoidance, r optimal For the optimal path reward, r terminal For terminal rewards; r goal =k1·(d prev -dcurr ) / d max -k2·d curr ; In the formula, d curr d represents the distance from the current position to the destination B. curr 2 =△x AB 2 +△y AB 2 ; d prev d is the distance from the previous step to the destination. max This is the initial distance from the starting point to the ending point; k1 and k2 are hyperparameters obtained through training. k1 is the weight parameter that rewards those who are close to the target, and k2 is the weight parameter that penalizes those who are far away. ; In the formula, d obs,min d is the distance to the nearest obstacle. obs,min =min(d obs1 ,d obs2 );d obs1 and d obs2 The straight-line distance between the two nearest obstacles; k3 is the safety distance control parameter, and k3 is a hyperparameter obtained through training; r optimal =-k4·v cmd 2 -k5·|ω cmd |-k6·△d detour ; In the formula, k4 is the energy consumption penalty parameter, k5 is the steering penalty parameter, and k6 is the penalty detour parameter; △d detour Let Δd be the detour distance. detour =d curr +d path,accum -d AB ; d path,accum Let d be the cumulative distance the agent has moved from its starting point to its current position. AB The straight-line distance between the starting point A and the ending point B; ; In the formula, ε is the minimum distance from the current position to the destination B, which is used to mark that the destination has been reached; ; Among them, v cmd,k Let be the linear velocity at step k, Δt be the preset interval between two adjacent states, and t be the current training step number. a2: The input to the policy network is the multi-dimensional state parameter S, and the output is ω. cmd and v cmd ; The input s of the policy network and its corresponding output are used to construct a mapping function π(s); When the agent perceives the current environmental state as s, it directly obtains the output control command ω corresponding to the policy network through the mapping function π(s). cmd and v cmd ; a3: Divide each cycle into two phases: a hybrid guided training phase and an adaptive exploration phase. a4: In the phase of the hybrid guided training, use A * The algorithm generates an obstacle avoidance path from starting point A to ending point B in a grid map, extracts key nodes on the path, and adds the distance d from the current position to the nearest key node to the multi-dimensional state parameter S. node Add an auxiliary reward r to the reward function for moving to the key node. guide ; Total reward r' = r + r guide = r goal +r obstacle +r optimal +r terminal +r guide ; r guide = -k7·d node ; In the formula, k7 is the parameter for rewarding distance from critical nodes; a5: During the adaptive exploration phase, remove A. * Key node guidance, removal of auxiliary rewards r guide The core reward function r is retained, and an adaptive noise intensity σ(t) is adopted; Explore action a explore = π(s)+N t ; ; ; In the formula, σ0 is the initial noise intensity, T is the decay period, α is the obstacle distance coefficient, t is the current training step number, and d obs,min d is the distance to the nearest obstacle. safe θ represents the minimum safe distance between the center of the agent and the boundary of the obstacle. ou μ is the regression coefficient for OU noise. ou W is the noise mean. t W is randomly distributed Gaussian white noise t ~N(0,1); N tFor adaptive OU noise; Step S5, the training method for the path planning and obstacle avoidance network model, further includes the following operations: b1: Define the error δ: δ = |y i -Q i (s i ,a i )|; In the formula, y i The evaluation value for the i-th state and action; Q i (s i ,a i ) represents the value network for the i-th state s i and action a i The target Q value to be output; y i =r i +γ·Q i ·(1-done i ); In the formula, r i The reward for the i-th action performed by the agent; γ is a discount factor that balances immediate and future rewards; done i The state flag of the agent executing the action has a value of 1, which indicates that the destination has been reached or a collision has occurred, and a value of 0 indicates that the action continues. b2: Optimize the Priority Experience Replay (PER) and integrate the optimized PER into the training process of the path planning and obstacle avoidance network model; The weighting methods used by PER include: Collision avoidance experience weight: If the nearest obstacle distance d in the experience is... obs,min Minimum safe distance d safe And no collision done i =0, then δ×2; The goal is to quickly approach the experience weight: if d in the experience prev -d curr If the target threshold is approached in a single step, then δ×1.5; Path optimization experience: If the detour distance △d in the experience is... detour If the reduction exceeds the optimization threshold, then δ×1.2; Where, d prev The straight-line distance from the agent's previous control cycle to the endpoint B; After the policy network outputs an action, a collision prediction module is added. Based on the collision prediction module, collision prediction and action correction are performed. The collision prediction module specifically includes the following operations: c1: The collision prediction module is based on the current action (v) cmd ,ωcmd ), predict the position (x', y') of the agent at future time; ; In the formula, θ curr Let Δt be the heading angle state of the agent at the current moment. pred To predict future time windows, △t pred >△t; c2: Calculate the distance between the predicted position (x',y') and all obstacles, find the nearest obstacle, and denote the distance between the nearest obstacle and (x',y') as d'; ; In the formula, (x obsk ,y obsk ) represents the center coordinates of the k-th obstacle; r robot Let r be the radius of the agent. obsk Let the radius be the radius of the k-th obstacle; c3: Read the preset minimum safe distance d between the agent's center and the obstacle boundary. safe Compare d' and d safe ; If d' <d safe If so, the action correction logic is triggered, and step c4 is executed; Otherwise, proceed to step c5; c4: Calculate the corrected steering angle ω cmd,corrected and the corrected velocity v cmd,corrected Then, ω cmd,corrected Assigned to ω cmd , will v cmd,corrected Assigned to v cmd Execute step c5; ; In the formula, sign(φ) is the obstacle orientation symbol, which is 1 when φ>0 and -1 when φ<0; k ω,corr k is the angular velocity correction factor. v,corr φ is the linear velocity correction factor. obsk，min Let be the azimuth angle of the nearest obstacle relative to the agent's heading at position (x', y'); φ obsk =arctan2(y obsk –y',x obsk –x')-θ'; Where θ' is the heading angle state of the agent at position (x',y'); c5: Smooths the velocity and heading angle, and outputs the smoothed velocity v. cmd,smoothed and smoothed heading angle ω cmd,smoothed ; vcmd,smoothed =β*v cmd +(1-β)*v cmd , prev ; ω cmd,smoothed =β*ω cmd +(1-β)*ω cmd , prev ; In the formula, β is the filter coefficient, (v cmd , prev , ω cmd , prev () represents the smoothed action following the previous step; In step S5, during the training of the path planning and obstacle avoidance network model, the target network update method is as follows: We design adaptive parameters τ and optimize the target network through adaptive soft updates. The adaptive parameter τ: τ=τ min +(τ max -τ min )·exp (-epoch / epoch0); In the formula, epoch is the training epoch, epoch0 is the baseline parameter, and τ max and τ min These are the preset maximum and minimum values ​​of τ.

[0007] This application provides a multi-objective continuous control method based on DDPG autonomous path planning and obstacle avoidance. It expands the state dimension by using multi-dimensional state parameters S as network input parameters, encompassing the network's own state, target information, and obstacle perception information. This ensures the network comprehensively acquires decision-making information and improves the accuracy of the model's final output. The application improves the policy network and value network in the DDPG algorithm. For the multi-dimensional input state parameter S, multiple feature extraction branches are simultaneously set in both the policy and value networks. Different types of input state parameters are input into corresponding feature extraction branches, effectively avoiding mutual interference between different types of feature parameters and improving network model stability. An attention fusion network structure is set in the policy network's attention fusion layer, which, combined with multi-branch feature extraction in the feature extraction layer, effectively enhances feature representation capabilities, allowing the network to focus on targets and dangerous obstacles. Two structurally identical but parameter-independent principal value calculation branches are designed in the hidden layer of the value network to evaluate the Q-value of the same set of "states and actions." The minimum value is taken as the target Q-value output through the output layer, effectively avoiding bias caused by overestimation by a single network and further improving network stability. This method divides the model training process into a hybrid guidance phase and an adaptive exploration phase. In the hybrid guidance phase, the A* algorithm is introduced. By setting key nodes on the path, an auxiliary reward of "moving towards key nodes" is added to the reward function, allowing the agent to quickly learn the basic behaviors of "moving towards the target without collision," avoiding a large number of collisions caused by initial random exploration and improving training efficiency. In the adaptive exploration phase, an adaptive noise intensity σ(t) is introduced, allowing the noise intensity to be dynamically adjusted according to the current training step t and the obstacle distance d, ensuring noise attenuation in the later stages of training. This effectively reduces invalid exploration and enables the model to effectively focus on the target. Compared with other existing planning and obstacle avoidance methods based on the DDPG algorithm, this method does not treat the policy network and value network as black boxes. Instead, it improves the structure of the policy network and value network for specific problems occurring in complex environments, effectively improving the expressive power of multi-dimensional parameters. This makes this method particularly suitable for path planning and obstacle avoidance applications in complex environments, enabling the planning of the shortest, lowest-carbon, and environmentally friendly path while achieving obstacle avoidance in complex scenarios. Attached Figure Description

[0008] Figure 1 A schematic diagram of the network structure of policy networks and value networks in existing technologies; Figure 2 This is a schematic diagram of the network structure of the strategy network in this application; Figure 3 This is a schematic diagram of the network structure of the value network in this application; Figure 4 This is a schematic diagram of the training process for the path planning and obstacle avoidance network model in this application. Detailed Implementation

[0009] like Figure 1 The diagram shows the network structure of the policy network (Actor) and value network (Critic) in the standard DDPG algorithm. The Actor and Critic networks are trained separately. The input to the Critic network is the state parameters and the action output from the Actor network. The action output from the Actor network is the input to the Critic network, and the output of the Critic network is used for backpropagation in the Actor network. However, the standard DDPG algorithm suffers from unreliable planning results when applied to complex environments, limiting its applicability in practical scenarios such as navigation in unknown environments and control of autonomous robots. This method improves the network structure of the policy network (Actor) and value network (Critic) to address the characteristics of complex environments.

[0010] This application includes a multi-objective continuous control method based on DDPG autonomous path planning and obstacle avoidance, which includes the following steps.

[0011] S1: Mathematical modeling of path planning and obstacle avoidance problems based on the DDPG algorithm.

[0012] Step S1 specifically includes the following steps: S11: Environment Modeling; The motion scene of the intelligent agent is defined as a continuous planar space, containing a configurable starting point A(x). A ,y A ) and endpoint B(x B ,y B Obstacles include static obstacles and dynamic obstacles; the minimum safe distance between the agent's center and the obstacle boundary is defined as d. safe Among them, dynamic obstacles carry a constant velocity (v). obsx , v obsy The state synchronously includes its speed information. Among them, (x) A ,y A ) and (x B ,y B These are all physical coordinates in planar space, directly reflecting the distance and direction between the agent and the target location.

[0013] In this embodiment, the agent is set to its own radius r. robot For a robot with a height of 0.15m, what is the minimum safe distance d between the robot's center and the obstacle boundary? safe ≥0.35m.

[0014] During training, static obstacles with a radius of r are randomly generated. obsUse 1 to 5 circles with a length l = 0.2m and a width w = 0.4m, and randomly change them during training to improve generalization.

[0015] S12: Construct the state space; Define multidimensional state parameters S=[x curr ,y curr ,θ curr ,v curr ,ω curr ,△x AB ,△y AB ,d obs1 ,φ obs1 ,d obs2 ,φ obs2 ].

[0016] This method expands the state dimension to 11 dimensions, covering its own state, target information, and obstacle perception information, ensuring that the network comprehensively acquires decision-making information. Among these, (x curr ,y curr θ represents the real-time position coordinates of the agent. curr v is the agent's heading angle, representing the agent's direction of travel; curr For real-time motion speed, ω curr Angular velocity is the real-time velocity, representing the agent's turning speed.

[0017] (△x AB ,△y AB ) represents the relative coordinates between the current position and the destination B, used to guide the agent to move towards the target; △x AB =x B -x curr , △x AB Let x be the relative x-coordinate between the agent and the endpoint B; △y AB =y B -y curr , △y AB Let y be the relative coordinates of the agent and the endpoint B in the y-direction. d obs1 and d obs2 The straight-line distance between the two nearest obstacles is used to cover the main obstacle avoidance risks; φ obs1 and φ obs2 φ represents the azimuth angles of the two nearest obstacles relative to the agent's heading, used to determine the location and direction of the obstacles, where φ∈[-π,π].

[0018] In practical implementation, the parameter s in the multi-dimensional state parameter S is collected based on multimodal sensors installed on the intelligent agent, such as optical cameras, infrared sensors, radar, Beidou navigation modules, gyroscopes, accelerometers, vehicle speed sensors, compass sensors, etc., installed on intelligent vehicles. The specific collection method is based on existing technologies.

[0019] This embodiment only designs the collection of information for the two most recent obstacles. In practical applications, more obstacles need to be supported, and the obstacle perception information can be collected according to actual needs. obs and φ obs To expand.

[0020] S13: The state update method is constructed as follows: The agent starts from the current state (x) curr , y curr, θ curr ) with control action (v cmd ,ω cmd The update method is as follows: x curr =v cmd ·cosθ curr ;y curr =v cmd ·sinθ curr ;θ curr =ω cmd ; x next =x curr +v cmd ·cosθ curr ·△t;y next =y curr +v cmd ·sinθ curr ·△t;θ next =θ curr +ω cmd ·△t; In the formula, v cmd To control the speed during the action, ω cmd To control the angular velocity during the action, Δt is the time interval between the current state and the next state, x next The x-coordinate and y-coordinate of the next state position next Let θ be the y-coordinate of the next state position. next The heading angle for the next state.

[0021] The specific methods for calculating the distance and azimuth angle between obstacles and the agent are as follows: ; Where, x obsk ,y obskLet r be the center coordinates of the k-th obstacle, k=1,2; robot r is the radius of the agent itself. obsk d is the radius of the k-th obstacle; obsk φ is the distance between the robot and the k-th obstacle, in meters. obsk The azimuth angle of the k-th obstacle relative to the robot's heading is expressed in rad, ranging from -π to π. In this embodiment, the positive direction is to the right of the robot's forward direction.

[0022] To improve the network training convergence speed and avoid feature suppression caused by differences in dimensions, each state parameter s in the multi-dimensional state parameter S of this application undergoes state standardization before being input into the network: s norm =(s-μ s ) / (σ s +ε); In the formula, s norm The eigenvalues ​​μ after standardization for each parameter s s Let σ be the mean of the training set states corresponding to each parameter s. s Let s be the standard deviation of the training set states corresponding to parameter s, and ε = 10. -6 Avoid having a denominator of 0. μ s and σ s All of these are obtained by statistical analysis of the parameters s included in the training set.

[0023] In this embodiment, the x-coordinate range is [0,10] from A[0,0] to B[10,10] in the scene. The mean is taken as the midpoint of the range, and the standard deviation is taken as 1 / 3 of the range, which covers more than 95% of the sample distribution. curr corresponding μ s =5.0m, σ s =3.0m; y curr corresponding μ s =5.0m, σ s =3.0m. The heading angle is randomly distributed and unbiased, so the mean is 0 and the standard deviation is 1.0 rad, covering most turning scenarios. For example, ±1 rad corresponds to ±57°, so θ curr corresponding μ s =0 rad, σ s =1.0 rad. The linear velocity is uniformly distributed within the constraint range, with a mean of 0.3 m / s at the midpoint of the range, covering 95% of the samples, and a standard deviation of 1 / 3.3 of the range. Therefore, the robot's current linear velocity v is... curr corresponding μ s =0.3m / s, σ s=0.12m / s. The angular velocity is unbiased, with a mean of 0 and a standard deviation of 0.5rad / s. This covers normal steering and avoids frequent large steering maneuvers, therefore ω... curr corresponding μ s =0 rad / s, σ s =0.5 rad / s. And the nearest obstacle's d obs1 corresponding μ s =2.0m, σ s =1.5m, used to adapt to "number of obstacles 1-5, safety distance d safe In scenarios with a range of ≥0.35m, the azimuth angle is randomly distributed with no fixed bias. A standard deviation of 1.2 rad covers the positions of most obstacles within 360°. For example, ±1.2 rad corresponds to ±68.8°. Therefore, the azimuth angle of nearby obstacles... obs corresponding μ s =0 rad, σ s =1.2 rad. In this example, the other obstacle is the second nearest obstacle, and the distance is usually greater than that of the nearest obstacle, d obs2 corresponding μ s =2.5m, σ s =1.8m.

[0024] An action space A is constructed to describe the continuous control outputs of the policy network. The action space consists of the continuous control variables that the robot can execute, directly mapped to actuator instructions, expressed as: A = [v...]. cmd ,ω cmd Based on the characteristics of each agent, constraints are set for each agent, v cmd ∈[v min ,v max ],ω cmd ∈[ω min ,ω max ]. Where, v min To minimize speed and avoid stagnation, v max To limit energy consumption for maximum speed, or to ensure safety; ω min For the minimum heading angle, ω max To achieve the maximum heading angle, the turning range is controlled to adapt to the operating environment of the intelligent agent and enable obstacle avoidance and detour. In this embodiment, the constraint for the robot moving indoors is: v cmd ∈[0.1,0.5]m / s, ω cmd ∈[-1.5,1.5] rad / s. The action is output through the Actor network and scaled to ensure compliance with physical constraints.

[0025] The relationship between action and path: ω cmd The change in heading angle indirectly adjusts the path direction. cmdThey determine the movement speed and work together to move towards the target and avoid obstacles.

[0026] S2: Construct a policy network that supports multimodal fusion; like Figure 2 As shown, the structure of the policy network includes: a policy network input layer, a policy network feature extraction layer, a policy network attention fusion layer, a policy network hidden layer, and a policy network output layer connected in sequence. The input to the policy network is a multi-dimensional state parameter S; the multi-dimensional state parameter S includes: self-state parameter, target information parameter and obstacle information parameter; the feature extraction layer of the policy network includes three parallel feature extraction branches: self-state feature extraction branch, target direction feature extraction branch and obstacle risk feature extraction branch.

[0027] The policy network input layer splits the parameters included in the multi-dimensional state parameters S according to their types, resulting in three independent input streams: the 5-dimensional self-state parameters are sent to the self-state feature extraction branch, the 2-dimensional target information parameters are sent to the target direction feature extraction branch, and the 4-dimensional obstacle information parameters are sent to the obstacle risk feature extraction branch.

[0028] Three feature extraction branches extract features, outputting 64-dimensional self-state features, 32-dimensional target direction features, and 64-dimensional obstacle risk features. The outputs of these three branches are simultaneously fed into the policy network's attention fusion layer for weighted fusion of multi-branch features via an attention mechanism, resulting in a 128-dimensional fused feature. This fused feature is then fed into the policy network's hidden layer for global feature fusion, outputting a 64-dimensional refined feature. Finally, the policy network's output layer maps the refined feature to an action, outputting a 2-dimensional control action: v. cmd and ω cmd .

[0029] The operations in the attention fusion layer of the policy network include: The features output from multiple branches in the feature extraction layer of the policy network are weighted and fused using an attention mechanism to obtain the fused feature f. fso ; f fso =w s ·f s +w g ·f g +w o ·f o ; Among them, f s f g f o These are feature vectors representing the user's own state, target information, and obstacle information, with dimensions of 64, 32, and 64, respectively.

[0030] ws w is the weight of its own state. s Use base weights to ensure smooth operation.

[0031] w g The target information weight is the distance d from the current position to the destination B. curr The larger w g The larger the size, the more priority the target.

[0032] w o The obstacle information weights are the distances d from the nearest obstacle. obs,min The smaller, w o The larger the size, the more priority is given to obstacle avoidance.

[0033] Satisfy condition w s +w g +w o =1; ; .

[0034] The hidden layer of the policy network is implemented based on a fully connected layer combined with the nonlinear activation function ReLU; the fully connected layer is implemented using the classic structure of a deep neural network with a weight matrix plus a bias term.

[0035] The output layer of the policy network is a fully connected layer. The 64-dimensional features undergo a linear transformation through this layer, are then mapped to [-1, 1] using the tanh activation function, and finally scaled to the physical action range. The fully connected layer has two neurons, corresponding to two control actions. ; In the formula, v min and ω min For the preset minimum speed and angular velocity to avoid stalling, v max and ω max The maximum speed and maximum angular velocity are preset; tanh() is the hyperbolic tangent function; o v and o ω The original values ​​are the output layer values ​​of the policy network; In this method, a multi-branch feature extraction and attention fusion network structure is designed in the policy network to improve the feature representation ability and allow the network to focus on the target and dangerous obstacles.

[0036] S3: Constructing a value network with dual Q values; like Figure 3 As shown, the value network includes: a value network input layer, a value network feature extraction layer, a value network fusion layer, a value network hidden layer, and a value network output layer connected in sequence.

[0037] The value network feature extraction layer includes four parallel feature extraction branches: self-state feature extraction branch, target direction feature extraction branch, obstacle risk feature extraction branch, and action feature extraction branch.

[0038] The hidden layer of the value network consists of two principal value computation branches with identical structures, set up in parallel.

[0039] The input to the value network includes: 11-dimensional multi-dimensional state parameters S and 2-dimensional actions v output by the policy network. cmd and ω cmd .

[0040] The value network input layer decomposes the parameters included in the multi-dimensional state parameters S into self-state parameters, target information parameters, and obstacle information parameters, and then inputs them into the parallel self-state feature extraction branch, target direction feature extraction branch, and obstacle risk feature extraction branch, respectively. The 2D action parameters output by the policy network are sent into the action feature extraction branch.

[0041] Four feature extraction branches extract features, outputting 64-dimensional self-state features, 32-dimensional target direction features, 64-dimensional obstacle risk features, and 32-dimensional action features. The outputs of these four feature extraction branches are simultaneously fed into the value network fusion layer for concatenation of the state and action features, resulting in a 192-dimensional concatenated feature set. The concatenated feature set is then fed into two principal value calculation branches in the hidden layer of the value network.

[0042] In the hidden layer of the value network, the principal value calculation branch is implemented based on the fully connected layer and the ReLU activation function; each principal value calculation branch outputs a 64-dimensional Q value and outputs two Q values ​​to the output layer.

[0043] The value network output layer consists of two parallel single-neuron fully connected layers, employing a linear mapping design without activation functions. Since Q-values ​​are continuous, the original value scale must be preserved. The output of each principal value calculation branch is fed into a single-neuron fully connected layer for linear mapping. Each group of 64 refined features undergoes linear transformation through a single-neuron fully connected layer, directly outputting a 1-dimensional Q-value. The minimum Q-value is then taken as the Q-value. target Output; Q target =min(Q1'(s',a'),Q2'(s',a')); Q1'(s',a') and Q2'(s',a') are two Q-values ​​output by the hidden layer of the value network, respectively. s' is the state corresponding to the i-th action performed by the agent, and a' is the action vector obtained by the policy network Actor based on the state s'. The core function of the Critic network is to evaluate the value of the Actor's output action and output the corresponding Q value. In this method, two parallel principal value calculation branches with the same structure but independent parameters are set in the hidden layer of the value network. They evaluate the Q value corresponding to the same set of states s and actions a respectively, and select the smaller Q value as the output. This can avoid the bias caused by the overestimation of the value by a single network, and thus avoid the problem of actions being biased towards "artificially high value".

[0044] Standard DDPG uses a soft update method with a fixed hyperparameter τ to control the parameter update magnitude. However, in complex scenarios, this may lead to slow updates to the target network. This method optimizes the target network based on adaptive soft updates. The adaptive parameter τ is designed as follows: τ=τ min +(τ max -τ min )·exp (-epoch / epoch0); In the formula, epoch is the training round. In the training process, one epoch corresponds to the robot starting from the starting point A and attempting to complete a complete task cycle of "autonomous path planning, obstacle avoidance, and reaching the destination B". At the same time, a timeout period is set to limit the number of rounds. That is, regardless of success or failure, if the timeout exceeds 15 seconds or a collision occurs, the epoch will terminate.

[0045] epoch0 is the baseline parameter used to control the rate at which the adaptive parameter τ decays with epoch. In this embodiment, epoch0 = 10. 4 One training round.

[0046] τ max and τ min The maximum and minimum values ​​of τ are preset. The specific values ​​are pre-set based on the type of agent and its operating environment, using empirical values. In this embodiment, τ is for an indoor walking robot. max =0.01, τ min =0.001.

[0047] When epoch=0, τ=0.01, fast follow-up; when epoch=10 4 At that time, τ = 0.001 + 0.009·exp(-1) ≈ 0.0043, and updates slowly.

[0048] Based on the adaptive parameter τ in this method, τ is larger in the early stage of training, so that the target network can quickly follow the local network and accelerate convergence; τ is reduced in the later stage of training to keep the target network stable and improve the final performance.

[0049] S4: Construct a path planning and obstacle avoidance network model based on a policy network and a value network that support multimodal fusion.

[0050] This method addresses complex scenarios by improving the network structures of the policy network (Actor) and the value network (Critic). This effectively enhances the mapping accuracy from multi-dimensional state parameters S to actions. However, a targeted training method must be designed for the improved network structure to ensure that issues such as gradient explosion and slow convergence do not occur during training. The specific training method is shown below.

[0051] S5: Construct training sample datasets and validation sample datasets, train the path planning and obstacle avoidance network model, and obtain the trained path planning and obstacle avoidance network model.

[0052] This application reconstructs the reward function as: target proximity reward + obstacle avoidance safety reward + optimal path reward + terminal reward. By balancing multiple objectives with weights, it guides the agent to learn the optimal path under the premise of obstacle avoidance, and realizes the planning of the shortest, lowest carbon and environmentally friendly path on the basis of obstacle avoidance.

[0053] a1: Construct a multi-objective reward function r; r = r goal +r obstacle +r optimal +r terminal ; In the formula, r goal To achieve the target reward, r obstacle As a safety reward for obstacle avoidance, r optimal For the optimal path reward, r terminal For terminal rewards.

[0054] Target proximity reward r goal With a core orientation, it encourages movement toward the finish line and punishes those who stray from it.

[0055] r goal =k1·(d prev -d curr ) / d max -k2·d curr ; In the formula, d curr d represents the distance from the current position to the destination B. curr 2 =△x AB 2 +△y AB 2 ; d prev d is the distance from the previous step to the destination. max The initial distance from the starting point to the finish line is used to normalize the reward. k1 and k2 are hyperparameters obtained through training. k1 is the weight parameter for rewarding proximity to the target, and k2 is the weight parameter for penalizing distance. In this embodiment, k1=10 and k2=2, ensuring the core logic of continuously approaching the target.

[0056] Obstacle avoidance safety reward r obstacle This is used to ensure safety by punishing close contact with obstacles and rewarding safe distances. Specifically, dangerous distances are severely punished, while safe distances are lightly rewarded, thus preventing agents from risking getting too close to obstacles.

[0057] ; In the formula, d obs,min d is the distance to the nearest obstacle. obs,min =min(d obs1 ,d obs2 );d obs1 and d obs2 d is the straight-line distance between the two nearest obstacles; obs,min ≤d safe / 2 is defined as an impending collision; d safe / 2<d obs,min ≤d safe Defined as the danger distance, d obs,min >d safe Defined as a safe distance.

[0058] k3 is a safe distance control parameter. A positive reward is given when the safe distance is reached to encourage people to stay away from obstacles. k3 is a hyperparameter, which is obtained through training. In this embodiment, k3=3.

[0059] Path-optimal reward r optimal Used to improve path quality, penalizing detours, energy consumption, and frequent turns.

[0060] r optimal =-k4·v cmd 2 -k5·|ω cmd |-k6·△d detour ; In the formula, k4 is the energy consumption penalty parameter; k5 is the steering penalty parameter to avoid frequent changes of direction; k6 is the penalty detour parameter to guide the path closer to a straight line; k4, k5, and k6 are hyperparameters obtained through training. In this embodiment, k4=0.1, k5=0.5, and k6=0.8.

[0061] △d detour Let Δd be the detour distance. detour =d curr +d path,accum -d AB ; d ABd is the straight-line distance between the starting point A and the ending point B. curr This represents the straight-line distance from the current agent to the destination B.

[0062] d path,accum This is the cumulative distance the agent has traveled from its starting point to its current position, which is also the actual path length traveled. ; Among them, v cmd,k Let be the linear velocity at step k, Δt be the preset interval between two adjacent states, and t be the current training step number. In this embodiment, Δt is set to 0.01s.

[0063] Terminal reward r terminal Used to enhance goal achievement; ; In the formula, ε is the minimum distance from the current position to the endpoint B, used to mark reaching the endpoint; in this embodiment, it is set to 0.1m. When d curr A timeout of <0.1m indicates that the endpoint has been reached. The specific timeout period is set according to the actual scenario; in this embodiment, it is set to 15s.

[0064] a2: The input to the policy network is the multi-dimensional state parameter S, and the output is ω. cmd and v cmd ; The input s of the policy network and its corresponding output are used to construct a mapping function π(s); When the agent perceives the current environmental state as s, it directly obtains the output control command ω corresponding to the policy network through the mapping function π(s). cmd and v cmd .

[0065] DDPG is a model-free policy optimization algorithm based on reinforcement learning, used to solve reinforcement learning problems in continuous action spaces. The biggest difference between reinforcement learning and supervised learning is that reinforcement learning does not have pre-prepared training data and corresponding output labels like supervised learning. Reinforcement learning only provides reward values, but these reward values ​​differ from the output values ​​in supervised learning; they are not given in advance but rather deferred. In practical applications, it has been found that pure reinforcement learning is inefficient in training scenarios with complex obstacles, is prone to getting trapped in local optima, and can lead to problems such as overshooting obstacles. Therefore, this method introduces the A* algorithm for prior guidance, accelerating the agent's learning of the core logic for moving towards the target and avoiding obstacles.

[0066] a3: Divide each cycle into two phases: a hybrid guided training phase and an adaptive exploration phase.

[0067] In this embodiment, the first 20% of training rounds are designated as the guidance period, and the remaining 80% of training rounds are designated as the autonomous exploration period.

[0068] a4: In the phase of hybrid guided training, use A * The algorithm generates an obstacle avoidance path from starting point A to ending point B in a grid map, extracts key nodes on the path, and adds the distance d from the current position to the nearest key node to the multi-dimensional state parameter S. node Add an auxiliary reward r to the reward function for moving to the key node. guide ; Total reward r' = r + r guide = r goal +r obstacle +r optimal +r terminal +r guide ; r guide = -k7·d node ; In the formula, k7 is the parameter for rewarding distance from critical nodes, and k7 is a hyperparameter obtained through training, which is set to 2 in this embodiment.

[0069] By introducing the A* algorithm prior guidance, combined with auxiliary reward r guide It allows the agent to quickly learn the basic behavior of moving towards the target without collision, avoiding a large number of collisions caused by initial random exploration.

[0070] a5: During the adaptive exploration phase, remove A. * Key node guidance, removal of auxiliary rewards r guide The core reward function r is retained, and an adaptive noise intensity σ(t) is adopted. The agent autonomously explores a better path that is shorter and less energy-consuming than the A* path, thereby improving the path optimality.

[0071] Traditional DDPG uses fixed-parameter OU noise (e.g., σ=0.3, θ=0.15), and the exploration behavior is independent of the scene. This leads to a large number of explorations and turns even when the object is far away from it, causing path oscillations. In this method, we design obstacle-aware adaptive noise σ(t), which dynamically adjusts the noise intensity according to the training progress and the distance to the obstacle, so as to explore when it is appropriate and focus when it is appropriate.

[0072] Explore action a explore =π(s)+N t ; ; ; In the formula, σ0 is the initial noise intensity, used to quickly cover the basic obstacle avoidance actions and avoid oscillations; σ0∈[0.1~0.5]; in this embodiment, it is set to 0.3 to ensure sufficient initial exploration without losing control. T is the decay period, and the value of T is aligned with the overall training objective and the phased training rhythm; in this embodiment, it is set to 5×10. 4 This allows the exploration intensity to decrease smoothly with strategy maturity. α is the obstacle distance coefficient, controlling the impact of obstacle distance on noise intensity. The basic logic for setting α is to both strengthen the exploration needs in dangerous scenarios and not suppress the goal orientation in safe scenarios. In this example, α = 0.5. t is the current training step number, d... obs,min d is the distance to the nearest obstacle. safe θ represents the minimum safe distance between the center of the agent and the boundary of the obstacle. ou θ represents the OU noise regression coefficient, indicating the rate at which the noise regresses to the mean, ensuring both continuity and sufficient randomness to guarantee exploration diversity. In this embodiment, θ... ou =0.15. μ ou To achieve the desired noise mean, the agent's exploration actions are symmetrical in both the positive and negative directions, avoiding policy rigidity due to noise bias. t Randomly distributed Gaussian white noise is used to introduce a random exploration component, W t ~N(0,1); N t This is for adaptive OU noise. In practical applications, parameters σ0, α, and θ... ou μ ou The specific value can be obtained based on empirical values ​​or by determining it after simulation using simulation software.

[0073] Based on obstacle-aware adaptive noise σ(t), noise attenuation is implemented in the later stages of training to reduce invalid exploration; when approaching an obstacle, d obs,min As the noise level decreases, the agent is encouraged to explore different obstacle avoidance paths; as the agent moves away from the obstacle, the noise level decreases, and the agent focuses on moving toward the target.

[0074] Traditional DDPG algorithms introduce a priority-based experience replay (PER) mechanism. Priority-based PER can significantly reduce the training time of DDPG, improve the stability of the training process, and enhance the robustness of the model. However, traditional PER only assigns experience sampling weights based on the magnitude of the TD error; the larger the TD error, the higher the sampling probability. It does not distinguish the value type of experience, such as obstacle avoidance experience or goal achievement experience, and cannot achieve focused learning. This leads to slower learning speeds when applied to complex environments. This method optimizes the PRE mechanism by adding extra weights to key experiences in complex scenarios, such as collision avoidance and rapid target approach, allowing the agent to focus on learning core scenarios and making the training results more applicable to complex environments.

[0075] Step S5, the training method for the path planning and obstacle avoidance network model, also includes the following operations: We employ PER based on multi-objective TD error to enhance the sampling weight of high-value experiences: TD.

[0076] b1: Define the error δ: δ = |y i -Q i (s i ,a i )|; In the formula, y i The evaluation value is for the i-th state and action; δ is used to measure the uncertainty of the value of experience. Q i (s i ,a i For the value network, s represents the i-th state. i and action a i The target Q value to be output; y i =r i +γ·Q i ·(1-done i ); In the formula, r i The reward obtained by the agent for the i-th action is specifically the result of the reward function calculation; γ is a discount factor that balances immediate and future rewards, and in this embodiment, γ = 0.99; done i The state flag of the agent executing the action has a value of 1 indicating that the destination has been reached or a collision has occurred, and a value of 0 indicating that the action continues. By using y i The design ensures accurate evaluation of the value of each set of states s and actions a, taking into account goal achievement, obstacle avoidance, and optimality.

[0077] b2: Optimize the Priority Experience Replay (PER) and integrate the optimized PER into the training process of the path planning and obstacle avoidance network model; Weight Adjustment: Based on experience with "collision avoidance," "rapid target approach," and "significant path optimization," additional weight coefficients are added; specifically, PER employs the following weight adjustment methods: Collision avoidance experience weight: If the nearest obstacle distance d in the experience is... obs,min Minimum safe distance d safe This indicates a dangerous distance, and that no collision has occurred. i =0, then δ×2; The goal is to quickly approach the experience weight: if d in the experience prev -d curr If the target threshold is approached in a single step, then δ×1.5; The specific single-step approach threshold is preset according to the actual environment. In this embodiment, the single-step approach threshold is set to 0.5m.

[0078] Path optimization experience: If the detour distance △d in the experience is... detour If the reduction exceeds the optimization threshold, then δ×1.2; the specific optimization threshold is set according to the actual environment. In this embodiment, the optimization threshold is set to 0.3m, and the current detour distance Δd detour If the detour distance decreases by more than 0.3m compared to the previous moment, then δ×1.2.

[0079] Where, d prev This is the straight-line distance from the agent's previous control cycle to the endpoint B.

[0080] By improving the weight adjustment method of PRE, the agent can focus on learning from key scenarios such as obstacle avoidance and goal achievement, thereby improving training efficiency.

[0081] Traditional DDPGs rely solely on reward functions, such as a collision penalty of -200, to provide soft guidance for obstacle avoidance. This approach has two major drawbacks: Security vulnerabilities exist: In the early stages of training and in complex scenarios, the network may output collision-risk actions such as failing to turn at close range, and relying solely on rewards and penalties cannot avoid physical collisions in real time; There is also the issue of control jitter: the actions output by the Actor network may have small, high-frequency oscillations, such as frequent fluctuations in the steering angle after obstacle avoidance, which causes frequent start-stop of actuators such as servos and motors, increasing mechanical losses and affecting path stability.

[0082] To avoid the above problems, this application adds a collision prediction module after the policy network outputs the action to avoid collisions caused by insufficient network learning; hard constraints are implemented based on the collision prediction module to achieve collision prediction and action correction.

[0083] The collision prediction module includes the following operations.

[0084] c1: The collision prediction module is based on the current action (v cmd ,ω cmd ), predict the position (x', y') of the agent at future time; ; In the formula, θ curr Let Δt be the heading angle state of the agent at the current moment. pred To predict future time windows, △t pred >△t, ensuring that predictions can be made in advance, and △t pred It is necessary to choose a suitable time to allow the agent sufficient reaction time, but not so long that it would cause other problems. In this embodiment, Δt is...pred Set to 0.1s.

[0085] c2: Calculate the distance between the predicted position (x',y') and all obstacles, find the nearest obstacle, and denote the distance between the nearest obstacle and (x',y') as d'; ; In the formula, (x obsk ,y obsk ) represents the center coordinates of the k-th obstacle; r robot Let r be the radius of the k-th agent. obsk Let be the radius of the k-th obstacle.

[0086] c3: Read the preset minimum safe distance d between the agent's center and the obstacle boundary. safe Compare d' and d safe ; If d' <d safe If so, the action correction logic is triggered, and step c4 is executed; Otherwise, proceed to step c5.

[0087] c4: Calculate the corrected steering angle ω cmd,corrected and the corrected velocity v cmd,corrected Then, ω cmd,corrected Assigned to ω cmd , will v cmd,corrected Assigned to v cmd Execute step c5; ; In the formula, sign(φ) is the obstacle orientation symbol. It is 1 when φ>0 and -1 when φ<0. The sign() function guides the agent to turn away from the obstacle, ensuring that the turning direction is away from the obstacle. If the obstacle is on the right side with φ=30°, the right side of the agent's heading is taken as positive in this method. When φ>0, sign(φ) is 1, which guides the agent to turn right to avoid the obstacle.

[0088] k ω,corr For angular velocity correction coefficients, in this embodiment, k ω,corr =1.2, appropriately increase the steering amplitude when there is a risk of collision, ensure that the robot quickly deviates from the direction of the obstacle, and avoid equipment damage or loss of path control due to over-steering. v,corr The linear velocity correction coefficient is used to "appropriately reduce the moving speed" in the event of a collision risk, allowing reaction time for steering and obstacle avoidance, while avoiding actuator impact or path efficiency loss caused by a sudden drop in speed. In this embodiment, k v,corr Set it to 0.8.

[0089] φ obsk，minLet be the azimuth angle of the nearest obstacle relative to the agent's heading at position (x', y').

[0090] φ obsk =arctan2(y obsk –y',x obsk –x')-θ'; Where θ' is the heading angle state of the agent at position (x',y').

[0091] After correction, the steering angle ω cmd,corrected and the corrected velocity v cmd,corrected This causes the agent to increase its steering angle in the direction away from the obstacle, while simultaneously reducing its speed, allowing it to quickly escape the danger zone.

[0092] c5: Smooths the velocity and heading angle, and outputs the smoothed velocity v. cmd,smoothed and smoothed heading angle ω cmd,smoothed ; v cmd,smoothed =β*v cmd +(1-β)*v cmd , prev ; ω cmd,smoothed =β*ω cmd +(1-β)*ω cmd , prev ; In the formula, β is the filter coefficient, (v cmd , prev ,ω cmd , prev ) represents the smoothed action following the previous step.

[0093] The actions output by the Actor may exhibit slight oscillations in some situations, such as after obstacle avoidance. This method uses a first-order low-pass filter in the collision prediction module to reduce abrupt action changes. This smoothing process improves control feasibility and effectively avoids frequent starts and stops of actuators such as servos or motors, reducing mechanical losses. Figure 3 As shown, when a collision prediction module is set in the Actor network, the steering angle ω after correction by the collision prediction module will be... cmd,corrected and the corrected velocity v cmd,corrected The output (v, ω) of the Actor network is fed into the Critic network for subsequent calculations.

[0094] S6: Based on a trained path planning and obstacle avoidance network model, the agent can achieve autonomous path planning and obstacle avoidance multi-objective continuous control.

[0095] In this embodiment, for an environment where the maximum distance from the starting point A to the ending point B is 10m, the success condition is defined as: reaching the ending point without collision, and the path length ≤ the straight-line distance between A and B × 1.5.

[0096] Simulation environment: ROSNoetic (Robot Operating System Noetic Ninjemys), Gazebo11 (Gazebo11.11.0). Maximum training epochs set: 5×10 4 Maximum number of steps per round: 800.

[0097] The core parameters are set as follows: TAU_MIN = 0.001; TAU_MAX = 0.01; Set epoch0 to 1e4: EPOCH0 = 10000; Discount factor: GAMMA = 0.99; Batch size: BATCH_SIZE = 128; PER pool capacity: BUFFER_SIZE = 1,000,000; Actor learning rate: LR_ACTOR = 1e-4; Critic learning rate: LR_CRITIC = 1e-3; Robot radius: ROBOT_RADIUS = 0.15; Obstacle radius: OBS_RADIUS = 0.2; Minimum safe distance: D_SAFE = 0.35.

[0098] This method generates random dynamic scenes, allowing the agent to encounter diverse obstacle layouts, thereby improving generalization ability. The generation rules and examples are as follows: Randomly generate planar coordinates that satisfy the straight-line distance d between A and B. AB ∈[5,15]m; covers both near and far scenes. d AB ≥5, avoids path optimization space due to excessive proximity, d AB ≤15, to avoid excessively long training steps.

[0099] Randomly generate 1-5 static obstacles, all in circular shape, r obs =0.2m or rectangle l=0.4m, w=0.4m; the distance between the center of the obstacle and A / B is ≥1m to avoid initial collision, and the distance between obstacles is ≥0.5m to avoid overlap.

[0100] Randomly select 1-2 static obstacles and add a constant velocity v to them, turning them into dynamic obstacles. obs ∈[-0.2,0.2]m / s. Simultaneously, the dynamic obstacle's direction of motion is not directly towards A, avoiding danger at the initial moment.

[0101] After generating the scenario, the A* algorithm is used to verify whether a feasible path exists. If A* cannot find a path, the scenario is regenerated to prevent the agent from getting stuck in an "unsolvable scenario".

[0102] The deployment optimization paradigm is as follows: Network lightweighting: Remove redundant parameters in the attention mechanism, use INT8 quantization, and achieve inference latency ≤5ms; Accelerated obstacle perception: Only the two most recent obstacles are extracted from LiDAR data, reducing preprocessing time; Fault tolerance: If the sensor malfunctions (such as loss of obstacle information), switch to "safe mode" and move towards the target at low speed, stopping upon collision.

[0103] Based on the improvements to the policy network (Actor) and value network (Critic) structures in this method, as well as the optimization of the training method, the loss function used in the training process of the path planning and obstacle avoidance network model is designed as follows.

[0104] The loss function L of the value network Critic Q for: ; In the formula, N is the batch size, λ is the L2 regularization coefficient to avoid overfitting; θ Q For all trainable network parameters of the value network; Q(s) i ,a i Let y be the target Q value of the output of the i-th experience in the value network. i This represents the evaluation value of the path planning and obstacle avoidance network model for the i-th state and action. In this embodiment, N=128, λ=10. -5 .

[0105] In the formula, The MSE error term minimizes the deviation between the "current Q value" and the "target Q value" to ensure accurate value assessment. λ|θ Q | 2 2 represents the L2 norm squared, which is the sum of the squares of all parameters. The L2 regularization term suppresses excessively large parameters in the Critic network and avoids overfitting.

[0106] The loss function L of the policy network Actor π for: ; In the formula, Qmin (s i ,π(s i The value network Critic outputs action π(s) to the policy network Actor. i Value assessment of ).

[0107] pass Maximizing Q is transformed into minimizing negative Q, and gradient ascent is transformed into gradient descent. A high Q value indicates a high action value. π If the value decreases, no response or adjustment is needed; however, if the Q value is low, meaning the action value is small, then L... π As the value increases, the action needs to be optimized to adapt to gradient descent optimization. Batch averaging is achieved using 1 / N*Σ to reduce the impact of random fluctuations in individual experience on action optimization. In practical applications, N=128, consistent with the batch size of the Critic network, to ensure synchronized training pace.

[0108] The optimizer update formula uses the Adam optimizer, and the parameter update logic uses gradient descent: θ t+1 =θ t -α·▽ θ_t L(θ t ), where θ t These are the model parameters for the t-th iteration, ▽ θ_t L(θ t ) represents the gradient at the t-th iteration, indicating the direction and magnitude of the parameter update, and α is the learning rate.

[0109] Parameter values: Actor network learning rate α Actor =10 -4 ; Critic network learning rate αCritic=10 -3 Faster convergence to guide the Actor; ▽ θ L: The gradient of the loss function with respect to the network parameters, after gradient clipping, clipnorm=1.0.

[0110] Based on such Figure 4 The experimental results of this method are shown below: Path planning capability: When there is no preset trajectory from point A to point B, the system can autonomously plan an obstacle avoidance path with a success rate of ≥95%. Obstacle avoidance performance: 100% success rate for static obstacles; ≥90% success rate for dynamic obstacles with a speed ≤0.2m / s. Path optimality: Path length ≤ 1.3 times the straight-line distance between A and B, energy consumption is reduced by 15% compared to the original tracking algorithm DDPG; Real-time performance: Inference latency ≤ 5ms, control frequency 100Hz, meeting the requirements of dynamic environments; Generalization: When faced with an untrained obstacle layout of ≤8, the success rate is ≥85%.

Claims

1. A multi-objective continuous control method based on DDPG autonomous path planning and obstacle avoidance, characterized in that, It includes the following steps: S1: Mathematical modeling of path planning and obstacle avoidance problems based on the DDPG algorithm; S2: Construct a policy network that supports multimodal fusion; The policy network structure includes: a policy network input layer, a policy network feature extraction layer, a policy network attention fusion layer, a policy network hidden layer, and a policy network output layer connected in sequence. The input to the policy network is a multi-dimensional state parameter S; The multi-dimensional state parameter S includes: self-state parameter, target information parameter, and obstacle information parameter; The policy network feature extraction layer includes three parallel feature extraction branches: self-state feature extraction branch, target direction feature extraction branch, and obstacle risk feature extraction branch; The policy network input layer splits the parameters included in the multi-dimensional state parameters S according to their types and inputs them into the corresponding feature extraction branches. The outputs of the three feature extraction branches are simultaneously fed into the policy network attention fusion layer for multi-branch feature weighting and fusion through an attention mechanism to obtain fused features. The fused features are then fed into the policy network hidden layer for global feature fusion. Finally, after action mapping through the policy network output layer, a 2D control action is output: v cmd and ω cmd ; S3: Constructing a value network with dual Q values; The value network comprises: a value network input layer, a value network feature extraction layer, a value network fusion layer, a value network hidden layer, and a value network output layer connected in sequence; The value network feature extraction layer includes four parallel feature extraction branches: self-state feature extraction branch, target direction feature extraction branch, obstacle risk feature extraction branch, and action feature extraction branch. The hidden layer of the value network includes two principal value calculation branches with identical structures, set in parallel. The inputs to the value network include: multi-dimensional state parameters S and the 2-dimensional action v output by the policy network. cmd and ω cmd ; The value network input layer decomposes the parameters included in the multi-dimensional state parameter S into self-state parameters, target information parameters, and obstacle information parameters, and then inputs them into the parallel self-state feature extraction branch, target direction feature extraction branch, and obstacle risk feature extraction branch, respectively. The 2D action parameters output by the policy network are sent into the action feature extraction branch. The outputs of the four feature extraction branches are simultaneously fed into the value network fusion layer for concatenating state and action features. The concatenated features are then fed into two principal value calculation branches in the hidden layer of the value network. Each principal value calculation branch outputs a Q-value to the output layer, and the value network output layer selects the smallest Q-value as the target Q-value. target ; S4: Construct a path planning and obstacle avoidance network model based on the policy network that supports multimodal fusion and the value network; S5: Construct a training sample dataset and a validation sample dataset, train the path planning and obstacle avoidance network model, and obtain the trained path planning and obstacle avoidance network model. S6: Based on the trained path planning and obstacle avoidance network model, the agent achieves autonomous path planning and obstacle avoidance multi-objective continuous control.

2. The multi-objective continuous control method based on DDPG autonomous path planning and obstacle avoidance according to claim 1, characterized in that: Step S1 specifically includes the following steps: S11: Environment Modeling; The motion scene of the intelligent agent is defined as a continuous planar space, containing a configurable starting point A(x). A ,y A ) and endpoint B(x B ,y B Obstacles include static obstacles and dynamic obstacles; the minimum safe distance between the agent's center and the obstacle boundary is defined as d. safe ; S12: Construct the state space; Define multidimensional state parameters S=[x curr ,y curr ,θ curr ,v curr ,ω curr ,△x AB ,△y AB ,d obs1 ,φ obs1 ,d obs2 ,φ obs2 ]; Among them, (x curr ,y curr θ represents the real-time position coordinates of the agent. curr v is the heading angle of the intelligent agent. curr For real-time motion speed, ω curr For real-time angular velocity; (△x) AB ,△y AB () represents the relative coordinates between the current position and the destination B; △x AB =x B -x curr ,△y AB =and B -and curr ; d obs1 and d obs2 φ is the straight-line distance between the two nearest obstacles. obs1 and φ obs2 Let φ be the azimuth angle of the two nearest obstacles relative to the agent's heading angle, where φ∈[-π,π]. S13: The state update method is constructed as follows: x curr =v cmd ·cosθ curr ;y curr =v cmd ·sinθ curr ;θ curr =ω cmd ; x next =x curr +v cmd ·cosθ curr ·△t;y next =y curr +v cmd ·sinθ curr ·△t;θ next =θ curr +oh cmd ·△t; In the formula, v cmd To control the speed during the action, ω cmd To control the angular velocity during the action, Δt is the time interval between the current state and the next state, x next The x-coordinate and y-coordinate of the next state position next Let θ be the y-coordinate of the next state position. next The heading angle for the next state.

3. The multi-objective continuous control method based on DDPG autonomous path planning and obstacle avoidance according to claim 2, characterized in that: Each state parameter s included in the multi-dimensional state parameter S undergoes state normalization before being input into the network: s norm =(s-μ s ) / (s s +e); In the formula, s norm The eigenvalues ​​μ after standardization for each parameter s s Let σ be the mean of the training set states corresponding to each parameter s. s Let s be the standard deviation of the training set states corresponding to parameter s, and ε = 10. -6 .

4. The multi-objective continuous control method based on DDPG autonomous path planning and obstacle avoidance according to claim 1, characterized in that: The operations in the attention fusion layer of the policy network include: The features output from multiple branches in the feature extraction layer of the policy network are weighted and fused using an attention mechanism to obtain the fused feature f. fso ; f fso =w s ·f s +w g ·f g +w o ·f o ; Among them, f s f g f o These are feature vectors representing the user's own state, target information, and obstacle information, respectively. w s w is the weight of its own state. g For the target information weight, w o The obstacle information weights satisfy the condition w s +w g +w o =1; ； ； Where, d safe For the minimum safe distance, d obs,min d is the distance to the nearest obstacle. AB d is the straight-line distance between the starting point A and the ending point B. curr This represents the distance from the current position to the destination B.

5. The multi-objective continuous control method based on DDPG autonomous path planning and obstacle avoidance according to claim 1, characterized in that: The hidden layer of the policy network is implemented based on a fully connected layer in conjunction with the ReLU function; The output layer of the policy network is a fully connected layer. The 64-dimensional features are linearly transformed through the fully connected layer, mapped to [-1,1] by tanh, and then scaled to the physical action range. ； In the formula, v min and ω min For the preset minimum speed and angular velocity to avoid stalling, v max and ω max The maximum speed and maximum angular velocity are preset; tanh() is the hyperbolic tangent function; o v and o ω These are the original values ​​for the output layer of the policy network.

6. The multi-objective continuous control method based on DDPG autonomous path planning and obstacle avoidance according to claim 1, characterized in that: In the hidden layer of the value network, the principal value calculation branch is implemented based on a fully connected layer in conjunction with the ReLU activation function; The value network output layer comprises two parallel single-neuron fully connected layers. The output of each principal value calculation branch is fed into a single-neuron fully connected layer for linear mapping, directly outputting a 1D Q-value, and then the minimum Q-value is obtained as Q. target ; Q target =min (Q1’(s’,a’),Q2’(s’,a’)); Q1'(s',a') and Q2'(s',a') are two Q-values ​​output by the hidden layer of the value network, respectively. s' is the state corresponding to the i-th action performed by the agent, and a' is the action vector obtained by the policy network Actor based on the state s'.

7. The multi-objective continuous control method based on DDPG autonomous path planning and obstacle avoidance according to claim 2, characterized in that: In step S5, the training method for the path planning and obstacle avoidance network model includes: a1: Construct a multi-objective reward function r; r = r goal +r obstacle +r optimal +r terminal ； In the formula, r goal To achieve the target reward, r obstacle As a safety reward for obstacle avoidance, r optimal For the optimal path reward, r terminal For terminal rewards; r goal =k1·(d prev -d curr ) / d max -k2·d curr ; In the formula, d curr d represents the distance from the current position to the destination B. curr 2 =△x AB 2 +△y AB 2 ; d prev d is the distance from the previous step to the destination. max This is the initial distance from the starting point to the ending point; k1 and k2 are hyperparameters obtained through training. k1 is the weight parameter that rewards those who are close to the target, and k2 is the weight parameter that penalizes those who are far away. ； In the formula, d obs,min d is the distance to the nearest obstacle. obs,min =min(d obs1 ,d obs2 );d obs1 and d obs2 The straight-line distance between the two nearest obstacles; k3 is the safety distance control parameter, and k3 is a hyperparameter obtained through training; r optimal =-k4·v cmd 2 -k5·|ω cmd |-k6·△d detour ; In the formula, k4 is the energy consumption penalty parameter, k5 is the steering penalty parameter, and k6 is the penalty detour parameter; △d detour Let Δd be the detour distance. detour =d curr +d path,accum -d AB ; d path,accum Let d be the cumulative distance the agent has moved from its starting point to its current position. AB The straight-line distance between the starting point A and the ending point B; ； In the formula, ε is the minimum distance from the current position to the destination B, which is used to mark that the destination has been reached; ； Among them, v cmd,k Let be the linear velocity at step k, Δt be the preset interval between two adjacent states, and t be the current training step number. a2: The input to the policy network is the multi-dimensional state parameter S, and the output is ω. cmd and v cmd ; The input s of the policy network and its corresponding output are used to construct a mapping function π(s); When the agent perceives the current environmental state as s, it directly obtains the output control command ω corresponding to the policy network through the mapping function π(s). cmd and v cmd ; a3: Divide each cycle into two phases: a hybrid guided training phase and an adaptive exploration phase. a4: In the phase of the hybrid guided training, use A * The algorithm generates an obstacle avoidance path from starting point A to ending point B in a grid map, extracts key nodes on the path, and adds the distance d from the current position to the nearest key node to the multi-dimensional state parameter S. node Add an auxiliary reward r to the reward function for moving to the key node. guide ; Total reward r' = r + r guide = r goal +r obstacle +r optimal +r terminal +r guide ; r guide = -k7 d node ; In the formula, k7 is the parameter for rewarding distance from critical nodes; a5: During the adaptive exploration phase, remove A. * Key node guidance, removal of auxiliary rewards r guide The core reward function r is retained, and an adaptive noise intensity σ(t) is adopted; Explore action a explore = π(s)+N t ; ； ； In the formula, σ0 is the initial noise intensity, T is the decay period, α is the obstacle distance coefficient, t is the current training step number, and d obs,min d is the distance to the nearest obstacle. safe θ represents the minimum safe distance between the center of the agent and the boundary of the obstacle. ou μ is the regression coefficient for OU noise. ou W is the noise mean. t W is randomly distributed Gaussian white noise t ~N(0,1); N t For adaptive OU noise.

8. The multi-objective continuous control method based on DDPG autonomous path planning and obstacle avoidance according to claim 7, characterized in that: Step S5, the training method for the path planning and obstacle avoidance network model, further includes the following operations: b1: Define the error δ: δ = |y i -Q i (s i ,a i )|; In the formula, y i The evaluation value for the i-th state and action; Q i (s i ,a i ) represents the value network for the i-th state s i and action a i The target Q value to be output; y i =r i +γ·Q i ·(1-done i ); In the formula, r i The reward for the i-th action performed by the agent; γ is a discount factor that balances immediate and future rewards; done i The state flag of the agent executing the action has a value of 1, which indicates that the destination has been reached or a collision has occurred, and a value of 0 indicates that the action continues. b2: Optimize the Priority Experience Replay (PER) and integrate the optimized PER into the training process of the path planning and obstacle avoidance network model; The weighting methods used by PER include: Collision avoidance experience weight: If the nearest obstacle distance d in the experience is... obs,min Minimum safe distance d safe And no collision done i =0, then δ×2; The goal is to quickly approach the experience weight: if d in the experience prev -d curr If the target threshold is approached in a single step, then δ×1.5; Path optimization experience: If the detour distance △d in the experience is... detour If the reduction exceeds the optimization threshold, then δ×1.2; Where, d prev This is the straight-line distance from the agent's previous control cycle to the endpoint B.

9. The multi-objective continuous control method based on DDPG autonomous path planning and obstacle avoidance according to claim 2, characterized in that: After the policy network outputs an action, a collision prediction module is added. Based on the collision prediction module, collision prediction and action correction are performed. The collision prediction module specifically includes the following operations: c1: The collision prediction module is based on the current action (v) cmd ,ω cmd ), predict the position (x', y') of the agent at future time; ； In the formula, θ curr Let Δt be the heading angle state of the agent at the current moment. pred To predict future time windows, △t pred >△t; c2: Calculate the distance between the predicted position (x',y') and all obstacles, find the nearest obstacle, and denote the distance between the nearest obstacle and (x',y') as d'; ； In the formula, (x obsk ,y obsk ) represents the center coordinates of the k-th obstacle; r robot Let r be the radius of the agent. obsk Let the radius be the radius of the k-th obstacle; c3: Read the preset minimum safe distance d between the agent's center and the obstacle boundary. safe Compare d' and d safe ; If d' <d safe If so, the action correction logic is triggered, and step c4 is executed; Otherwise, proceed to step c5; c4: Calculate the corrected steering angle ω cmd,corrected and the corrected velocity v cmd,corrected Then, ω cmd,corrected Assigned to ω cmd , will v cmd,corrected Assigned to v cmd Execute step c5; ； In the formula, sign(φ) is the obstacle orientation symbol, which is 1 when φ>0 and -1 when φ<0; k ω,corr k is the angular velocity correction factor. v,corr φ is the linear velocity correction factor. obsk，min Let be the azimuth angle of the nearest obstacle relative to the agent's heading at position (x', y'); φ obsk =arctan2(y obsk –y',x obsk –x')-θ': Where θ' is the heading angle state of the agent at position (x',y'); c5: Smooths the velocity and heading angle, and outputs the smoothed velocity v. cmd,smoothed and smoothed heading angle ω cmd,smoothed ; v cmd,smoothed =β*v cmd +(1-β)*v cmd , prev ; oh cmd,smoothed =b*ω cmd +(1-β)*ω cmd , prev ; In the formula, β is the filter coefficient, (v cmd , prev , ω cmd , prev ) represents the smoothed action following the previous step.

10. The multi-objective continuous control method based on DDPG autonomous path planning and obstacle avoidance according to claim 1, characterized in that: In step S5, during the training of the path planning and obstacle avoidance network model, the target network update method is as follows: Design adaptive parameters τ and achieve target network optimization through adaptive soft updates; The adaptive parameter τ: τ=τ min +(t max -t min )·exp (-epoch / epoch0); In the formula, epoch is the training epoch, epoch0 is the baseline parameter, and τ max and τ min These are the preset maximum and minimum values ​​of τ.

Images