A dynamic path planning method for mobile robots based on improved DDQN algorithm

Abstract

The application discloses a kind of dynamic path planning method of mobile robot based on improved DDQN algorithm, comprising: build dynamic simulation environment;Path planning model is constructed, including neural network module, experience replay module, intelligent agent control module;Initialize dynamic simulation environment and path planning model;The path planning model is trained based on the fusion strategy of improved DDQN and BOAE mechanism;Load the path planning model of mobile robot trained, and the optimal action decision of intelligent agent is output according to strategy network and drives robot to run in dynamic environment, and environment simulation module real-time feedback robot position and state, and visual module shows the optimal path of robot.The application can be more efficient and accurate in dynamic environment by the innovative way of combining BOAE and improved DDQN algorithm, and the mobile robot can plan the path, significantly improve the obstacle avoidance ability and the success rate of reaching the target, and has extremely broad application prospect and popularization value.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of mobile robots, and relates to dynamic path planning technology, in particular to a mobile robot dynamic path planning method based on an improved DDQN algorithm. BACKGROUND

[0002] In the development process of mobile robots, path planning has always been the core link for the realization of its autonomous navigation function. Traditional path planning algorithms, typified by A* algorithm, exhibit certain planning efficiency in static environments, and can plan relatively reasonable paths according to preset environmental maps and target information. However, with the increasing expansion and complexity of robot application scenarios, especially in scenarios with dynamic obstacles and real-time changes in environmental information, traditional algorithms have serious limitations.

[0003] Traditional algorithms lack adaptability to environmental dynamic changes and autonomous learning ability, and cannot adjust path planning strategies in a timely manner according to real-time changes in the environment. This defect causes the robot to often have difficulty making effective obstacle avoidance decisions when facing dynamic obstacles, increasing the risk of collision and reducing the efficiency and accuracy of path planning.

[0004] The emergence of DDQN (Deep Double Q Network) algorithm in the field of deep reinforcement learning brings a glimmer of hope for solving dynamic environment path planning problems. DDQN algorithm approximates the Q value function through a deep neural network, enabling the robot to gradually learn the optimal action strategy in the interaction process with the environment. However, under the test of complex dynamic scenarios, DDQN algorithm still has many shortcomings. Its sample efficiency is low, and a large number of interaction samples are needed to achieve a relatively ideal training effect, which undoubtedly increases the training time and resource consumption. In addition, it has poor adaptability to dynamic obstacles and is difficult to accurately respond to complex and variable motion patterns of obstacles, resulting in a significant reduction in the accuracy and stability of path planning in actual applications. SUMMARY

[0005] The present application belongs to the field of mobile robots, and relates to dynamic path planning technology, in particular to a mobile robot dynamic path planning method based on an improved DDQN algorithm.

[0006] Technical scheme: In order to achieve the above-mentioned purpose, the present application provides a mobile robot dynamic path planning method based on an improved DDQN algorithm, comprising the following steps:

[0007] S1: build a dynamic simulation environment;

[0008] S2: build a path planning model, including a neural network module, an experience replay module, and an agent control module;

[0009] S3: initialize the dynamic simulation environment and the path planning model;

[0010] S4: train the path planning model based on the improved DDQN and BOAE mechanism fusion strategy;

[0011] S5: load the trained mobile robot path planning model, the agent makes decisions based on the optimal action output by the policy network to drive the robot to run in the dynamic environment, the environment simulation module feedbacks the robot's position and state in real time, the visualization module dynamically records the robot's path, and after the test is completed, the visualization module displays the robot's optimal path.

[0012] Further, the step S1 of building a dynamic simulation environment includes:

[0013] The starting coordinates (such as np.array([-2.2, -3.3])) and target coordinates (such as np.array([2.88, 3.2])) of the mobile robot are accurately set to clearly define the start and end points of the path planning, providing a basic target orientation for subsequent path planning and navigation;

[0014] For dynamic obstacles, their attribute list is defined in detail, covering geometric information (such as shape, size) and dynamic motion rules (including initial position, velocity, acceleration, etc. Motion parameters); These rich obstacle information provides a key basis for the robot in the obstacle avoidance decision-making process, enabling it to predict the motion trajectory and trend of the obstacle in advance, and thus develop more accurate and efficient obstacle avoidance strategies;

[0015] For various sensors, the GPS sensor sampling period is set to 1ms, and the touch sensor and distance sensor collect data synchronously; The distance sensor is obtained by strict screening and sorting, the screening rule is that the name contains'so' and contains a digital character, and is arranged in ascending order according to the digital part; The collected sensor data is normalized and integrated into the observation vector, according to the maximum and minimum values of the range of each sensor, the data range is standardized to [0, 1] to meet the input requirements of the neural network, ensuring the consistency and effectiveness of the data, facilitating efficient processing and analysis by the neural network, thereby providing accurate data support for the robot's decision-making;

[0016] The action space is defined as spaces.Discrete(3), corresponding to three discrete actions of forward, left turn and right turn. This concise and effective action definition can meet the basic motion requirements of the robot in most scenarios; the observation space shape is set as ([X]), which comprehensively integrates key elements such as target distance, sensor data, current position and obstacle avoidance related information provided by the BOAE mechanism, etc. [X] is the observation space dimension considering the BOAE information, which is determined according to the specific output of BOAE, so that the robot can comprehensively and accurately plan the path based on multiple information, and improve its perception and response ability to complex environment.

[0017] Further, the neural network module in step S2 includes a constructed policy network and a target network, which is built on the basis of a PyTorch framework to form a multi-layer fully connected layer structure. In the multi-layer fully connected layer structure, the first layer fc1 maps the input dimension to 256 dimensions, the middle layer fc2 maintains 256 dimensions for processing, and the last layer fc3 outputs a dimension matching the action space dimension. The activation function between layers is ReLU. The policy network generates Q values corresponding to each discrete action based on the observation vector fused with BOAE information, providing accurate basis for robot action decision. The target network is used to assist the training of the policy network, and the initial parameters of the policy network are copied, and the parameters are adjusted according to a specific update rule subsequently. Through this double network structure and reasonable network layer design, the learning ability and decision accuracy of the network can be effectively improved, and with the assistance of the target network, the training process is stabilized, and the occurrence of problems such as Q value overestimation is reduced.

[0018] Further, the experience replay module in step S2 uses a priority experience replay mechanism based on a SumTree data structure to store rich experience data generated by the robot interacting with the dynamic environment; each experience data unit covers core information such as current state, executed action, obtained reward, next state, and whether the task is completed. This module can efficiently sample experience batches according to the set priority, generate importance sampling weights by combining experience priority during sampling, effectively balance the bias caused by priority sampling, and dynamically update the priority of stored experience according to the training process. Especially for experience related to dynamic obstacles and the BOAE mechanism plays an important role in obstacle avoidance, the priority of the experience can be adjusted appropriately, so that high-value experience is more easily sampled. This experience replay mechanism can make full use of valuable experience generated during the interaction between the robot and the environment, improve the sample utilization rate, and speed up the network training convergence speed, and through reasonable priority adjustment, further improve the adaptability and response ability to dynamic obstacle scenarios.

[0019] Further, the intelligent agent control module in step S2 cooperates with each module to reasonably configure key training parameters, including discount factor, learning rate, exploration rate and its decay rate, minimum exploration rate, target network update frequency, memory capacity of experience replay, etc., and introduces n-step parameter, soft update parameter τ and regularization coefficient λ to realize specific improvement strategies.

[0020] Further, the path planning model is trained in step S4 by combining n-step reward calculation mechanism, soft update and regularization combination strategy and BOAE mechanism fusion strategy, wherein,

[0021] n-step reward calculation mechanism: The existing single-step reward calculation often cannot fully consider the long-term impact of robot action, while the n-step reward calculation mechanism of the present application can comprehensively consider the cumulative reward effect brought by the action of the robot in the next n steps. In this way, the action strategy of the robot can be more comprehensively evaluated, the overestimation problem of Q value caused by single-step reward calculation in the DDQN algorithm can be alleviated, the robot can make more forward-looking and long-term decisions in the path planning process, and thus the accuracy and stability of path planning can be improved.

[0022] Soft update and regularization combination strategy: In the target network update link, the traditional hard update method is abandoned, and a soft update strategy is adopted. The traditional hard update method may cause sudden changes in target network parameters, affecting the stability of training. The soft update strategy introduces a soft update parameter τ, so that the target network parameters can gradually and smoothly approach the policy network parameters, avoiding sharp fluctuations in the parameter update process; at the same time, a regularization term is added in the loss function calculation, and the loss function is adjusted by a regularization coefficient λ; the regularization term can effectively prevent network overfitting, so that the network can not only adapt to the training data, but also maintain the generalization ability to unknown data. Through the combination of soft update and regularization, the training process is stabilized, the Q value overestimation phenomenon is reduced, and overfitting is prevented, improving the accuracy and efficiency of path planning of the robot in complex dynamic environments.

[0023] BOAE mechanism fusion strategy: In the training stage, the agent selects the action to drive the robot action according to the exploration rate rule combined with the BOAE mechanism's obstacle avoidance suggestion action or action adjustment information. The BOAE mechanism analyzes the obstacle avoidance response from the state and action, uses cross-attention and adversarial network, and generates auxiliary reward factors to strengthen the robot's obstacle avoidance ability in multiple dimensions. Combined with the improved DDQN algorithm, the robot can more accurately perceive the environment and more effectively adjust the action strategy when facing dynamic obstacles, further improving the obstacle avoidance ability and the adaptability of path planning.

[0024] Further, the training of the path planning model in step S4 includes the following processes:

[0025] The agent selects an action cautiously according to the current observation vector, the exploration rate and the obstacle avoidance suggestion action or action adjustment information provided by the BOAE mechanism; after the robot executes the selected action in the dynamic environment, the environment simulation module feeds back the reward, the next state and whether the end information in real time; the agent stores the interactive experience into the experience replay module, the experience replay module samples the experience according to the priority, and provides a data batch for training; during the training process, the n-step reward, the time difference error and the like are calculated, the soft update strategy and the regularization term are used to update the parameters of the strategy network and the target network, the experience priority is adjusted according to the evaluation of the obstacle avoidance effect of the BOAE mechanism, and the convergence of the strategy network is accelerated; in this process, the agent continuously interacts with the environment, and gradually improves the path planning capability through learning and optimization, and the close cooperation between the modules and the parameter adjustment are the key to efficient training.

[0026] In combination with the above content, the innovation points of the present application mainly lie in the following three aspects: 1. The n-step reward calculation mechanism is introduced, the long-term cumulative reward of the robot action is comprehensively considered, and the Q value overestimation problem is effectively alleviated; 2. The soft update and regularization are combined, the target network parameters are updated smoothly, and the network overfitting is prevented, thereby improving the stability and accuracy of the training; 3. The DDQN and the BOAE mechanism are organically combined, the obstacle avoidance ability and the path planning adaptability of the robot are strengthened from multiple dimensions, and strong technical support is provided for the autonomous navigation of the mobile robot in the complex dynamic environment.

[0027] Beneficial effects: Compared with the prior art, through the above three innovative aspects, through the innovative way of combining the BOAE and the improved DDQN algorithm, the mobile robot can more efficiently and accurately plan the path in the dynamic environment, significantly improve the obstacle avoidance ability and the success rate of reaching the target, and has extremely broad application prospect and popularization value. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1 It is a flowchart of the method of the present application;

[0029] Figure 2 It is a workflow diagram of the BOAE mechanism;

[0030] Figure 3 It is a multi-layer fully connected layer structure diagram;

[0031] Figure 4 It is a smooth curve diagram of the rewards of the three DRL algorithms;

[0032] Figure 5 It is a trajectory planning effect diagram of the three DRL algorithms;

[0033] Figure 6 It is a trajectory length diagram planned by the three DRL algorithms; DETAILED DESCRIPTION

[0034] The present application will be further clarified by the following examples, which should be considered as merely illustrative of the present application and not limiting thereof. Modifications of the present application, in various forms, will be apparent to those skilled in the art, upon reading the present disclosure, and it is intended that the scope of the application disclosed herein encompass such modifications.

[0035] As shown in Figure 1 The present application provides a dynamic path planning method for mobile robots based on improved DDQN algorithm, comprising the following steps:

[0036] S1: Build a dynamic simulation environment;

[0037] The starting coordinates (such as np.array([-2.2,-3.3])) and target coordinates (such as np.array([2.88,3.2])) of the mobile robot are accurately set to clearly define the start and end points of the path planning, providing a basic target orientation for subsequent path planning and navigation;

[0038] For dynamic obstacles, their attribute list is defined in detail, covering geometric information (such as shape, size) and dynamic motion law (including initial position, velocity, acceleration, etc. Motion parameters); These rich obstacle information provides a key basis for the robot in the obstacle avoidance decision-making process, enabling it to predict the motion trajectory and trend of the obstacle in advance, so as to develop more accurate and efficient obstacle avoidance strategies;

[0039] For various sensors, the sampling period of the GPS sensor is set to 1ms, and the touch sensor and the distance sensor collect data synchronously according to this; The distance sensor is obtained by strict screening and sorting, the screening rule is that the name contains'so' and contains digital characters, and is arranged in ascending order according to the digital part; The collected sensor data is fused into the observation vector after normalization processing, according to the maximum and minimum values of the range of each sensor, the data range is standardized to [0,1] to meet the input requirements of the neural network, ensure the consistency and effectiveness of the data, and facilitate the neural network to process and analyze efficiently, so as to provide accurate data support for the decision of the robot;

[0040] The action space is defined as spaces.Discrete(3), corresponding to three discrete actions: forward, left turn, and right turn. This concise and effective action definition can meet the basic motion requirements of the robot in most scenarios. The shape of the observation space is set as ([X]), which fully integrates key elements such as target distance, sensor data, current position, and obstacle avoidance information provided by the BOAE mechanism. Among them, [X] is the dimension of the observation space after considering BOAE information, which is determined according to the specific output of BOAE. This enables the robot to make comprehensive and accurate path planning decisions by integrating multiple aspects of information, thereby improving its perception and response capabilities to complex environments.

[0041] S2: Construct a path planning model, including a neural network module, an experience playback module, and an agent control module;

[0042] The neural network module includes a policy network and a target network, built using the PyTorch framework to create a multi-layer fully connected layer structure; for example... Figure 3 As shown, in the multi-layer fully connected structure, the first layer fc1 maps the input dimension to 256 dimensions, the middle layer fc2 maintains 256-dimensional processing, and the last layer fc3 outputs a dimension that matches the action space dimension. ReLU is used as the activation function between layers. The policy network generates Q-values ​​for each discrete action based on the observation vector fused with BOAE information, providing accurate basis for robot action decisions. The target network is used to assist the training of the policy network. Initially, the policy network parameters are copied, and subsequently, the parameters are adjusted according to a specific update rule. Through this dual-network structure and reasonable network layer design, the learning ability and decision accuracy of the network can be effectively improved. At the same time, with the assistance of the target network, the training process is stabilized, and problems such as Q-value overestimation are reduced.

[0043] Multi-layer fully connected layers extract features and learn representations from the input data through each layer, realizing feature combination, transformation and mapping from low level to high level; in classification and regression tasks, based on the matching of output dimension and task, the data is classified or numerically predicted; with powerful function fitting ability, the data is fitted, and at the same time, the generalization ability is ensured through regularization and other means, learning the general rules of the data to accurately predict and classify.

[0044] The policy network generates Q-values ​​for each discrete action based on the observation vectors fused with BOAE information. The calculation process for the Q-values ​​includes:

[0045] 1) Input processing

[0046] First, the observation vector x, which incorporates BOAE information, is used as input. Assume the observation vector x = {x1, x2, ..., x...} n}, where n is the dimension of the observation vector;

[0047] 2) Calculation of the first fully connected layer

[0048] The observation vector x is input to the first fully connected layer fc1, each neuron j in fc1 will do a weighted sum of the input and add a bias term b 1j i.e. where w 1ij is the connection weight from the i-th input to the j-th neuron in fc1;

[0049] Then apply the ReLU activation function: a 1j = ReLU(z 1j ) = max(0, z 1j ), get the output of fc1: a1 = [a 11 , a 12 ,..., a 1m ], where m = 256, the output dimension of fc1;

[0050] 3) Middle layer fully connected layer calculation

[0051] The calculation of the middle layer fc2 is similar to fc1, for each neuron k in fc2, w 2jk is the connection weight from the j-th input to the k-th neuron in fc2, b 2k is the bias term;

[0052] Again apply the ReLU activation function: a 2k = ReLU(z 2k ) = max(0, z 2k ), get the output of fc2: a2 = [a 21 , a 22 ,..., a 2m ]

[0053] 4) Last layer fully connected layer calculation

[0054] The calculation of the last layer fc3, for each action corresponding dimension l of the output, w 3kl is the connection weight from the k-th input to the l-th output in fc3, b 3l is the bias term. Here the range of l corresponds to the action space dimension;

[0055] At this time, z 3l is the Q value corresponding to each discrete action, i.e. Q l = z 3l , the final Q value vector Q = [Q1, Q2,..., Q s ] is obtained, s is the dimension of the action space, which represents the value of the robot taking each discrete action under the current observation.

[0056] The experience replay module uses a priority experience replay mechanism based on a SumTree data structure to store rich experience data generated by the robot interacting with the dynamic environment. Each experience data unit covers core information such as the current state, executed action, obtained reward, next state, and whether the task is completed. This module can efficiently sample experience batches according to the set priority. When sampling, it generates importance sampling weights by combining experience priority, effectively balancing the bias caused by priority sampling. It can also dynamically update the priority of stored experiences based on the training process. Especially for experiences related to dynamic obstacles and the BOAE mechanism plays an important role in obstacle avoidance, the priority can be adjusted appropriately to make high-value experiences more easily sampled. This experience replay mechanism can fully utilize the valuable experience generated during the interaction between the robot and the environment, improve sample utilization, and speed up network training convergence. At the same time, through reasonable priority adjustment, it further improves the adaptability and response ability to dynamic obstacle scenarios.

[0057] The agent control module cooperates with each module to operate and reasonably configures key training parameters, including discount factor, learning rate, exploration rate and its decay rate, minimum exploration rate, target network update frequency, experience replay memory capacity, etc. It also introduces n-step parameters, soft update parameter τ, and regularization coefficient λ to implement specific improvement strategies.

[0058] S3: Initialize dynamic simulation environment and path planning model;

[0059] 1. Dynamic simulation environment initialization

[0060] 1) Coordinate setting: accurately set the starting coordinates and target coordinates of the mobile robot, such as setting the starting coordinates to np.array([-2.2, -3.3]) and the target coordinates to np.array([2.88, 3.2]) to clearly define the start and end points of path planning.

[0061] 2) Obstacle definition: define the attribute list of dynamic obstacles in detail, including geometric information (such as shape, size) and dynamic motion law (including initial position, velocity, acceleration, etc. Motion parameters).

[0062] 3) Sensor setting: set the GPS sensor sampling period to 1ms, and the touch sensor and distance sensor collect data synchronously based on this. Sort the distance sensor strictly, the filtering rule is that the name contains'so' and contains digital characters, and arrange them in ascending order according to the digital part. The collected sensor data is normalized and integrated into the observation vector.

[0063] 4) Space definition: The action space is defined as spaces.Discrete(3), corresponding to the three discrete actions of forward, left turn, and right turn. The observation space shape is set as ([X]), which comprehensively integrates key elements such as target distance, sensor data, current position, and obstacle avoidance information provided by the BOAE mechanism.

[0064] 2. Path planning model initialization

[0065] Neural network module: The policy network and target network are built based on the PyTorch framework with multiple fully connected layers. The first layer fc1 maps the input dimension to 256 dimensions, the middle layer fc2 maintains 256 dimensions, and the last layer fc3 outputs a dimension matching the action space. The activation function between layers is ReLU. The target network is initially copied from the policy network parameters.

[0066] Experience replay module: Initialize the priority experience replay mechanism based on the SumTree data structure and set parameters such as memory capacity for experience replay.

[0067] Agent control module: Reasonably configure key training parameters, including discount factor, learning rate, exploration rate and its decay rate, minimum exploration rate, target network update frequency, etc. At the same time, initialize n-step parameter, soft update parameter τ, and regularization coefficient λ.

[0068] S4: Train the path planning model based on the improved DDQN and BOAE mechanism fusion strategy, including:

[0069] Train the path planning model based on the n-step reward calculation mechanism, soft update and regularization combination strategy, and BOAE mechanism fusion strategy, where,

[0070] n-step reward calculation mechanism: Existing single-step reward calculation often fails to fully consider the long-term impact of robot actions, while the n-step reward calculation mechanism of the present invention can consider the cumulative reward effect of robot actions in consecutive n steps. In this way, the robot's action strategy can be more comprehensively evaluated, alleviating the Q-value overestimation problem caused by single-step reward calculation in the DDQN algorithm, enabling the robot to make more forward-looking and long-term decisions in the path planning process, thereby improving the accuracy and stability of path planning.

[0071] The n-step reward calculation mechanism is represented as: assuming that the single-step reward at the t-th step is r t , and the discount factor γ, then the n-step reward R t can be represented as

[0072] Soft update and regularization combination strategy: In the target network update link, abandon the traditional hard update method, adopt soft update strategy. The traditional hard update method may cause sudden changes of target network parameters, affecting the stability of training. The soft update strategy introduces a soft update parameter tau, which makes the target network parameters gradually and smoothly approach the policy network parameters, avoiding the violent fluctuations in the parameter update process. At the same time, a regularization term is added in the loss function calculation, and the loss function is adjusted by the regularization coefficient lambda. The regularization term can effectively prevent network overfitting, so that the network can not only adapt to the training data, but also maintain the generalization ability to unknown data. Through the combination of soft update and regularization, the training process is stabilized, the Q value overestimation phenomenon is reduced, and overfitting is prevented, which improves the accuracy and efficiency of path planning of the robot in complex dynamic environment.

[0073] Soft update strategy: The update formula of the target network parameter theta' is theta' <- (1-tau) theta' + tau theta, where theta' is the policy network parameter, tau is the soft update parameter, and the value is usually small, so that the target network parameter can gradually and smoothly approach the policy network parameter.

[0074] Regularization term: A regularization term is added in the calculation of the loss function L. Assuming that the original loss function is L0, the regularization term is Omega (theta), and the regularization coefficient is lambda, then the loss function after adding regularization is L = L0 + lambda Omega (theta). By adjusting lambda, the strength of regularization is controlled to prevent network overfitting.

[0075] Reference Figure 2 BOAE mechanism fusion strategy: In the training stage, the agent selects the action to drive the robot to act according to the exploration rate rule combined with the obstacle avoidance suggestion action or action adjustment information of the BOAE mechanism. The BOAE mechanism analyzes the obstacle avoidance reaction from the state and action, and uses cross attention and pair decision network and generates auxiliary reward factor to strengthen the robot obstacle avoidance ability in multiple dimensions. Combined with the improved DDQN algorithm, the robot can more accurately perceive the environment and more effectively adjust the action strategy when facing dynamic obstacles, further improving the obstacle avoidance ability and the adaptability of path planning.

[0076] The improvement of the improved DDQN algorithm in the present application is reflected in:

[0077] First: Introduce n-step reward and combine multi-step return with priority experience replay. (Multi-step return is an important return calculation method in reinforcement learning, which considers the cumulative reward of future multiple time steps to optimize the target function more comprehensively)

[0078] Second: Soft update strategy is adopted in network update combined with regularization.

[0079] The above improvements are to solve the Q value overestimation problem of the traditional DDQN algorithm

[0080] The combination of the BOAE mechanism and the improved DDQN algorithm:

[0081] In the training phase, the agent selects actions to drive the robot according to the exploration rate rule combined with the BOAE mechanism obstacle avoidance suggestion action or action adjustment information. The BOAE mechanism analyzes the obstacle avoidance response from the state and action, and the information provided is integrated into the observation vector. The policy network generates Q values corresponding to each discrete action based on the observation vector fused with BOAE information, to drive the robot's action decision.

[0082] The workflow of the BOAE mechanism is described below:

[0083] 1. Analysis of obstacle avoidance response unit

[0084] 1.1 State evaluation

[0085] The robot collects environmental and self-state information through various sensors such as GPS, touch, distance sensor, etc. After normalization processing, these information is integrated into the observation vector s. Taking distance sensor data d as an example, the normalization formula is where d min and d max are the minimum and maximum values of the sensor range, ensuring that the data is in the range [0, 1] for easy neural network processing.

[0086] The system determines whether the robot is close to the obstacle based on the observation vector s. Set the distance threshold D thresh When the normalized distance d norm between the robot and the obstacle is less than D thresh , it is determined that the robot is in a potentially dangerous area and needs to start the obstacle avoidance program.

[0087] 1.2 Action evaluation

[0088] The action space is defined as a discrete set A = {forward, left turn, right turn}, corresponding to a ∈ {0, 1, 2}. This unit analyzes whether the current action selection will trigger a collision risk. For example, if the robot chooses to move forward while detecting an obstacle in front, this action has a high risk. Based on the current state s and action a, it is determined whether the robot is in a dangerous state. If it is in a dangerous state, the action needs to be adjusted according to the real-time situation of the environment to avoid collision.

[0089] 2. Obstacle avoidance direction suggestion

[0090] 2.1 Direction judgment based on sensor data

[0091] According to the distance sensor data in multiple directions, the orientation of the obstacle relative to the robot is determined. Assume there are n distance sensors around the robot, and the data are d1, d2,..., dn respectively. n . Compare the sensor data, if d i is the smallest and significantly smaller than other sensor data, it indicates that the obstacle is in the direction corresponding to sensor i.

[0092] The robot is planned to move away from this direction. For example, if the obstacle is in front (assuming the front sensor corresponds to d j is the smallest), the recommended action is to prefer to turn left or right to avoid direct collision.

[0093] 2.2 Optimization combined with environmental information

[0094] Considering the target position information of the robot, the obstacle avoidance direction is selected that can both avoid obstacles and approach the target direction as much as possible. Let the current position of the robot be P = (x, y), the target position be T = (x t , y t ), and the obstacle position be O = (x0, y0). Calculate the vector from the current position of the robot to the target position and the vector from the current position of the robot to the obstacle position

[0095] Through vector operations, such as calculating the angle between the two vectors If the angle is small, it indicates that the obstacle is in the direction close to the target direction, and the obstacle avoidance direction should be more inclined to maintain the proximity to the target direction.

[0096] 3. Urgency assessment of obstacle avoidance

[0097] 3.1 Distance-velocity evaluation model

[0098] According to the distance d between the robot and the obstacle and the relative speed v rel of the obstacle, the urgency of obstacle avoidance is evaluated. Define the urgency index U, the formula is where v rel is the relative speed considering the motion of the obstacle and the motion of the robot itself. When U exceeds the set threshold U thresh , it indicates that the obstacle avoidance situation is urgent and immediate action is needed; when U is low, the obstacle avoidance path can be planned relatively calmly.

[0099] 3.2 Time-collision evaluation

[0100] Predict the collision time T collision. Assume the trajectories of robot and obstacle are straight lines (simplified model, more accurate trajectories can be obtained by motion prediction model), calculate the collision time T according to their current positions, velocities and moving directions. If T collision is less than the set safety time threshold T safe , the obstacle avoidance urgency is high, and a quick decision is needed; otherwise, the regular strategy can be used to plan obstacle avoidance.

[0101] 4. Action value evaluation optimization

[0102] 4.1 Introducing auxiliary features

[0103] In the calculation of action value Q(s, a), in addition to state s and action a, auxiliary features related to obstacle avoidance are introduced. For example, the distance d min between the robot and the nearest obstacle, the relative velocity v rel , and the angle θ between the obstacle avoidance direction and the target direction are taken as additional input features. These features are integrated into the input layer of the neural network, and participate in the calculation through an additional weight matrix together with the original input features, to more comprehensively evaluate the value of the action in the current obstacle avoidance scenario.

[0104] 4.2 Priority-based value adjustment

[0105] According to the results of obstacle avoidance urgency evaluation, the action value is adjusted. For high urgency, more attention is paid to the value improvement of actions that can immediately avoid collision; for low urgency, the long-term goal optimization action value can be considered.

[0106] Let the urgency adjustment factor be α (0 < α < 1, the higher the urgency, the closer α is to 1), the adjusted action value Q'(s, a) = (1-α)Q(s, a) + αQ urgent (s, a), where Q urgent (s, a) is the action value calculated for the emergency obstacle avoidance case.

[0107] 5. Reward factor generation unit

[0108] 5.1 Distance reward

[0109] Design a distance reward function r d to encourage the robot to move away from the obstacle. When the distance d between the robot and the obstacle is greater than the safety distance d safe , r d = k1(d-d safe ), where k1 is a positive coefficient representing the reward obtained by the robot for moving away from the obstacle; when d ≤ d safe , r d = -k2(d safe -d), where k2 is a positive coefficient representing the punishment suffered by the robot for approaching the obstacle.

[0110] 5.2 Direction reward

[0111] The direction reward function r is defined by considering the consistency between the obstacle avoidance direction and the target direction. θ Let the angle between the current direction of the robot and the target direction be θ. When the robot chooses a direction that makes θ smaller during obstacle avoidance, a positive reward is given; otherwise, a negative reward is given. For example, r θ = k3cosθ, where k3 is a positive coefficient, and cosθ measures the closeness of the direction.

[0112] 5.3 Comprehensive reward factor

[0113] The distance reward r d , the direction reward r θ , and other related rewards (such as action smoothness reward, etc.) are combined to obtain the final auxiliary reward factor r aux = r d + r θ + r other This auxiliary reward factor serves as an additional reward signal, combined with the reward r fed back by the environment, to train the agent and encourage the agent to pay more attention to obstacle avoidance and target-oriented behavior during learning.

[0114] The training of the path planning model includes the following processes:

[0115] The agent carefully selects an action based on the current observation vector, the exploration rate, and the obstacle avoidance suggestion action or action adjustment information provided by the BOAE mechanism. After the robot executes the selected action in the dynamic environment, the environment simulation module provides real-time feedback of the reward, the next state, and whether the interaction is over. The agent stores the interaction experience in the experience replay module, which samples the experience according to the priority to provide data batches for training. During training, the n-step reward and the time difference error are calculated, and the policy network and the target network parameters are updated using the soft update strategy and the regularization term. At the same time, the experience priority is adjusted according to the evaluation of the BOAE mechanism on the obstacle avoidance effect to accelerate the convergence of the policy network. In this process, the agent continuously interacts with the environment, gradually improving the path planning ability through learning and optimization. The close cooperation and parameter adjustment among the various modules are the key to efficient training.

[0116] The specific operation process of the environment simulation module includes:

[0117] 1) Environment state feedback

[0118] Position feedback:

[0119] The environment simulation module can track the robot's position information in real time in a dynamic environment. For example, in a two-dimensional plane environment, it records the robot's coordinates (x, y) and feeds this information back to the agent. This position information is crucial for the agent to understand the robot's current location and plan the next action.

[0120] State feedback:

[0121] In addition to position, other states of the robot such as speed, direction, etc. are also fed back. If the robot is in an environment with complex terrain or special conditions, the environment simulation module will also feed back state information such as whether the robot is on a slope, whether it is affected by a magnetic field, etc. These state information together with the robot's position information constitute a complete state feedback, helping the agent to fully understand the current state of the robot.

[0122] 2) Reward feedback

[0123] Reward generation principle:

[0124] The reward signal is generated according to the robot's action and environmental changes. For example, if the robot successfully avoids a dynamic obstacle, the environment simulation module will generate a positive reward r (such as r = +1), and the size and sign of the reward value are determined according to the pre-set reward mechanism.

[0125] The reward mechanism is usually related to factors such as whether the robot is close to the target, whether it has successfully avoided obstacles, whether the action is efficient, etc. If the robot moves towards the target direction, it may receive a small positive reward; if the robot collides with obstacles or moves away from the target, it may receive a negative reward (such as r = -1).

[0126] Reward feedback role:

[0127] The reward signal is fed back to the agent to update the agent's policy network. The agent learns the action strategy that can obtain more rewards by constantly trying different actions and receiving the reward feedback from the environment simulation module, thereby optimizing the path planning.

[0128] 3) Next state provision

[0129] State transition simulation:

[0130] When the agent selects an action a (such as moving forward, turning left, or turning right), the environment simulation module will simulate the changes in the environment and the updates of the robot's state according to the robot's current state and the selected action, to obtain the next state s' of the robot.

[0131] This process involves complex environment models and robot motion models. For example, for a robot moving in a simple two-dimensional plane environment, if the current position of the robot is (x, y) and the velocity is (v x ,v y ), after selecting a forward action, the next position may be updated to (x + v x Δt, y + v y Δt), where Δt is the time step. At the same time, the environment simulation module will also update the position of dynamic obstacles in the environment, considering their own motion rules, such as updating the position according to the velocity and acceleration. The importance of providing the next state:

[0132] The next state s' together with the current state s, the executed action a and the obtained reward r forms an experience data unit (s, a, r, s', done) which is stored in the experience replay module. These experience data are important basis for training the agent, helping the agent to learn the relationship between state-action-reward, so as to continuously optimize the path planning strategy.

[0133] 4) Task completion judgment

[0134] Judgment basis:

[0135] The environment simulation module will judge whether the task is completed according to whether the robot has reached the target position. For example, set the target position as (x t ,y t ), when the position of the robot (x, y) satisfies certain distance conditions (such as the Euclidean distance is less than a certain threshold), it is determined that the task is completed, and the done flag is set to True. Impact on training process:

[0136] The done flag is also part of the experience data unit. When done = True, the processing of this experience data unit in the training process may be different. For example, when calculating the cumulative reward, the reward after the task is completed may be specially processed, or the priority of this experience data unit in the experience replay module may be adjusted according to whether the task is successfully completed, thereby affecting the training process of the agent.

[0137] Based on the above, the training of the path planning model specifically includes the following steps:

[0138] Step 1: Build a dynamic simulation environment

[0139] 1) Set the robot coordinates:

[0140] Precisely set the starting coordinates S start of the mobile robot = np.array([x s ,y s]) and target coordinate S start = np.array([x t ,y t ]) to determine the start and end points of path planning.

[0141] 2) Define obstacle attributes:

[0142] Define the attribute list of dynamic obstacles in detail. For each obstacle, let its shape be shape, size be size, initial position be O init = np.array([x o0 ,y o0 ]), velocity be and acceleration be Through these parameters, the position of the obstacle at time t can be predicted

[0143] 3) Process sensor data:

[0144] The sampling period of the GPS sensor is set to T GPS = 1 ms, and the touch sensor and distance sensor collect data synchronously.

[0145] Filter and sort the distance sensor data. Let the distance sensor set be {S d1 ,S d2 ,...}, filter out the sensor data whose name contains'so' and contains a numeric character, assume the filtered data is and arrange it in ascending order of the numeric part. Normalize the collected sensor data. For distance sensor data d, the normalization formula is where d min and d max are the minimum and maximum values of the sensor range respectively. Integrate the normalized data into the observation vector s.

[0146] 4) Define action and observation space:

[0147] The action space is defined as A = spaces.Discrete(3), corresponding to three discrete actions: forward (a = 0), left turn (a = 1), and right turn (a = 2).

[0148] The observation space shape is set to ([X]), and the observation vector s integrates key elements such as target distance sensor data, current position (x s ,y s ), and obstacle avoidance related information provided by the BOAE mechanism.

[0149] Step 2: Train the path planning model

[0150] 1) Constructing the neural network module:

[0151] The policy network policy_net and the target network target_net are both built with a multi-layer fully connected layer structure based on the PyTorch framework.

[0152] Let the input dimension be I, the first layer of the policy network fc1: h1 = ReLU(W1s + b1), where W is the weight matrix and b1 is the bias vector, which maps the input dimension I to 256 dimensions; the middle layer fc2: h2 = ReLU(W2h1 + b2) maintains 256 dimensions; the last layer fc3: Q(s, a) = W3h2 + b3, the output dimension matches the action space dimension, generating Q values corresponding to each discrete action.

[0153] The target network initially copies the parameters of the policy network, and then adjusts the parameters according to a specific update rule.

[0154] 2) Set the experience replay module:

[0155] Use the priority experience replay mechanism based on the SumTree data structure. Let the experience data unit be (s, a, r, s', done), where s is the current state, a is the executed action, r is the reward obtained, s' is the next state, and done indicates whether the task is completed.

[0156] According to the set priority p, sample the experience batch, and combine the experience priority to generate importance sampling weight ω i , such as where N is the number of sampled samples, p i is the priority of the i-th sample. At the same time, the priority of the stored experience can be dynamically updated according to the training process, and the priority of the experience related to the dynamic obstacle and the BOAE mechanism can be appropriately increased.

[0157] 3) Configure the agent control module:

[0158] Configure key training parameters, including discount factor γ, learning rate lr, exploration rate ε and its decay rate ε decay , minimum exploration rate ε min , target network update frequency target_update, memory capacity memory_size of experience replay, etc. Introduce step parameter n, soft update parameter τ and regularization coefficient λ.

[0159] 4) Use n-step reward calculation mechanism:

[0160] The existing single-step reward calculation is R1 = r, while in the n-step reward calculation mechanism, the n-step reward R n is where r t+iis the reward obtained in the i-th step from the current time t. In this way, the robot's action strategy is more comprehensively evaluated.

[0161] 5) Implementing soft update combined with regularization strategy:

[0162] In the target network update link, the soft update strategy is adopted, and the target network parameter θ target The update formula is θ target ←(1-τ)θ target +τθ policy , where θ policy is the policy network parameter, avoiding the violent fluctuation in the process of updating the target network parameter.

[0163] In the loss function calculation, a regularization term is added. Let the loss function be L, the original loss function be L original , and the regularization term be λ||θ|| 2 (θ is the network parameter). Then the adjusted loss function L = L original + λ||θ|| 2 , which prevents network overfitting and enables the network to not only adapt to training data but also maintain the generalization ability to unknown data during the learning process.

[0164] 6) Fusion BOAE mechanism strategy:

[0165] In the training phase, the agent selects actions to drive the robot to act by combining the BOAE mechanism's obstacle avoidance suggestion actions or action adjustment information with the exploration rate rule. The BOAE mechanism analyzes the obstacle avoidance response from the state and action, and uses cross-attention and adversarial networks, as well as generating auxiliary reward factors, to strengthen the robot's obstacle avoidance ability. For example, through the cross-attention mechanism, the attention score between the observation vector s and the action preference information is calculated, and then through the adversarial network, the value function is decomposed into the state value function V(s) and the advantage function A(s,a), i.e. Q(s,a) = V(s) + A(s,a), and the advantage function is adjusted so that its mean value is 0, i.e. The final Q value is Q(s,a) = V(s) + A'(s,a). At the same time, auxiliary reward factors r aux , such as distance reward r d (d > d safe , r d = k1(d-d safe ); when d ≤ d safe , r d = -k2(d safe -d)) and direction reward r θ (such as r θ = k3cosθ, θ is the angle between the current direction of the robot and the target direction) are generated to obtain raux = r d + r θ + r other , combined with the reward r from the environment, is used for training.

[0166] Step three: Obtain the path planning result

[0167] 1) Run the robot:

[0168] Load the trained mobile robot path planning model, and the agent makes decisions based on the optimal action output by the policy network Drive the robot to run in a dynamic environment.

[0169] 2) Feedback and record:

[0170] The environment simulation module provides real-time feedback on the robot's position and state, and the visualization module dynamically records the robot's path.

[0171] Performance evaluation and saving:

[0172] After the test, the visualization module displays the optimal path of the robot, and calculates the running performance indicators such as path length ((x i ,y i ) is the point on the path), time to reach the target t total , collision times n collision , minimum safety distance d min-safe from dynamic obstacles, etc., and saves these indicators in the form of charts for detailed data support for algorithm performance analysis and subsequent improvement.

[0173] Based on the above, the update of the policy network and the target network parameters can be summarized as follows:

[0174] Calculate n-step reward: Calculate the cumulative reward R of the robot in the next n steps according to the n-step reward calculation mechanism t .

[0175] Calculate the time difference error: Calculate the time difference error Q θ (s t ,a t ) is the Q value of the policy network in state s t , performing action a t , and Q θ' (s t+n ,a') is the Q value of the target network in state s t+n .

[0176] Updating network parameters: according to the time difference error δ, the strategy network parameters θ are updated by using gradient descent method, and the target network parameters θ' are updated according to the soft update strategy θ' <- (1- τ) θ' + τ θ.

[0177] Evaluation of the BOAE mechanism on obstacle avoidance effect: the BOAE mechanism analyzes the obstacle avoidance reaction from the state and action, and evaluates the obstacle avoidance effect through cross attention and pair network and generates auxiliary reward factors in multiple dimensions. For example, if the robot successfully avoids the dynamic obstacle according to the recommended action of the BOAE mechanism and maintains a safe distance from the obstacle, it can be considered that the obstacle avoidance effect is good, and the priority of the experience data related to the obstacle avoidance experience in the experience replay module is correspondingly improved; if the robot fails to effectively avoid the obstacle or is too close to the obstacle, the priority of the experience is reduced, so that high-priority obstacle avoidance experiences are sampled more in subsequent training, accelerating the convergence of the strategy network.

[0178] S5: Load the trained mobile robot path planning model, and the agent drives the robot to run in the dynamic environment according to the optimal action decision output by the strategy network. The environment simulation module feeds back the robot position and state in real time, and the visualization module dynamically records the robot path. After the test is completed, the visualization module displays the optimal path of the robot, and the running performance indicators such as path length, time to reach the target, collision times, and minimum safe distance from dynamic obstacles are recorded and saved locally in the form of charts, providing detailed data support for algorithm performance analysis and subsequent improvement. Through the display and data recording functions of the visualization module, not only can the path planning effect of the robot in a specific environment be intuitively understood, but also the advantages and disadvantages of the algorithm can be analyzed in depth based on the statistical data, providing a strong basis for further optimizing the algorithm and promoting the continuous development and improvement of mobile robot path planning technology.

[0179] In order to verify the effectiveness and actual effect of the method of the present application, the method of the present application and the existing DDQN algorithm dynamic path planning method are compared in simulation experiments, as follows:

[0180] In this embodiment, n-step rewards are combined with priority experience replay to apply to the DDQN algorithm (DDQN-NstepPRE), and simulation experiments are compared, and the experimental results are as follows:

[0181] DDQN, DDQN with priority replay, and DDQN-NstepPER algorithm of the present application are used for path planning under the same scenario, and the training reward value curves of the algorithms are as follows: Figure 4As shown, the horizontal axis is the number of training rounds, and the vertical axis is the average reward value accumulated per 200 rounds. The reward curves of the three algorithms eventually tend to be similar, but there are slight differences in convergence speed and reward values. It can be seen that the DDQN-NstepPER algorithm obtains a higher average reward and faster convergence speed compared to the DDQN algorithm and the DDQN algorithm with increased priority playback. After 4000 rounds of training, the DDQN-NstepPER algorithm gradually reduces the fluctuation trend of the average reward, while the DDQN algorithm and the DDQN algorithm with increased priority playback still have a large fluctuation trend. Moreover, after 6000 rounds of training, the average reward value is always higher than that of the DDQN algorithm and the DDQN algorithm with increased priority playback under the same conditions. This indicates that the DDQN-NstepPER algorithm exhibits better performance in path planning.

[0182] And by comparing the path length and path smoothness of the Pioneer 3-DX robot planned from the starting point to the target point, as shown in Figure 5 and Figure 6 It can be seen that although the paths planned by the three algorithms have certain similarities, the path planned by the DDQN-NstepPER algorithm exhibits better smoothness and continuity, and the planned path is shorter. Therefore, it can also be concluded that the performance of the DDQN-NstepPER algorithm provided by the present application is better than that of the other two algorithms.

[0183] According to the above experiments, the advantages of the method of the present application can be summarized as follows:

[0184] (1) Introduction of multi-step return priority experience playback: The present application combines multi-step return (N-step Returns) with priority experience playback (PER) and proposes the DDQN-NstepPER algorithm. In traditional Q learning, the calculation method of single-step return often cannot effectively capture long-term rewards, while multi-step return effectively reduces the variance caused by single-step return by aggregating reward information at multiple time steps. After combining with priority experience playback, those experiences that have a high contribution to strategy improvement can be preferentially used, thereby accelerating the learning process and improving training efficiency.

[0185] (2) Combination of DDQN: By combining the DDQN algorithm, the present application effectively solves the problem of overestimation of Q values, and further improves the stability of training. DDDQN estimates Q values by using a behavior network and a target network respectively, thereby avoiding the problem of overestimation caused by using the same network for both action selection and Q value estimation in traditional DQN. This improvement enables the algorithm to more stably and accurately update the strategy when dealing with long-term dependencies, delayed rewards, and other problems.

[0186] (3) Efficient experience replay mechanism: In the design of multi-step return priority experience replay, a SumTree data structure is adopted to efficiently implement the storage and sampling of priority experience. Through this structure, the sampling and updating of experience replay can be completed in O(logN) time complexity, greatly improving the efficiency of the training process.

[0187] Through these improvements, the DDQN-NstepPER algorithm proposed by the present application not only improves the training efficiency of the reinforcement learning model in complex environments, but also significantly improves the model's ability to perceive long-term rewards through the combination of priority replay and multi-step returns, enabling the agent to better handle complex decision-making tasks. It provides an effective method and tool for mobile robots to learn behavior in complex environments.

Claims

1. A dynamic path planning method for mobile robots based on an improved DDQN algorithm, characterized in that, Includes the following steps: S1: Set up a dynamic simulation environment; S2: Construct a path planning model, including a neural network module, an experience playback module, and an agent control module; S3: Initialize the dynamic simulation environment and path planning model; S4: The path planning model is trained based on the fusion strategy of the improved DDQN and BOAE mechanisms; S5: Load the trained mobile robot path planning model. The agent drives the robot to run in the dynamic environment based on the optimal action decision output by the policy network. The environment simulation module provides real-time feedback on the robot's position and status. The visualization module dynamically records the robot's path. After the test, the visualization module displays the robot's optimal path. In step S2, the neural network module includes a constructed policy network and a target network. A multi-layer fully connected structure is built using the PyTorch framework. In this structure, the first layer fc1 maps the input dimension to 256 dimensions, the middle layer fc2 maintains 256 dimensions, and the final layer fc3 matches the output dimension with the action space dimension. ReLU is used as the activation function between layers. The policy network generates Q-values ​​for each discrete action based on the observation vector fused with BOAE information, providing accurate basis for robot action decisions. The target network assists in training the policy network, initially replicating its parameters and subsequently adjusting them according to a specific update rule. In step S2, the policy network generates Q-values ​​for each discrete action based on the observation vectors fused with BOAE information. The calculation process for the Q-values ​​includes: 1) Input processing First, the observation vector x, which incorporates BOAE information, is used as input; assuming the observation vector x = {x1, x2, ..., x...} n }, where n is the dimension of the observation vector; 2) Calculation of the first fully connected layer The observation vector x is input into the first fully connected layer fc1. Each neuron j in fc1 performs a weighted sum of the input and adds a bias term b. 1j ,Right now: Where w 1ij It is the connection weight from the i-th input to the j-th neuron in fc1; Then apply the ReLU activation function: a 1j =ReLU(z) 1j ) = max(0, z 1j ), thus obtaining the output of fc1: a1 = [a 11 ,a 12 ,...,a 1m ], where m = 256, is the output dimension of fc1; 3) Calculation of the intermediate fully connected layer For each neuron k in fc2, w 2jk b is the connection weight from the j-th input to the k-th neuron in fc2. 2k It is a bias term; Apply the ReLU activation function again: a 2k =ReLU(z) 2k ) = max(0, z 2k ), thus obtaining the output of fc2: a2 = [a 21 ,a 22 ,...,a 2m ] 4) Calculation of the last fully connected layer The calculation of the final layer fc3, for each action in the output corresponding to dimension l, w 3kl b is the connection weight from the k-th input to the l-th output in fc3. 3l It is a bias term; At this time, z 3l This refers to the Q value corresponding to each discrete action, i.e., Q l =z 3l The final Q-value vector obtained is Q = [Q1, Q2, ..., Q...]. s ], where s is the dimension of the action space, representing the value of each discrete action taken by the robot under the current observation; In step S2, the agent control module coordinates the operation of each module and reasonably configures key training parameters, including discount factor, learning rate, exploration rate and its decay rate, minimum exploration rate, target network update frequency, memory capacity of experience replay, and introduces n-step parameter, soft update parameter τ and regularization coefficient λ to achieve specific improvement strategies. In step S4, the path planning model is trained by combining an n-step reward calculation mechanism, a soft update and regularization strategy, and a BOAE mechanism fusion strategy. n-step reward calculation mechanism: The soft update strategy combines soft update with regularization: The soft update strategy introduces a soft update parameter τ, which allows the target network parameters to gradually and smoothly approach the policy network parameters, avoiding drastic fluctuations during the parameter update process; at the same time, a regularization term is added to the loss function calculation, and the loss function is adjusted by the regularization coefficient λ. BOAE Mechanism Integration Strategy: The BOAE mechanism analyzes obstacle avoidance responses from the perspectives of state and action, and enhances the robot's obstacle avoidance capabilities from multiple dimensions, including cross-attention and adversarial networks, as well as the generation of auxiliary reward factors.

2. The method for dynamic path planning of a mobile robot based on the improved DDQN algorithm according to claim 1, characterized in that, The construction of the dynamic simulation environment in step S1 includes: Precisely set the starting and target coordinates of the mobile robot; For dynamic obstacles, a detailed list of their attributes is defined, covering geometric information and dynamic motion laws; For all types of sensors, the sampling period of the GPS sensor is set to 1ms, and the touch sensor and distance sensor collect data synchronously according to this. The distance sensor data is obtained through strict screening and sorting. The screening rule is that the name contains 'so' and contains numeric characters, and the data is arranged in ascending order according to the numeric part. The collected sensor data is normalized and then incorporated into the observation vector. Based on the maximum and minimum values ​​of each sensor's own range, the data range is standardized to [0,1] to meet the input requirements of the neural network. The action space is defined as spaces.Discrete(3), corresponding to three discrete actions: forward, left turn, and right turn. The shape of the observation space is set as ([X]), which fully integrates the target distance, sensor data, current position, and obstacle avoidance information provided by the BOAE mechanism. [X] is the dimension of the observation space after considering the BOAE information, which is determined according to the specific output of BOAE.

3. The method for dynamic path planning of a mobile robot based on the improved DDQN algorithm according to claim 1, characterized in that, In step S2, the experience playback module uses a priority experience playback mechanism based on the SumTree data structure to store rich experience data generated by the robot's interaction with the dynamic environment; each experience data unit covers the current state, executed actions, reward obtained, next state, and whether the task has ended.

4. The method for dynamic path planning of a mobile robot based on the improved DDQN algorithm according to claim 1, characterized in that, The training of the path planning model in step S4 includes the following process: Based on the current observation vector, exploration rate, and obstacle avoidance suggestions or action adjustment information provided by the BOAE mechanism, the agent carefully selects actions. After the robot performs the selected actions in the dynamic environment, the environment simulation module provides real-time feedback on rewards, the next state, and whether the process has ended. The agent stores the interaction experience in the experience playback module, which samples the experience according to priority to provide data batches for training. During training, the n-step reward and temporal difference error are calculated, and the parameters of the policy network and the target network are updated using a soft update strategy and regularization terms. At the same time, the empirical priority is adjusted according to the BOAE mechanism to evaluate the obstacle avoidance effect.

5. A dynamic path planning method for a mobile robot based on an improved DDQN algorithm according to claim 4, characterized in that, In the path planning model training of step S4: Calculate the n-step reward: Calculate the robot's cumulative reward R over n consecutive steps using the n-step reward calculation mechanism. t ; Calculate the time difference error: Calculate the time difference error Q θ (s t ,a t ) is the policy network in state s t Next, execute action a t Q value, Q θ' (s t+n ,a') is the target network in state s t+n The Q value below; Update network parameters: Based on the time difference error δ, update the policy network parameters θ using methods such as gradient descent, and at the same time update the target network parameters θ' according to the soft update strategy θ'←(1-τ)θ'+τθ.

Images