A Robot Path Planning Method Based on Improved Differential Evolution Algorithm and Deep Reinforcement Learning

Abstract

This invention discloses a robot path planning method based on an improved differential evolution algorithm and deep reinforcement learning, comprising the following steps: establishing a path planning map containing irregular obstacles; initializing multiple feasible solutions to the path planning problem as a population; designing a multi-policy improved differential evolution algorithm; utilizing the proximal policy optimization algorithm in deep reinforcement learning to open an independent buffer for each individual in the population; designing a state pool based on the population's position and fitness information; treating the multi-policy as an action pool; guiding the population to select appropriate policies under different states; and finally outputting the optimal path. This invention effectively solves the problem that differential evolution algorithms are prone to premature convergence and local optima when used for robot path planning, and improves the quality of path planning solutions for robots in complex environments.

Description

Technical Field

[0001] This invention belongs to the field of robot path planning technology, and in particular relates to a robot path planning method based on an improved differential evolution algorithm and deep reinforcement learning. Background Technology

[0002] With the rapid development of artificial intelligence and autonomous system technologies, mobile robots are increasingly being used in industrial automation, intelligent logistics, emergency rescue, and complex environment exploration. Robot path planning, as a core component for achieving autonomous navigation and task execution, has become a research hotspot and a key challenge in the field of robotics. Efficient and robust path planning methods can not only improve the operational efficiency and safety of robots but also have significant practical implications for promoting the intelligent upgrading of related industries.

[0003] Currently, various traditional algorithms and intelligent methods have emerged in the field of path planning. Among them, graph search or sampling-based methods, represented by Algebra, Digitization, and Regression-Range Theory (RRT), demonstrate good planning performance in structured or low-dimensional environments. Meanwhile, swarm intelligence optimization algorithms, such as Differential Evolutionary Algorithms, Genetic Algorithms, and Particle Swarm Optimization (PSO), by simulating natural evolution or group behavior mechanisms, exhibit strong global search capabilities and adaptability in multi-constraint, nonlinear path planning problems, and have become one of the important means of implementing path planning in complex scenarios.

[0004] However, existing path planning methods based on swarm intelligence, such as differential evolution, still have significant shortcomings. These methods are prone to getting trapped in local optima during iteration, especially in planning scenarios with dense obstacles and complex terrain. Decreased population diversity leads to premature convergence, making it difficult to obtain the globally optimal path. Furthermore, the algorithms typically use fixed or random evolutionary strategies, lacking adaptive judgment of the search state, thus limiting their convergence speed and the quality of the final solution, affecting the robot's planning performance and real-time capabilities in dynamic or large-scale environments. Summary of the Invention

[0005] Purpose of the invention: The purpose of this invention is to provide a robot path planning method based on an improved differential evolution algorithm and deep reinforcement learning, which solves the problems of existing path planning methods being prone to getting trapped in local optima and having slow convergence speed, thereby improving the global optimality and adaptive efficiency of robot path planning in complex environments.

[0006] Technical solution: The robot path planning method of the present invention includes the following steps:

[0007] S1. Construct a two-dimensional grid map environment based on the path finding requirements of the mobile robot, mark the location of obstacles in the impassable areas of the two-dimensional grid map environment, determine the starting position and target position of the mobile robot, and establish a fitness function based on the two-dimensional grid map environment.

[0008] S2. Initialize the population using a roulette wheel selection strategy, including: starting from the starting position, selecting feasible nodes in the 8-neighborhood of the current node, calculating the Euclidean distance from each feasible node to the target position, selecting the next node by roulette wheel probability, repeating the selection process until the target position is reached to generate a feasible path, repeating the generation process N times to obtain an initial population containing N feasible paths, and calculating the fitness value of the initial population according to the fitness function.

[0009] S3. Based on the position information and fitness value of each individual in the initial population, calculate the state features of each individual to form a feature vector, and input the feature vector into the policy network of the near-end policy optimization algorithm. The policy network selects the corresponding action from the preset path planning multi-mutation policy action pool according to the input feature vector.

[0010] S4. Generate a mutation vector based on the action, perform crossover and selection operations on the mutation vector and the corresponding current individual to generate offspring individuals, and obtain a reward based on the fitness value of the offspring individuals.

[0011] S5. Store the state, action and reward of each individual in a single iteration into an independent buffer corresponding to that individual, and repeat steps S3 to S5 until the preset maximum number of iterations is reached to complete one training round.

[0012] S6. Repeat steps S2 to S5 to perform the next training round. After a preset number of training rounds have been completed, use the experience data stored in each independent buffer to update the policy network parameters and value network parameters of the near-end policy optimization algorithm in batches. After clearing all the independent buffers, return to step S2 to reinitialize the population and continue training until the preset training round limit is reached to complete the training of the path planning method.

[0013] S7. Apply the trained path planning method to the two-dimensional grid map environment to output the optimal path for the robot from the starting position to the target position.

[0014] This invention establishes a planning foundation by constructing a grid map environment and designing a fitness function. It initializes the population using a roulette wheel selection strategy, effectively preserving diverse paths and providing a high-quality starting point for subsequent optimization. Based on a proximal policy optimization algorithm, it adaptively selects multiple mutation policy actions by analyzing individual state characteristics, achieving intelligent decision-making for mutation operations and dynamically adjusting the search direction, significantly avoiding premature convergence. During the iteration process, the results of crossover and selection to generate offspring individuals are transformed into reward signals. Combined with accumulated experience data in individual independent buffers, the algorithm can continuously optimize the policy network through batch learning, enhancing the balance between global exploration capability and local fine-grained search. Through multi-round cyclic training, this method gradually improves the adaptability of path planning, ultimately rapidly generating smooth paths approaching the global optimum in complex environments, comprehensively improving the convergence efficiency and solution quality of path planning.

[0015] Preferably, the expression for the fitness function in step S1 is:

[0016]

[0017] In the formula, Indicates the first The total fitness value of an individual Indicates the first The length fitness value of each individual, Indicates the first The obstacle fitness value of each individual;

[0018] The length fitness value The formula is as follows:

[0019]

[0020] In the formula, This indicates the total number of nodes in the current path. Indicates the first in the current population The first of the individuals 1 node Indicates the first in the current population The first of the individuals One node;

[0021] The obstacle fitness value The formula is as follows:

[0022]

[0023] In the formula, Represents a constant. This indicates the total number of obstacles encountered on the current path.

[0024] This fitness function comprehensively evaluates the overall performance of a path by cleverly integrating two key metrics: path length and obstacle avoidance. Length fitness precisely quantifies the cumulative distance between path nodes, directly driving the algorithm to seek shorter movement paths. Obstacle fitness significantly penalizes obstacle crossing behavior, amplifying the cost of unsafe paths through constant weights, thus powerfully guiding the search process away from infeasible regions. The linear combination of these two metrics allows the fitness function to simultaneously consider path optimality and safety, effectively balancing the conflict between distance minimization and obstacle avoidance reliability. This provides a clear and robust optimization direction for subsequent selection, crossover, and mutation operations, ultimately ensuring that the algorithm can stably and efficiently plan short and safe feasible paths in complex environments.

[0025] Preferably, the state features described in step S3 are configured as follows:

[0026]

[0027] In the formula, Represents state characteristics, Indicates the degree of global distribution. Indicates the degree of local nearest neighbor distribution. This indicates the relative magnitude of improvement in individual fitness. This indicates that the individual's position has not been updated for multiple consecutive generations. Indicates the iteration progress;

[0028] The degree of global distribution The formula is as follows:

[0029]

[0030] In the formula, Represents an individual Relative to the degree of population aggregation, Indicates population size;

[0031] The degree of local nearest neighbor distribution The formula is as follows:

[0032]

[0033] In the formula, Represents an individual The degree of local nearest neighbor distribution, Indicates distance from individual Recent Euclidean distance between individuals;

[0034] The relative magnitude of individual fitness improvement The formula is as follows:

[0035]

[0036] In the formula, Represents an individual The relative magnitude of the improvement in fitness. Represents an individual The improvement in fitness Represents the previous generation of individuals The improvement in fitness;

[0037] The individual's location has not been updated for multiple consecutive generations. The formula is as follows:

[0038]

[0039] The stagnation constraints are as follows:

[0040]

[0041] In the formula, Represents an individual The location has not been updated for several generations. Indicates the current population's... Individual, This indicates that the first generation of the previous population Individual;

[0042] The iteration progress The formula is as follows:

[0043]

[0044] in, This indicates the current iteration number. This indicates the maximum number of iterations.

[0045] This state feature design integrates multi-dimensional information such as global and local search states, individual evolutionary vitality, and overall iteration progress to construct a comprehensive descriptive system that portrays population dynamics and individual performance. Global and local distribution characteristics enable the algorithm to perceive the dispersion and aggregation of the population in the solution space, thus intelligently balancing exploration and development. Individual fitness improvement magnitude and stagnation generation characteristics provide real-time feedback on each individual's optimization potential and evolutionary deadlock, offering precise trigger signals for activating mutations. Iteration progress characteristics introduce a time dimension to the algorithm, prompting the strategy to adaptively adjust with each search phase. These features collectively provide subsequent proximal policy optimization algorithms with refined and highly informative environmental state inputs, enabling them to more intelligently select the most suitable mutation strategy from the action pool. This effectively breaks the rigid pattern of traditional algorithms using fixed operators in fixed phases, significantly improving the adaptive capability, convergence efficiency, and global optimization success rate of the path planning process.

[0046] Preferably, the degree of aggregation of individual i relative to the population The formula is as follows:

[0047]

[0048] In the formula, Represents an individual Relative to the individual The Euclidean distance is calculated using the following formula:

[0049]

[0050] In the formula, Indicates the current population's... Individual, and Each represents an individual and individuals dimensionality;

[0051] The individual fitness improvement The formula is as follows:

[0052]

[0053] In the formula, Represents the initial individual fitness value, Indicates the current individual fitness value;

[0054] The previous generation individuals fitness improvement The formula is as follows:

[0055]

[0056] In the formula, Represents the previous generation of individuals The fitness value.

[0057] By calculating population clustering and quantifying fitness improvement metrics, the algorithm's ability to perceive population structure and individual evolutionary dynamics is effectively enhanced. Specifically, the individual clustering calculation is compatible with distance measures between individuals across different dimensions, accurately assessing the density of individual distribution in the solution space and providing a reliable basis for identifying population diversity levels. The fitness improvement formula, by comparing the current fitness with the initial or historical fitness, quantifies the relative progress of an individual during evolution, clearly reflecting its optimization potential and convergence status. These formulas collectively constitute the core of state feature calculation, enabling the algorithm to gain deep insights into the global exploration breadth and local development depth of the population. This provides a crucial data foundation for subsequently adaptively selecting mutation strategies, significantly improving the path planning algorithm's ability to avoid premature convergence, accelerate convergence, and approach the global optimum in complex environments.

[0058] Preferably, the construction of the path planning multi-variation strategy action pool in step S3 includes:

[0059] Define four mutation operations: crossover ,mutation delete ,improve ;

[0060] Wherein, the cross The formula is as follows:

[0061]

[0062] In the formula, and This indicates the individual selected based on the action strategy. and individuals , Indicates in the individual and individuals The number of randomly selected indices in the smallest dimension;

[0063] The mutation The formula is as follows:

[0064]

[0065] In the formula, Indicates in the individual Randomly selected path nodes from those not included. Indicates in the individual The number of randomly selected indices in the dimension;

[0066] The deletion The formula is as follows:

[0067]

[0068] In the formula, This indicates the individual after the deletion operation. fitness value, Indicates the current individual fitness value;

[0069] The improvements The formula is as follows:

[0070]

[0071] In the formula, express Corresponding to The optimal location within the 8-neighborhood of a location node. Indicates the individual after the improvement operation fitness value;

[0072] Based on the four mutation operations above, an action pool is designed, which contains a total of six actions, as shown in the following formula:

[0073]

[0074] In the formula, This represents the generated mutated solution. , These represent the random selection of two different individuals from the population, excluding the current individual, as parent 1 and parent 2. This represents the optimal solution in the current population. This indicates the action value selected by the proximal policy optimization algorithm based on the current state of the individual.

[0075] By employing four mutation operations and their combinations, a feature-rich and hierarchically structured multi-mutation strategy action pool for path planning is constructed. The crossover operation merges different path segments to introduce population diversity; the mutation operation randomly perturbs path nodes to enhance exploration capabilities; the deletion operation removes redundant nodes to optimize path simplicity; and the improvement operation utilizes local neighborhood information for fine-tuning to improve path quality. Combining these four basic operations with different priorities and purposes forms six progressive action options, ranging from simple perturbations to complex recombinations, and from random exploration to guided optimization. This allows the algorithm to intelligently select the most suitable mutation strategy based on the current state characteristics of each individual. This design significantly enhances the algorithm's adaptability to different evolutionary stages and population states, enabling it to escape local optima through strong exploratory actions and accelerate local convergence through guided actions. Thus, it achieves a significant improvement in global search efficiency and final path quality in complex path planning problems.

[0076] Preferably, step S4, which involves generating a mutation vector based on the action and performing crossover and selection operations between the mutation vector and the corresponding current individual to generate offspring individuals, includes:

[0077] Mutant individuals are generated by actions selected by the policy network. , with the current individual Perform crossover operations to generate new individuals. The formula is as follows:

[0078]

[0079] In the formula, This represents a new individual generated through crossover;

[0080] The selection process generates new offspring, as shown in the following formula:

[0081]

[0082] In the formula, Individuals representing the next generation , This represents the fitness value of the individuals generated by crossover.

[0083] By combining policy network-guided mutation with classical genetic operations, an intelligent and controllable population evolution mechanism is achieved. First, based on the actions selected by the policy network, mutated individuals with specific exploration or exploitation tendencies are generated. These mutated individuals are then crossbred with current individuals, thereby introducing diversity or optimization-oriented perturbations while inheriting effective information from the original paths. Subsequently, a rigorous selection process retains individuals with better fitness for the next generation, ensuring that the population's evolutionary direction always leans towards shorter and safer solutions. This process deeply integrates the intelligent decision-making capabilities of reinforcement learning with the robust search framework of genetic algorithms. This transforms each generation's evolution from blind random attempts into an adaptive optimization process based on environmental state awareness, significantly improving the targeting and convergence efficiency of path search, effectively avoiding ineffective iterations, and ultimately leading to better global path planning results.

[0084] Preferably, the formula for obtaining the reward based on the fitness value of the offspring individual in step S4 is as follows:

[0085]

[0086]

[0087] in, Represents offspring individuals fitness value, For offspring individuals The ratio of the fitness value to the initial fitness estimate. As a reward.

[0088] This reward mechanism constructs an intuitive and effective evolutionary feedback signal by comparing the fitness ratio of offspring to the initial individual. Using a negative ratio as the reward allows the algorithm to clearly perceive the positive evolutionary direction of decreasing fitness values, thus providing a clear and stable optimization gradient for policy network updates. This design transforms the long-term goal of path planning—finding shorter and safer paths—into immediate rewards and penalties at each iteration, strengthening the algorithm's learning and selection of efficient mutation strategies. It effectively guides the agent to make decisions more conducive to global convergence between exploration and exploitation, ultimately promoting the continuous and stable improvement of the overall performance of the path planning method.

[0089] Preferably, the experience storage mechanism for the near-end policy optimization algorithm in step S5 is improved as follows:

[0090] Create and maintain an independent experience buffer for each individual in the population;

[0091] The storage structure of each of the experience buffers includes:

[0092] A state sequence is used to store the state feature vector extracted by the individual in each iteration;

[0093] Action sequence, used to store the action selected by the individual based on the policy network in each iteration, wherein the action is the number of the mutation policy in the path planning multi-mutation policy action pool;

[0094] The reward sequence stores the instantaneous reward value calculated for this individual in each iteration based on the improvement in fitness values ​​of its offspring individuals;

[0095] The termination flag sequence is used to record whether the individual has reached the preset iteration termination condition in each iteration;

[0096] During the iteration process, all individuals in the population synchronously perform state feature extraction, action selection, mutation and crossover operations, reward calculation, and experience buffer update operations, and the data storage and updates between each experience buffer are independent of each other.

[0097] This improved experience storage mechanism constructs a highly parallelized and data-isolated learning environment by establishing an independent experience buffer for each individual in the population. Each buffer structurally records the complete evolutionary trajectory of the corresponding individual, including state, action, reward, and termination flag, ensuring the integrity and temporal order of the experience data. This design enables the algorithm to capture rich and differentiated learning experiences within the population, avoiding policy confusion caused by experience mixing, thus providing a large number of high-quality and diverse training samples for subsequent batch updates of the policy network. Independent buffers support synchronous exploration and learning for all individuals, greatly improving data collection efficiency and the algorithm's parallel computing potential, strengthening the agent's adaptability to different evolutionary states and path forms, and ultimately significantly enhancing the learning stability, convergence speed, and optimization performance of the entire path planning method in complex environments.

[0098] Preferably, the training process of the near-end policy optimization algorithm described in step S6 includes:

[0099] A single optimization process, from initializing the population to reaching the maximum number of iterations, is defined as a training round of the improved differential evolution algorithm. In each training round, use the old parameters of the current policy network. Interact with the path planning environment to obtain stable empirical data sequences;

[0100] After a total of M training rounds are completed, all the experience data collected in the experience buffer are integrated as batch training samples to update the parameters of the policy network and the value network.

[0101] By optimizing the total loss function of the near-end strategy optimization algorithm To update the network parameters, the total loss function Defined as:

[0102]

[0103] In the formula, Represents the loss of the pruning strategy. Represents the loss of the value function. Represents the strategy entropy reward. and These are hyperparameters used to balance the weights of different loss functions;

[0104] The loss of the pruning strategy The calculation method is as follows:

[0105]

[0106] In the formula, Indicates the magnitude of the policy update. Represents the dominance function. As a hyperparameter to control the cropping amplitude;

[0107] The strategy update range The calculation method is as follows:

[0108]

[0109] In the formula, Indicates the magnitude of the policy update. and These represent the new strategy and the old strategy in state, respectively. Take action below The probability of;

[0110] The advantage function The calculation method is as follows:

[0111]

[0112] In the formula, Indicates the state Take action below value, Indicates the state The average value below;

[0113] The value function loss The calculation method is as follows:

[0114]

[0115] In the formula, Representing state The value function, Represents the target value function;

[0116] Calculate the total loss function Then, the parameters of the policy network and value network are updated through backpropagation; the process of accumulating experience and updating network parameters in batches is repeated until the upper limit of training rounds is reached, and the optimal policy network parameters after training are saved.

[0117] This training process cleverly combines the iterative search of evolutionary algorithms with the policy optimization of reinforcement learning by defining the complete differential evolution optimization process as training rounds, ensuring the consistency and stability of training data. The batch update mechanism, after accumulating experience across multiple rounds, performs concentrated learning, significantly improving data utilization efficiency and training stability. By constructing a composite loss function including pruning policy loss, value function loss, and policy entropy reward, this process achieves fine-grained control over the magnitude of policy updates, effectively avoiding training collapse. Simultaneously, it accurately evaluates the value of actions using the advantage function, guiding the policy to evolve towards improving long-term returns. The introduction of the entropy reward term encourages moderate exploration, preventing the policy from prematurely falling into local optima. This systematic training framework ensures that the policy network can learn robustly from massive path planning interaction experience, gradually improving its ability to adaptively select the optimal mutation policy, ultimately enabling the algorithm to exhibit excellent global convergence performance and robustness in complex environments.

[0118] Preferably, step S7 includes:

[0119] The optimal policy network parameters saved after training are loaded into the near-end policy optimization algorithm;

[0120] In the two-dimensional grid map environment, state features are extracted for each individual in each generation of the population to obtain the current state feature vector;

[0121] The current state feature vector is input into a policy network loaded with the optimal policy network parameters, and the policy network outputs the probability distribution of all selectable actions in the path planning multi-variation policy action pool.

[0122] Based on the probability distribution, actions are randomly selected, and corresponding mutation operations are performed to generate new population individuals;

[0123] The process of extracting state features, selecting actions, and updating the population is repeated until the maximum number of iterations is reached.

[0124] The optimal path obtained at this point is output as the final planned path for the robot from the starting position to the target position.

[0125] By deploying a pre-trained optimal policy network, the intelligent decision-making capabilities acquired through offline learning are efficiently applied to practical path planning tasks. In each iteration, the algorithm dynamically outputs the optimal probabilities of each mutation action based on real-time extracted population state features, guiding the generation of new individuals accordingly. This achieves a search process entirely driven by adaptive intelligent policies. This mechanism ensures that the algorithm can fully utilize learned experience during the planning phase, flexibly and accurately invoking the most effective mutation policy combinations for different environmental states and population distributions. This allows for the rapid and stable generation of high-quality paths without additional training overhead. Ultimately, this method significantly improves the real-time performance, reliability, and global optimality of the robot's online path planning, achieving a closed loop from learning to application.

[0126] Beneficial Effects: Compared with existing technologies, this invention has the following significant advantages: 1. By integrating the mutation strategy of the improved differential evolution algorithm with the intelligent decision-making mechanism of deep reinforcement learning, this invention enables the robot to dynamically evaluate the population state, individual stagnation status, and iteration progress during the path planning process. This allows it to adaptively select the most suitable optimization action from diverse combinations of mutation operations, significantly reducing the risk of traditional methods getting trapped in local optima in complex environments and ensuring the global optimality of the final planned path. 2. Through continuous learning and policy optimization of the path optimization process using a proximal policy optimization algorithm, the robot can quickly identify effective path improvement directions based on historical experience, reducing invalid searches and redundant iterations. This significantly improves the convergence speed and overall computational efficiency of path planning while ensuring path quality. 3. By designing an observation mechanism that integrates multi-dimensional state characteristics such as global distribution, local neighbors, individual improvement magnitude, and stagnation time, and combining it with a deep reinforcement learning policy network for real-time decision-making, the robot can autonomously adjust its optimization strategy according to environmental characteristics and search status at different stages, improving the adaptability and robustness of path planning in complex, dynamic, or unknown environments. Attached Figure Description

[0127] Figure 1 This is a schematic diagram of the method flow of the present invention;

[0128] Figure 2 This is a 20*20 2D grid map of the present invention;

[0129] Figure 3 This is a path comparison diagram of the robot path planning method and the differential evolution algorithm of the present invention;

[0130] Figure 4 This is a convergence curve of the fitness value of the robot path planning method and differential evolution algorithm of the present invention. Detailed Implementation

[0131] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0132] In this application, the present invention proposes a robot path planning method based on an improved differential evolution algorithm and deep reinforcement learning, the process of which is as follows: Figure 1 As shown. The steps of the robot path planning method based on the improved differential evolution algorithm and deep reinforcement learning are as follows:

[0133] Step 1: Construct a 2D grid map environment based on the robot's search requirements, mark impassable areas on the map as obstacles, and determine the starting and target positions of the mobile robot, such as... Figure 2 As shown, a fitness function is established based on the map environment, where the fitness function is expressed by the following formula:

[0134]

[0135] In the formula, Indicates the first The total fitness value of an individual Indicates the first The length fitness value of each individual, Indicates the first The obstacle fitness value of each individual.

[0136] Length fitness value The formula is as follows:

[0137]

[0138] In the formula, This indicates the total number of nodes in the current path. Indicates the first in the current population The first of the individuals 1 node Indicates the first in the current population The first of the individuals Each node.

[0139] Obstacle fitness value The formula is as follows:

[0140]

[0141] In the formula, Represents a constant. This indicates the total number of obstacles encountered on the current path.

[0142] Step 2: Initialize the population using a roulette wheel selection strategy. Starting from the starting point, select feasible nodes from the 8 neighbors of the current node, calculate the Euclidean distance from each feasible point to the target location, and select the next node using roulette wheel probability. Repeat this process until the target location is reached. Repeat the above process N times to obtain an initial population containing N feasible paths. Calculate the fitness value of the initial population based on the established fitness function.

[0143] Step 3: Based on the population location and fitness information, calculate the state features of each individual in the population in turn, construct a feature vector, and then input it into the near-end policy optimization algorithm. The state features are composed as follows:

[0144]

[0145] In the formula, Represents state characteristics, Indicates the degree of global distribution. Indicates the degree of local nearest neighbor distribution. This indicates the relative magnitude of improvement in individual fitness. This indicates that the individual's position has not been updated for multiple consecutive generations. Indicates the iteration progress.

[0146] Global distribution level The formula is as follows:

[0147]

[0148] In the formula, Represents an individual Relative to the degree of population aggregation, Indicates population size.

[0149] individual Relative to the degree of population aggregation The formula is as follows:

[0150]

[0151] In the formula, Represents an individual Relative to the individual The Euclidean distance.

[0152] individual Relative to the individual Euclidean distance The formula is as follows:

[0153]

[0154] In the formula, Indicates the current population's... Individual, and Each represents an individual and individuals The dimension of.

[0155] Local nearest neighbor distribution degree The formula is as follows:

[0156]

[0157] In the formula, Represents an individual The degree of local nearest neighbor distribution, Indicates distance from individual Recent Euclidean distance between individuals.

[0158] Relative magnitude of individual fitness improvement The formula is as follows:

[0159]

[0160] In the formula, Represents an individual The relative magnitude of the improvement in fitness. Represents an individual The improvement in fitness Represents the previous generation of individuals The improvement in fitness.

[0161] individual fitness improvement The formula is as follows:

[0162]

[0163] In the formula, Represents the initial individual fitness value, Indicates the current individual The fitness value.

[0164] Previous generation individuals fitness improvement The formula is as follows:

[0165]

[0166] In the formula, Represents the previous generation of individuals The fitness value.

[0167] Individual location has not been updated for multiple generations The formula is as follows:

[0168]

[0169] The stagnation constraints are as follows:

[0170]

[0171] In the formula, Represents an individual The location has not been updated for several generations. Indicates the current population's... Individual, This indicates that the first generation of the previous population Individual.

[0172] Iteration progress The formula is as follows:

[0173]

[0174] in, This indicates the current iteration number. This indicates the maximum number of iterations.

[0175] Then, a path planning multi-variation strategy is designed as an action pool, and the action in this state is obtained based on the policy network. The action pool is constructed as follows:

[0176] First, we define four mutation operations: crossover. ,mutation delete e. Improvement .

[0177] Among them, cross operation The formula is as follows:

[0178]

[0179] In the formula, and This indicates the individual selected based on the action strategy. and individuals , Indicates in the individual and individuals The number of randomly selected indices from the smallest dimension.

[0180] Mutation operation The formula is as follows:

[0181]

[0182] In the formula, Indicates in the individual Randomly selected path nodes from those not included. Indicates in the individual The number of randomly selected indices in the dimension.

[0183] Deletion operation The formula is as follows:

[0184]

[0185] In the formula, This indicates the individual after the deletion operation. fitness value, Indicates the current individual The fitness value.

[0186] Improved operation The formula is as follows:

[0187]

[0188] In the formula, express Corresponding to The optimal location within the 8-neighborhood of a location node. Indicates the individual after the improvement operation The fitness value.

[0189] Based on the four mutation operations above, an action pool is designed, which contains a total of six actions, as shown in the following formula:

[0190]

[0191] In the formula, This represents the generated mutated solution. , These represent the random selection of two different individuals from the population, excluding the current individual, as parent 1 and parent 2. This represents the optimal solution in the current population. This indicates the action value selected by the proximal policy optimization algorithm based on the current state of the individual.

[0192] Step 4: Generate a mutation vector based on the action, cross-reference it with the current individual, select the offspring, and execute the process as follows:

[0193] Mutant individuals are generated by actions selected by the policy network. , with the current individual Perform crossover operations to generate new individuals. The formula is as follows:

[0194]

[0195] In the formula, This represents a new individual generated through crossover.

[0196] The selection process generates new offspring, as shown in the following formula:

[0197]

[0198] In the formula, Individuals representing the next generation , This represents the fitness value of the individuals generated by crossover.

[0199] Then, a reward is given based on the offspring's fitness value, with the reward formula as follows:

[0200]

[0201]

[0202] In the formula, Represents offspring individuals fitness value, For offspring individuals The ratio of the fitness value to the initial fitness estimate. As a reward.

[0203] Step 5: Store the state, action, and reward of each individual in a single iteration into an individual's independent buffer. The improvement to the experience storage mechanism of the near-end policy optimization algorithm is as follows:

[0204] Each individual's experience buffer storage structure includes:

[0205] State sequence: Stores the search feature vector extracted by an individual in each iteration;

[0206] Action sequence: Stores the mutation policy number selected based on the policy network;

[0207] Reward sequence: Stores the immediate reward value calculated based on the fitness improvement of offspring;

[0208] Termination flag: Records whether the termination condition has been met in each iteration;

[0209] The near-end strategy optimization algorithm allocates an independent buffer for each individual in the population to achieve isolated storage of individual experience data;

[0210] All individuals in the population synchronously perform state extraction, action selection, mutation and crossover, reward calculation, and buffer update operations, with data updates in each buffer independent of each other. Then, return to step 3 until the maximum number of iterations is reached, completing one training round.

[0211] Step 6: Return to Step 2 and begin the next training round. This round will consist of a complete single optimization process of the improved differential evolution algorithm, from initializing the population until reaching the maximum number of iterations. In this round, the old policy network parameters are used. Interact with the path planning environment to obtain stable empirical data sequences.

[0212] When the cumulative completion After a training epoch, the empirical data collected from all individual buffers are integrated as training samples for batch updating the policy network and value network. The network parameters are updated by optimizing the total loss function of the proximal policy optimization algorithm, which includes pruning policy loss, value function loss, and policy entropy reward.

[0213] Total loss function of near-end policy optimization algorithm The definition is as follows:

[0214]

[0215] In the formula, Represents the loss of the pruning strategy. Represents the loss of the value function. Represents the policy entropy reward. The greater the policy entropy, the more uniform the probability distribution of the actions taken by the policy in each state, and the stronger the policy's exploratory nature. and These are hyperparameters used to balance the weights of different loss functions.

[0216] Pruning strategy loss The definition is as follows:

[0217]

[0218] In the formula, Indicates the magnitude of the policy update. Represents the dominance function. Used as a hyperparameter to control the clipping amplitude.

[0219] Strategy update range The formula is as follows:

[0220]

[0221] In the formula, Indicates the magnitude of the policy update. and These represent the new strategy and the old strategy in state, respectively. Take action below The probability of.

[0222] Advantage function The calculation is as follows:

[0223]

[0224] In the formula, Indicates the state Take action below value, Indicates the state The average value is calculated based on the advantage function. This advantage function guides the policy generation network to continuously adjust its policy, improving the algorithm's reward feedback.

[0225] Value function loss The formula is as follows:

[0226]

[0227] In the formula, Representing state The value function, This represents the objective value function.

[0228] Calculate the total loss function The data is then passed back in reverse to update the algorithm's strategy. The algorithm iterates through the value function until the convergence condition is met. This interaction-collection-update loop is repeated until the preset training epoch limit is reached. After training, the optimal policy network parameters are saved.

[0229] Step 7: Apply the proposed path planning method to the map environment to obtain the optimal path. Feature extraction is performed on each generation of individuals using the optimal policy network parameters. The proximal policy optimization algorithm randomly selects actions to update based on the action probability distribution in that state until the maximum number of iterations is reached, outputting the optimal path. A path comparison diagram between the robot path planning method based on the improved differential evolution algorithm and deep reinforcement learning and the differential evolution algorithm is shown below. Figure 3 As shown, the convergence curves of the fitness values ​​of the robot path planning method based on the improved differential evolution algorithm and deep reinforcement learning, and the differential evolution algorithm are as follows: Figure 4 As shown.

[0230] The effectiveness of this embodiment can be further illustrated through simulation experiments:

[0231] Set the parameter, namely the maximum number of iterations. The population size is 500. The training count is 50.

[0232] For 100 times, the cumulative number of rounds The learning rate of the value network is 10. The learning rate of the policy network is 0.001. The value is 0.0003, the discount factor is 0.99, the number of rounds of parameter updates after data collection is 80, and the trimming parameters are... It is 0.2. , .

[0233] The improved differential evolution algorithm was compared with the robot path planning method and differential evolution algorithm based on deep reinforcement learning. The algorithm was executed a total of 10 times. The experimental results were compared with the maximum optimal value, minimum optimal value, average optimal value and standard deviation. The results are shown in Table 1.

[0234] Table 1 Performance comparison between the improved algorithm and the differential evolution algorithm

[0235] algorithm Maximum value Minimum value average value PPO_DE algorithm 33.17 30.81 32.04 DE algorithm 34.61 32.75 33.63

[0236] It can be proven that, based on the maximum optimal value, minimum optimal value, and average optimal value, the improved differential evolution algorithm combined with deep reinforcement learning for robot path planning has a greater advantage in terms of results. Overall, the improved differential evolution algorithm combined with deep reinforcement learning for robot path planning performs better, and its results are generally superior to the original algorithm.

Claims

1. A robot path planning method based on an improved differential evolution algorithm and deep reinforcement learning, characterized in that, Includes the following steps: S1. Construct a two-dimensional grid map environment based on the path finding requirements of the mobile robot, mark the location of obstacles in the impassable areas of the two-dimensional grid map environment, determine the starting position and target position of the mobile robot, and establish a fitness function based on the two-dimensional grid map environment. S2. Initialize the population using a roulette wheel selection strategy, including: starting from the starting position, selecting feasible nodes in the 8-neighborhood of the current node, calculating the Euclidean distance from each feasible node to the target position, selecting the next node by roulette wheel probability, repeating the selection process until the target position is reached to generate a feasible path, repeating the generation process N times to obtain an initial population containing N feasible paths, and calculating the fitness value of the initial population according to the fitness function. S3. Based on the position information and fitness value of each individual in the initial population, calculate the state features of each individual to form a feature vector, and input the feature vector into the policy network of the near-end policy optimization algorithm. The policy network selects the corresponding action from the preset path planning multi-mutation policy action pool according to the input feature vector. S4. Generate a mutation vector based on the action, perform crossover and selection operations on the mutation vector and the corresponding current individual to generate offspring individuals, and obtain a reward based on the fitness value of the offspring individuals. S5. Store the state, action and reward of each individual in a single iteration into an independent buffer corresponding to that individual, and repeat steps S3 to S5 until the preset maximum number of iterations is reached to complete one training round. S6. Repeat steps S2 to S5 to perform the next training round. After a preset number of training rounds have been completed, use the experience data stored in each independent buffer to update the policy network parameters and value network parameters of the near-end policy optimization algorithm in batches. After clearing all the independent buffers, return to step S2 to reinitialize the population and continue training until the preset training round limit is reached to complete the training of the path planning method. S7. Apply the trained path planning method to the two-dimensional grid map environment to output the optimal path for the robot from the starting position to the target position; In step S5, the empirical storage mechanism of the near-end policy optimization algorithm is improved as follows: Create and maintain an independent experience buffer for each individual in the population; The storage structure of each of the experience buffers includes: A state sequence is used to store the state feature vector extracted by the individual in each iteration; Action sequence, used to store the action selected by the individual based on the policy network in each iteration, wherein the action is the number of the mutation policy in the path planning multi-mutation policy action pool; The reward sequence stores the instantaneous reward value calculated for this individual in each iteration based on the improvement in fitness values ​​of its offspring individuals; The termination flag sequence is used to record whether the individual has reached the preset iteration termination condition in each iteration; During the iteration process, all individuals in the population synchronously perform state feature extraction, action selection, mutation and crossover operations, reward calculation, and experience buffer update operations, and the data storage and updates between each experience buffer are independent of each other.

2. The method according to claim 1, characterized in that, The expression for the fitness function mentioned in step S1 is: In the formula, Indicates the first The total fitness value of an individual Indicates the first The length fitness value of each individual, Indicates the first The obstacle fitness value of each individual; The length fitness value The formula is as follows: In the formula, This indicates the total number of nodes in the current path. Indicates the first in the current population The first of the individuals 1 node Indicates the first in the current population The first of the individuals One node; The obstacle fitness value The formula is as follows: In the formula, Represents a constant. This indicates the total number of obstacles encountered on the current path.

3. The method according to claim 1, characterized in that, The state features described in step S3 are constituted as follows: In the formula, Representing state characteristics, Indicates the degree of global distribution. Indicates the degree of local nearest neighbor distribution. This indicates the relative magnitude of improvement in individual fitness. This indicates that the individual's position has not been updated for multiple consecutive generations. Indicates the iteration progress; The degree of global distribution The formula is as follows: In the formula, Represents an individual Relative to the degree of population aggregation, Indicates population size; The degree of local nearest neighbor distribution The formula is as follows: In the formula, Represents an individual The degree of local nearest neighbor distribution, Indicates distance from individual Recent Euclidean distance between individuals; The relative magnitude of individual fitness improvement The formula is as follows: In the formula, Represents an individual The relative magnitude of the improvement in fitness. Represents an individual The improvement in fitness Represents the previous generation of individuals The improvement in fitness; The individual's location has not been updated for multiple consecutive generations. The formula is as follows: The stagnation constraints are as follows: In the formula, Represents an individual The location has not been updated for several generations. Indicates the current population's... Individual, This indicates that the first generation of the previous population Individual; The iteration progress The formula is as follows: ;in, Indicates the current iteration number. This indicates the maximum number of iterations.

4. The method according to claim 3, characterized in that, The degree of aggregation of individual i relative to the population The formula is as follows: In the formula, Represents an individual Relative to the individual The Euclidean distance is calculated using the following formula: In the formula, Indicates the current population's... Individual, and Each represents an individual and individuals dimensionality; The individual fitness improvement The formula is as follows: In the formula, Represents the initial individual fitness value, Indicates the current individual fitness value; The previous generation individuals fitness improvement The formula is as follows: In the formula, Represents the previous generation of individuals The fitness value.

5. The method according to claim 1, characterized in that, The construction of the path planning multi-variation strategy action pool in step S3 includes: Define four mutation operations: crossover ,mutation delete e. Improvement ; Wherein, the cross The formula is as follows: In the formula, and This indicates the individual selected based on the action strategy. and individuals , Indicates in the individual and individuals The number of randomly selected indices in the smallest dimension; The mutation The formula is as follows: In the formula, Indicates in the individual Randomly selected path nodes from those not included. Indicates in the individual The number of randomly selected indices in the dimension; The deletion The formula is as follows: In the formula, This indicates the individual after the deletion operation. fitness value, Indicates the current individual fitness value; The improvements The formula is as follows: In the formula, express Corresponding to The optimal location within the 8-neighborhood of a location node. Indicates the individual after the improvement operation fitness value; Based on the four mutation operations above, an action pool is designed, which contains a total of six actions, as shown in the following formula: In the formula, This represents the generated mutated solution. , These represent the random selection of two different individuals from the population, excluding the current individual, as parent 1 and parent 2. This represents the optimal solution in the current population. This indicates the action value selected by the proximal policy optimization algorithm based on the current state of the individual.

6. The method according to claim 1, characterized in that, Step S4, which involves generating a mutation vector based on the action and performing crossover and selection operations between the mutation vector and the corresponding current individual to generate offspring individuals, includes: Mutant individuals are generated by actions selected by the policy network. , with the current individual Perform crossover operations to generate new individuals. The formula is as follows: In the formula, This represents a new individual generated through crossover; The selection process generates new offspring, as shown in the following formula: In the formula, Individuals representing the next generation , This represents the fitness value of the individuals generated by crossover.

7. The method according to claim 1, characterized in that, The formula for obtaining the reward based on the fitness value of the offspring individual in step S4 is as follows: ; ;in, Represents offspring individuals fitness value, For offspring individuals The ratio of the fitness value to the initial fitness estimate. As a reward.

8. The method according to claim 1, characterized in that, The training process of the near-end policy optimization algorithm described in step S6 includes: A single optimization process, from initializing the population to reaching the maximum number of iterations, is defined as a training round of the improved differential evolution algorithm. In each training round, use the old parameters of the current policy network. Interact with the path planning environment to obtain stable empirical data sequences; After a total of M training rounds are completed, all the experience data collected in the experience buffer are integrated as batch training samples to update the parameters of the policy network and the value network. By optimizing the total loss function of the near-end strategy optimization algorithm To update the network parameters, the total loss function Defined as: In the formula, Represents the loss of the pruning strategy. Represents the loss of the value function. Represents the policy entropy reward. and These are hyperparameters used to balance the weights of different loss functions; The loss of the pruning strategy The calculation method is as follows: In the formula, Indicates the magnitude of the policy update. Represents the dominance function. As a hyperparameter to control the cropping amplitude; The strategy update range The calculation method is as follows: In the formula, Indicates the magnitude of the policy update. and These represent the new strategy and the old strategy in state, respectively. Take action below The probability of; The advantage function The calculation method is as follows: In the formula, Indicates the state Take action below value, Indicates the state The average value below; The value function loss The calculation method is as follows: In the formula, Representing state The value function, Represents the target value function; Calculate the total loss function Then, the parameters of the policy network and value network are updated through backpropagation; the process of accumulating experience and updating network parameters in batches is repeated until the upper limit of training rounds is reached, and the optimal policy network parameters after training are saved.

9. The method according to claim 1, characterized in that, Step S7 includes: The optimal policy network parameters saved after training are loaded into the near-end policy optimization algorithm; In the two-dimensional grid map environment, state features are extracted for each individual in each generation of the population to obtain the current state feature vector; The current state feature vector is input into a policy network loaded with the optimal policy network parameters, and the policy network outputs the probability distribution of all selectable actions in the path planning multi-variation policy action pool. Based on the probability distribution, actions are randomly selected, and corresponding mutation operations are performed to generate new population individuals; The process of extracting state features, selecting actions, and updating the population is repeated until the maximum number of iterations is reached. The optimal path obtained at this point is output as the final planned path for the robot from the starting position to the target position.

Images