Robot path planning method based on improved differential evolution algorithm and deep reinforcement learning

Abstract

The invention discloses a robot path planning method based on an improved differential evolution algorithm and deep reinforcement learning. The robot path planning method comprises the following steps: establishing a path planning map containing irregular obstacles; initializing a plurality of feasible solutions of the path planning problem as a population, and designing a multi-strategy improved differential evolution algorithm; a near-end strategy optimization algorithm in deep reinforcement learning is utilized to open up an independent buffer area for each individual in the population, a state pool is designed based on population position and fitness information, multiple strategies are regarded as an action pool, the population is guided to select proper strategies in different states, and finally an optimal path is output; according to the method, the problem that the differential evolution algorithm is easy to fall into premature convergence and local optimum when being used for robot path planning is effectively solved, and the quality of a path planning solution of the robot in a complex environment is improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of robot path planning, and particularly relates to a robot path planning method based on an improved differential evolution algorithm and deep reinforcement learning. BACKGROUND

[0002] With the rapid development of artificial intelligence and autonomous system technology, mobile robots are increasingly widely used in industrial automation, intelligent logistics, emergency rescue and complex environment exploration. Robot path planning, as the core link of realizing autonomous navigation and task execution of robots, has become a research hotspot and key challenge in the field of robot technology. An efficient and robust path planning method can not only improve the work efficiency and safety of robots, but also has important practical significance for promoting the intelligent upgrading of related industries.

[0003] At present, various traditional algorithms and intelligent methods have emerged in the field of path planning. Among them, the methods based on graph search or sampling represented by A, D and RRT* show good planning effect in structured or low-dimensional environments. At the same time, swarm intelligence optimization algorithms represented by differential evolution algorithm, genetic algorithm and particle swarm algorithm simulate natural evolution or group behavior mechanism, and show strong global search ability and adaptability in multi-constraint and nonlinear path planning problems, which have become one of the important implementation means of path planning in complex scenarios.

[0004] However, the existing path planning methods based on swarm intelligence such as differential evolution still have obvious deficiencies. Such methods are prone to fall into local optimum in the iteration process, especially in the planning scene with dense obstacles and complex terrain. The early convergence caused by the decline of population diversity makes it difficult to obtain the global optimal path. In addition, the algorithm usually fixes or randomly selects the evolution strategy, lacks adaptive judgment of the search state, and thus limits the convergence speed and the quality of the final solution, affecting the planning effect and real-time performance of robots in dynamic or large-scale environments. SUMMARY

[0005] The purpose of the present application is to provide a robot path planning method based on an improved differential evolution algorithm and deep reinforcement learning, which solves the problem of existing path planning methods falling into local optimum and slow convergence, thereby improving the global optimality and adaptive efficiency of robot path planning in complex environments.

[0006] Technical scheme: The robot path planning method provided by the present application comprises the following steps:

[0007] S1, constructing a two-dimensional grid map environment according to the path finding requirement of the mobile robot, marking the obstacle position of the non-passable area in the two-dimensional grid map environment, determining the starting position and target position of the mobile robot, and establishing a fitness function based on the two-dimensional grid map environment;

[0008] S2, initializing the population by using a roulette wheel selection strategy, including: starting from the starting position, screening feasible nodes in the 8-neighborhood of the current node, calculating the Euclidean distance of each feasible node to the target position, selecting the next node by roulette wheel probability, repeating the selection process until reaching the target position to generate a feasible path, repeating the generation process N times to obtain an initial population containing N feasible paths, and calculating the fitness value of the initial population according to the fitness function;

[0009] S3, based on the position information and fitness value of each individual in the initial population, calculating the state feature of each individual to form a feature vector, inputting the feature vector into the policy network of the proximal policy optimization algorithm, and the policy network selects the corresponding action from the preset path planning multi-mutation strategy action pool according to the input feature vector;

[0010] S4, generating a mutation vector according to the action, and performing crossover and selection operations on the mutation vector and the corresponding current individual to generate offspring individuals, and obtaining a reward based on the fitness value of the offspring individuals;

[0011] S5, storing the state, action and reward of each individual in a single iteration process in an independent buffer area corresponding to the individual, and repeating steps S3 to S5 until a preset maximum iteration number is reached, completing a training round;

[0012] S6, repeating steps S2 to S5 to perform the next training round, when a preset number of training rounds are completed, updating the policy network parameters and value network parameters of the proximal policy optimization algorithm using the experience data stored in each independent buffer area, emptying all the independent buffer areas, returning to step S2 to reinitialize the population, and continuing training until a preset upper limit of training rounds is reached, completing the training of the path planning method;

[0013] S7, applying the trained path planning method to the two-dimensional grid map environment to output the optimal path of the robot from the starting position to the target position.

[0014] This invention establishes a planning foundation by constructing a grid map environment and designing a fitness function. It initializes the population using a roulette wheel selection strategy, effectively preserving diverse paths and providing a high-quality starting point for subsequent optimization. Based on a proximal policy optimization algorithm, it adaptively selects multiple mutation policy actions by analyzing individual state characteristics, achieving intelligent decision-making for mutation operations and dynamically adjusting the search direction, significantly avoiding premature convergence. During the iteration process, the results of crossover and selection to generate offspring individuals are transformed into reward signals. Combined with accumulated experience data in individual independent buffers, the algorithm can continuously optimize the policy network through batch learning, enhancing the balance between global exploration capability and local fine-grained search. Through multi-round cyclic training, this method gradually improves the adaptability of path planning, ultimately rapidly generating smooth paths approaching the global optimum in complex environments, comprehensively improving the convergence efficiency and solution quality of path planning.

[0015] Preferably, the expression for the fitness function in step S1 is:

[0016]

[0017] In the formula, Indicates the first The total fitness value of an individual Indicates the first The length fitness value of each individual, Indicates the first The obstacle fitness value of each individual;

[0018] The length fitness value The formula is as follows:

[0019]

[0020] In the formula, This indicates the total number of nodes in the current path. Indicates the first in the current population The first of the individuals 1 node Indicates the first in the current population The first of the individuals One node;

[0021] The obstacle fitness value The formula is as follows:

[0022]

[0023] In the formula, Represents a constant. This indicates the total number of obstacles encountered on the current path.

[0024] The fitness function comprehensively evaluates the comprehensive performance of the path by skillfully integrating the path length and obstacle avoidance, two key indicators. The length fitness accurately quantifies the cumulative distance of the path nodes, directly driving the algorithm to seek a shorter distance movement scheme. The obstacle fitness significantly penalizes the behavior of crossing obstacles, amplifying the cost of unsafe paths through a constant weight, thereby strongly guiding the search process away from infeasible regions. The linear combination of the two indicators enables the fitness function to balance the optimality and safety of the path, effectively balancing the conflict between distance minimization and obstacle avoidance reliability, providing a clear and robust optimization direction for subsequent selection, crossover, and mutation operations, ultimately ensuring that the algorithm can stably and efficiently plan a feasible path that is both short and safe in complex environments.

[0025] Preferably, the state characteristics in step S3 are as follows:

[0026]

[0027] In the formula, represents the state characteristics, represents the global distribution degree, represents the local neighborhood distribution degree, represents the individual fitness improvement relative amplitude, represents the individual position continuous multiple generations without updating, represents the iteration progress;

[0028] The global distribution degree The formula is as follows:

[0029]

[0030] In the formula, represents the individual relative to the aggregation degree of the population, represents the population size;

[0031] The local neighborhood distribution degree The formula is as follows:

[0032]

[0033] In the formula, represents the local neighborhood distribution degree of the individual , represents the Euclidean distance between the individual and the nearest individuals;

[0034] The individual fitness improvement relative amplitude The formula is as follows:

[0035]

[0036] wherein, denotes the fitness improvement relative amplitude of the individual , denotes the fitness improvement situation of the individual , denotes the fitness improvement situation of the individual of the previous generation;

[0037] the individual position is not updated for consecutive generations The formula is as follows:

[0038]

[0039] The stagnation constraint is as follows:

[0040]

[0041] wherein, denotes the individual position not updated for consecutive generations, denotes the i-th individual of the current population, denotes the i-th individual of the previous generation population. The iteration progress The formula is as follows:

[0042] The iteration progress The formula is as follows:

[0043]

[0044] wherein, denotes the number of the current iteration, denotes the maximum number of iterations.

[0045] The state feature design constructs a comprehensive description system of population dynamics and individual performance by fusing multi-dimensional information such as global and local search state, individual evolution vitality, and overall iteration progress. The global and local distribution degree features enable the algorithm to perceive the dispersion and aggregation of the population in the solution space, thereby intelligently balancing exploration and development. The individual fitness improvement amplitude and stagnation number features provide real-time feedback on the optimization potential and evolutionary stagnation of each individual, providing precise trigger signals for activating mutation. The iteration progress feature introduces time dimension cognition for the algorithm, prompting the strategy to adaptively adjust with the search stage. These features collectively provide fine and high-information environmental state inputs for subsequent proximal strategy optimization algorithms, enabling them to more intelligently select the most suitable mutation strategy from the action pool, effectively breaking the rigid mode of traditional algorithms using fixed operators in fixed stages, and significantly improving the adaptive ability, convergence efficiency, and global optimization success rate of the path planning process.

[0046] ​Preferably, the degree of aggregation of the individual i with respect to the population The formula is as follows:

[0047]

[0048] wherein, denotes the individual with respect to the individual with respect to the individual

[0049]

[0050] wherein, denotes the individual of the current population, and denote the dimension of the individual and the individual respectively;

[0051] the individual the degree of improvement of the fitness The formula is as follows:

[0052]

[0053] wherein, denotes the fitness value of the initial individual denotes the fitness value of the current individual

[0054] the individual the degree of improvement of the fitness The formula is as follows:

[0055]

[0056] wherein, denotes the fitness value of the previous generation individual

[0057] ​​​The population aggregation degree calculation and the improved fitness quantization index effectively enhance the algorithm's perception ability of population structure and individual evolution dynamics. The individual aggregation degree calculation is compatible with the distance measurement between individuals of different dimensions, which can accurately evaluate the distribution density of individuals in the solution space and provide a reliable basis for identifying the population diversity level. The improved fitness formula quantifies the relative progress of individuals in the evolution process by comparing the current fitness with the initial or historical fitness, which can clearly reflect the optimization potential and convergence state of individuals. These formulas together constitute the core of state feature calculation, enabling the algorithm to deeply understand the global exploration breadth and local development depth of the population, providing key data basis for the subsequent adaptive selection of mutation strategies, thereby significantly improving the path planning algorithm's ability to avoid premature convergence, accelerate convergence, and approach the global optimal solution in complex environments.

[0058] Preferably, the construction of the path planning multi-mutation strategy action pool in step S3 includes:

[0059] Define four mutation operations: crossover , mutation , deletion , and improvement .

[0060] The crossover formula is as follows:

[0061]

[0062] In the formula, and represent the individual and individual selected according to the action strategy, represents the number of sequence numbers randomly selected in the minimum dimension of individual and individual .

[0063] The mutation formula is as follows:

[0064]

[0065] In the formula, represents a path node randomly selected from the nodes not contained in individual , represents the number of sequence numbers randomly selected in the dimension of individual .

[0066] The deletion formula is as follows:

[0067]

[0068] In the formula, This indicates the individual after the deletion operation. fitness value, Indicates the current individual fitness value;

[0069] The improvements The formula is as follows:

[0070]

[0071] In the formula, express Corresponding to The optimal position within the 8-neighborhood of a location node. Indicates the individual after the improvement operation fitness value;

[0072] Based on the four mutation operations above, an action pool is designed, which contains a total of six actions, as shown in the following formula:

[0073]

[0074] In the formula, This represents the generated mutated solution. , These represent the random selection of two different individuals from the population, excluding the current individual, as parent 1 and parent 2. This represents the optimal solution in the current population. This indicates the action value selected by the proximal policy optimization algorithm based on the current state of the individual.

[0075] By employing four mutation operations and their combinations, a feature-rich and hierarchically structured multi-mutation strategy action pool for path planning is constructed. The crossover operation merges different path segments to introduce population diversity; the mutation operation randomly perturbs path nodes to enhance exploration capabilities; the deletion operation removes redundant nodes to optimize path simplicity; and the improvement operation utilizes local neighborhood information for fine-tuning to improve path quality. Combining these four basic operations with different priorities and purposes forms six progressive action options, ranging from simple perturbations to complex recombinations, and from random exploration to guided optimization. This allows the algorithm to intelligently select the most suitable mutation strategy based on the current state characteristics of each individual. This design significantly enhances the algorithm's adaptability to different evolutionary stages and population states, enabling it to escape local optima through strong exploratory actions and accelerate local convergence through guided actions. Thus, it achieves a significant improvement in global search efficiency and final path quality in complex path planning problems.

[0076] Preferably, the step S4 of generating a variation vector according to the action, and performing a crossover operation and a selection operation on the variation vector and the corresponding current individual to generate a child individual includes:

[0077] Generating a variation individual by the action selected by the policy network , and the current individual performing a crossover operation to generate a new individual The formula is as follows:

[0078]

[0079] In the formula, indicates the new individual generated by the crossover;

[0080] The new child is generated through the selection operation, and the formula is as follows:

[0081]

[0082] In the formula, indicates the individual of the next generation , indicates the fitness value of the individual generated by the crossover.

[0083] By combining the policy network guided mutation and the classical genetic operation, an intelligent controllable population evolution mechanism is realized. First, a variation individual with a specific exploration or development tendency is generated according to the action selected by the policy network, and then a crossover operation is performed on the variation individual and the current individual, so that diversity or optimization disturbance is introduced in a targeted manner while inheriting the effective information of the original path. Subsequently, through strict selection operation, the individual with better fitness is reserved to enter the child, so as to ensure that the evolution direction of the population is always towards the solution with shorter and safer path. This process deeply integrates the intelligent decision-making ability of reinforcement learning and the robust search framework of genetic algorithm, so that the evolution of each generation is no longer a blind random attempt, but an adaptive optimization process based on the perception of the environment state, thereby significantly improving the pertinence and convergence efficiency of path search, effectively avoiding invalid iteration, and finally leading to a better global path planning result.

[0084] Preferably, the step S4 of obtaining the reward based on the fitness value of the child individual has the following formula:

[0085]

[0086]

[0087] In the formula, indicates the fitness value of the child individual , is the ratio of the fitness value of the child individual to the initial fitness estimate, To reward.

[0088] The reward mechanism builds an intuitive and effective evolutionary feedback signal by comparing the fitness ratio of offspring and initial individuals. Taking the negative ratio as the reward makes the algorithm clearly perceive the positive evolutionary direction of fitness value reduction, thereby providing a clear and stable optimization gradient for the update of the policy network. This design translates the long-term goal of path planning, i.e., finding shorter and safer paths, into immediate rewards and punishments at each step of iteration, strengthening the algorithm's learning and selection of efficient mutation strategies, effectively guiding the agent to make more beneficial decisions between exploration and utilization for global convergence, and ultimately promoting the continuous and stable improvement of the overall performance of the path planning method.

[0089] Preferably, the experience storage mechanism of the proximal policy optimization algorithm in step S5 is improved as follows:

[0090] An independent experience buffer is opened and maintained for each individual in the population;

[0091] The storage structure of each experience buffer includes:

[0092] A state sequence for storing the state feature vector extracted by the individual in each iteration;

[0093] An action sequence for storing the action selected by the individual based on the policy network in each iteration, the action being the number of mutation strategies in the path planning multi-mutation strategy action pool;

[0094] A reward sequence for storing the immediate reward value calculated by the individual according to the fitness value improvement of the offspring individual generated by it in each iteration;

[0095] A termination flag sequence for recording whether the individual reaches the preset iteration termination condition in each iteration;

[0096] During the iteration process, all individuals in the population synchronously perform state feature extraction, action selection, mutation and crossover operations, reward calculation, and experience buffer update operations, and the data storage and update between each experience buffer are independent of each other.

[0097] The improved experience storage mechanism builds a highly parallelized and data-isolated learning environment by setting up an independent experience buffer for each individual in the population. Each buffer structurally records the complete evolutionary trajectory of the corresponding individual, including state, action, reward, and termination flag, ensuring the integrity and timeliness of the experience data. This design enables the algorithm to capture a wealth of differentiated learning experiences within the population, avoiding strategy confusion caused by experience mixing, and providing a large number of high-quality and diverse training samples for subsequent batch updates of the policy network. The independent buffer supports synchronous exploration and learning for all individuals, greatly improving data collection efficiency and parallel computing potential, enhancing the adaptability of the agent to different evolutionary states and path patterns, and ultimately significantly enhancing the learning stability, convergence speed, and optimization performance of the entire path planning method in complex environments.

[0098] Preferably, the training process of the proximal policy optimization algorithm in step S6 includes:

[0099] The complete single optimization process from the initialization population to the maximum number of iterations is defined as a training round of the improved differential evolution algorithm In each training round, the old parameters of the current policy network are used to interact with the path planning environment to obtain a stable sequence of experience data;

[0100] After completing M training rounds, all experience data collected in the experience buffer is integrated as batch training samples to update the parameters of the policy network and the value network.

[0101] The network parameters are updated by optimizing the total loss function of the proximal policy optimization algorithm, wherein the total loss function is defined as:

[0102]

[0103] In the formula, represents the clipped policy loss, represents the value function loss, represents the policy entropy reward, and are hyperparameters used to balance the weights of different loss functions.

[0104] The calculation method of the clipped policy loss is:

[0105]

[0106] In the formula, represents the policy update amplitude, denotes the advantage function, as a hyperparameter to control the clipping range;

[0107] the policy update range is calculated as:

[0108]

[0109] wherein, denotes the policy update range, and respectively denote the probabilities of the new policy and the old policy taking action in state ;

[0110] the advantage function is calculated as:

[0111]

[0112] wherein, denotes the value of taking action in state , denotes the average value in state ;

[0113] the value function loss is calculated as:

[0114]

[0115] wherein, denotes the value function of state , denotes the target value function;

[0116] After the total loss function is calculated, the parameters of the policy network and the value network are updated by back propagation; the cyclic process of accumulating experience in training rounds and updating network parameters in batches is repeatedly executed until the upper limit of the training rounds is reached, and the optimal policy network parameters after training are saved.

[0117] The training process defines the complete differential evolution optimization process as a training round, skillfully combining the iterative search of evolutionary algorithms with the policy optimization of reinforcement learning, ensuring the continuity and stability of the training data. The batch update mechanism performs centralized learning after accumulating multiple rounds of experience, significantly improving data utilization efficiency and training stability. By constructing a composite loss function that includes the loss of pruning strategy, the loss of value function, and the reward of policy entropy, the process achieves fine control of the update amplitude of the strategy, effectively avoiding training collapse. At the same time, the advantage function is used to accurately evaluate the value of actions, guiding the evolution of the strategy towards improving long-term returns. The introduction of the entropy reward encourages moderate exploration and prevents the strategy from falling into local optima too early. This systematic training framework ensures that the policy network can learn stably from vast amounts of path planning interaction experience, gradually improving its ability to adaptively select the optimal mutation strategy, and ultimately enabling the algorithm to exhibit excellent global convergence performance and robustness in complex environments.

[0118] Preferably, step S7 comprises:

[0119] loading the saved optimal policy network parameters after training into the proximal policy optimization algorithm;

[0120] In the two-dimensional grid map environment, state feature extraction is performed on each individual in each generation of the population to obtain a current state feature vector;

[0121] The current state feature vector is input into the policy network loaded with the optimal policy network parameters, and the policy network outputs the probability distribution of all selectable actions in the path planning multi-mutation strategy action pool;

[0122] An action is randomly selected according to the probability distribution, and a corresponding mutation operation is performed to generate a new population individual;

[0123] The process of state feature extraction, action selection, and population update is repeated until the maximum number of iterations is reached;

[0124] The optimal path obtained at this time is output as the final planned path for the robot to move from the starting position to the target position.

[0125] By deploying the trained optimal policy network, the intelligent decision-making ability obtained through offline learning is efficiently applied to the actual path planning task. In each iteration, the algorithm dynamically outputs the optimal probability of each mutation action according to the real-time extracted population state features, and guides the generation of new individuals accordingly, realizing that the search process is completely driven by adaptive intelligent strategies. This mechanism ensures that the algorithm can fully utilize the learned experience and knowledge during the planning phase, and flexibly and accurately invoke the most effective mutation strategy combination for different environmental conditions and population distributions, thereby quickly and stably generating high-quality paths without additional training overhead. Ultimately, this method significantly improves the real-time performance, reliability, and global optimality of the solution of the online path planning of the robot, realizing a closed loop from learning to application.

[0126] Advantages: Compared with the prior art, the present application has the following significant advantages: 1. The present application fuses the mutation strategy of the improved differential evolution algorithm and the intelligent decision-making mechanism of deep reinforcement learning, so that the robot can dynamically evaluate the population state, individual stagnation and iteration progress during the path planning process, thereby adaptively selecting the most suitable optimization action from the diversified mutation operation combination, significantly reducing the risk of the traditional method falling into a local optimal solution in a complex environment, and ensuring the global optimality of the final planned path; 2. With the help of the continuous learning and strategy optimization of the path optimization process by the proximal policy optimization algorithm, the robot can quickly identify the effective path improvement direction based on historical experience, reducing invalid search and redundant iteration, thereby significantly improving the convergence speed and overall computational efficiency of the path planning while ensuring the quality of the path; 3. By designing an observation mechanism that fuses global distribution, local neighborhood, individual improvement amplitude and stagnation time, and combining the real-time decision-making of the strategy network of deep reinforcement learning, the robot can autonomously adjust the optimization strategy according to the environmental characteristics and search state at different stages, improving the adaptability and robustness of path planning in complex, dynamic or unknown environments. BRIEF DESCRIPTION OF DRAWINGS

[0127] Figure 1 The method flowchart of the present application is shown in the figure;

[0128] Figure 2 The 20*20 2D grid map of the present application is shown in the figure;

[0129] Figure 3 The path comparison graph of the robot path planning method and the differential evolution algorithm of the present application is shown in the figure;

[0130] Figure 4 The fitness value convergence curve graph of the robot path planning method and the differential evolution algorithm of the present application is shown in the figure. DETAILED DESCRIPTION

[0131] The technical solutions of the application will be further described below with reference to the drawings.

[0132] In the present application, the application proposes a robot path planning method based on improved differential evolution algorithm and deep reinforcement learning, and the flow is as shown in Figure 1 The robot path planning method based on improved differential evolution algorithm and deep reinforcement learning has the following steps:

[0133] Step 1: A two-dimensional grid map environment is constructed with the demand of the robot as the target, the impassable area in the map is marked as the obstacle position, the starting position and target position of the mobile robot are determined, as shown in Figure 2 The fitness function is established based on the map environment, wherein the fitness function is expressed by the following formula:

[0134]

[0135] In the formula, indicates the total fitness value of the i th individual, indicates the length fitness value of the i th individual, indicates the obstacle fitness value of the i th individual. The length fitness value is calculated according to the following formula:

[0136] The length fitness value is calculated according to the following formula:

[0137]

[0138] In the formula, indicates the total number of nodes of the current path, indicates the j th node in the i th individual in the current population, indicates the j th node in the i th individual in the current population. The obstacle fitness value is calculated according to the following formula:

[0139] The obstacle fitness value is calculated according to the following formula:

[0140]

[0141] In the formula, represents a constant, indicates the total number of obstacles passed by the current path.

[0142] ​​​Step 2: Initialize the population by roulette selection strategy, select the feasible nodes from the 8-neighborhood of the current node, calculate the Euclidean distance of each feasible node to the target position, select the next node by roulette probability, repeat the process until the target position is reached. Repeat the above process N times to obtain an initial population containing N feasible paths, and calculate the fitness value of the initial population according to the established fitness function;

[0143] Step 3: Based on the population position and fitness information, calculate the state characteristics of each individual in the population in turn, form a feature vector and input it into the proximal policy optimization algorithm, where the state characteristics are as follows:

[0144]

[0145] In the formula, represents the state characteristics, represents the global distribution degree, represents the local neighborhood distribution degree, represents the relative amplitude of individual fitness improvement, represents the continuous multiple generations of individual position update, represents the iteration progress.

[0146] Global distribution degree The formula is as follows:

[0147]

[0148] In the formula, represents the individual relative to the aggregation degree of the population, represents the population size.

[0149] Individual relative to the aggregation degree of the population The formula is as follows:

[0150]

[0151] In the formula, represents the individual Euclidean distance relative to the individual .

[0152] Individual Euclidean distance relative to the individual . The formula is as follows:

[0153]

[0154] In the formula, represents the current population, the i-th individual, and denote the dimension of the individual and the individual .

[0155] Local neighborhood distribution degree is given by

[0156]

[0157] where denotes the local neighborhood distribution degree of the individual , denotes the Euclidean distance between the individual and the closest individual.

[0158] Individual fitness improvement relative amplitude is given by

[0159]

[0160] where denotes the fitness improvement relative amplitude of the individual , denotes the fitness improvement of the individual , denotes the fitness improvement of the individual of the previous generation.

[0161] Individual fitness improvement is given by

[0162]

[0163] where denotes the fitness value of the initial individual , denotes the fitness value of the current individual .

[0164] Previous generation individual fitness improvement is given by

[0165]

[0166] where denotes the fitness value of the individual of the previous generation.

[0167] Individual position continuous generations not updated is given by

[0168] ​​

[0169] The stagnation constraints are as follows:

[0170]

[0171] In the formula, Represents an individual The position has not been updated for several generations. Indicates the current population's... Individual, This indicates that the first generation of the previous population Individual.

[0172] Iteration progress The formula is as follows:

[0173]

[0174] in, This indicates the current iteration number. This indicates the maximum number of iterations.

[0175] Then, a path planning multi-variation strategy is designed as an action pool, and the action in this state is obtained based on the policy network. The action pool is constructed as follows:

[0176] First, we define four mutation operations: crossover. ,mutation delete e. Improvement .

[0177] Among them, cross operation The formula is as follows:

[0178]

[0179] In the formula, and This indicates the individual selected based on the action strategy. and individuals , Indicates in the individual and individuals The number of randomly selected indices from the smallest dimension.

[0180] Mutation operation The formula is as follows:

[0181]

[0182] In the formula, Indicates in the individual Randomly selected path nodes from those not included. Indicates in the individual The number of randomly selected indices in the dimension.

[0183] Deletion operation The formula is as follows:

[0184]

[0185] In the formula, This indicates the individual after the deletion operation. fitness value, Indicates the current individual The fitness value.

[0186] Improved operation The formula is as follows:

[0187]

[0188] In the formula, express Corresponding to The optimal position within the 8-neighborhood of a location node. Indicates the individual after the improvement operation The fitness value.

[0189] Based on the four mutation operations above, an action pool is designed, which contains a total of six actions, as shown in the following formula:

[0190]

[0191] In the formula, This represents the generated mutated solution. , These represent the random selection of two different individuals from the population, excluding the current individual, as parent 1 and parent 2. This represents the optimal solution in the current population. This indicates the action value selected by the proximal policy optimization algorithm based on the current state of the individual.

[0192] Step 4: Generate a mutation vector based on the action, cross-reference it with the current individual, select the offspring, and execute the process as follows:

[0193] Mutant individuals are generated by actions selected by the policy network. , with the current individual Perform crossover operations to generate new individuals. The formula is as follows:

[0194]

[0195] In the formula, This represents a new individual generated through crossover.

[0196] The new offspring is generated by the selection operation, and the formula is as follows:

[0197]

[0198] In the formula, represents the individual of the next generation , represents the fitness value of the individual generated by crossover.

[0199] Then, the reward is obtained based on the fitness value of the offspring, and the reward formula is as follows:

[0200]

[0201]

[0202] In the formula, represents the fitness value of the offspring individual , is the ratio of the fitness value of the offspring individual to the initial fitness estimate, is the reward.

[0203] Step 5: Store the state, action, and reward of each individual in the individual's buffer area for a single iteration, and improve the experience storage mechanism of the proximal policy optimization algorithm in the following way:

[0204] The experience buffer storage structure of each individual includes:

[0205] State sequence: store the search feature vector extracted by the individual in each iteration;

[0206] Action sequence: store the mutation strategy number selected based on the policy network;

[0207] Reward sequence: store the immediate reward value calculated according to the offspring fitness improvement;

[0208] Termination flag: record whether the termination condition is reached in each iteration;

[0209] The proximal policy optimization algorithm opens an independent buffer area for each individual in the population, realizing isolated storage of individual experience data;

[0210] All individuals in the population perform state extraction, action selection, mutation and crossover, reward calculation and buffer area update operations synchronously, and the data update of each buffer area does not interfere with each other. Then return to step 3 until the maximum number of iterations is reached, complete a round of training round,

[0211] Step 6: Return to Step 2 to start the next round of training session, in which the complete single-optimization process of the improved differential evolution algorithm, i.e. from the initialization of the population to the maximum number of iterations, is taken as a complete training session Within this session, the old policy network parameters are used to interact with the path planning environment to obtain a stable sequence of experience data.

[0212] When the cumulative number of training sessions reaches , the experience data collected in all individual buffers is integrated as the training sample for batch updating the policy network and the value network. The network parameters are updated by optimizing the total loss function of the proximal policy optimization algorithm, which includes the clipped policy loss, the value function loss, and the policy entropy reward.

[0213] Total loss function of the proximal policy optimization algorithm is defined as follows:

[0214]

[0215] where represents the clipped policy loss, represents the value function loss, and represents the policy entropy reward. The greater the entropy of the policy, the more uniform the probability distribution of the actions taken by the policy in each state, and the stronger the exploration of the policy. and are hyperparameters used to balance the weights of different loss functions.

[0216] Clipped policy loss is defined as follows:

[0217]

[0218] where represents the policy update amplitude, represents the advantage function, and is a hyperparameter controlling the clipping amplitude.

[0219] Policy update amplitude is calculated as follows:

[0220]

[0221] where represents the update amplitude of the policy, and represent the probabilities of the new policy and the old policy taking action in state , respectively.

[0222] Advantage function The calculation is as follows:

[0223]

[0224] wherein, represents the value of the state under the action , and represents the average value in the state . The advantage function guides the generation of the policy network to continuously adjust the policy and improve the reward feedback of the algorithm.

[0225] The value function loss is calculated as follows:

[0226]

[0227] wherein, represents the value function of the state , and represents the target value function.

[0228] After the total loss function is calculated, the data is back-propagated so as to update the policy and the value function of the algorithm until the algorithm convergence condition is reached. The above-mentioned interaction-collection-updating cycle is repeated until the preset upper limit of the training rounds is reached. After the training is completed, the optimal policy network parameters are saved.

[0229] Step 7: The path planning method is applied to the map environment, and the optimal path can be obtained. The optimal policy network parameters are used to extract the features of each generation of individuals, and the near-end policy optimization algorithm randomly selects actions according to the action probability distribution in the state to update until the maximum number of iterations is reached, and the optimal path is output. The path planning method of the robot based on the improved differential evolution algorithm and deep reinforcement learning and the path of the differential evolution algorithm are compared as shown in Figure 3 , and the fitness value convergence curve of the path planning method of the robot based on the improved differential evolution algorithm and deep reinforcement learning and the differential evolution algorithm is shown in Figure 4 .

[0230] The effects of the embodiment can be further illustrated by simulation experiments:

[0231] The parameters are set, i.e., the maximum number of iterations is 500, the population size is 50, the number of training times

[0232] is 100 times, the cumulative number of rounds is 10, the learning rate of the value network is 0.001, and the learning rate of the policy network is 0.0003, the discount factor is 0.99, the number of rounds of parameter updating after collecting data is 80, and the parameters are clipped is 0.2, , .

[0233] The improved differential evolution algorithm and deep reinforcement learning robot path planning method are compared with the differential evolution algorithm. The algorithm is executed a total of 10 times, and the experimental results are compared in terms of maximum optimal value, minimum optimal value, average optimal value and standard deviation. The results are shown in Table 1.

[0234] Table 1 Performance comparison of improved algorithm and differential evolution algorithm

[0235] Algorithm Maximum Minimum Average PPO_DE algorithm 33.17 30.81 32.04 DE algorithm 34.61 32.75 33.63

[0236] It can be proved that, in terms of maximum optimal value, minimum optimal value and average optimal value, the improved differential evolution algorithm and deep reinforcement learning robot path planning method has more advantages in results. In summary, the improved differential evolution algorithm and deep reinforcement learning robot path planning method performs better, and its results are generally more advantageous than the original algorithm.

Claims

1. A robot path planning method based on improved differential evolution algorithm and deep reinforcement learning, characterized in that, The method comprises the following steps: S1, constructing a two-dimensional grid map environment according to the path finding requirement of the mobile robot, marking the obstacle position of the non-passable area in the two-dimensional grid map environment, determining the starting position and target position of the mobile robot, and establishing a fitness function based on the two-dimensional grid map environment; S2, initializing the population by adopting a roulette selection strategy, including: starting from the starting position, screening feasible nodes in the 8-neighborhood of the current node, calculating the Euclidean distance of each feasible node to the target position, selecting the next node by roulette probability, repeating the selection process until reaching the target position to generate a feasible path, repeating the generation process N times to obtain an initial population containing N feasible paths, and calculating the fitness value of the initial population according to the fitness function; S3, based on the position information and fitness value of each individual in the initial population, calculating the state characteristics of each individual to form a feature vector, inputting the feature vector into the policy network of the proximal policy optimization algorithm, and the policy network selects the corresponding action from the preset path planning multi-mutation strategy action pool according to the input feature vector; S4, generating a mutation vector according to the action, performing crossover and selection operations on the mutation vector and the corresponding current individual to generate offspring individuals, and obtaining a reward based on the fitness value of the offspring individuals; S5, storing the state, action and reward of each individual in a single iteration process in an independent buffer area corresponding to the individual, and repeating steps S3 to S5 until a preset maximum iteration number is reached to complete a training round; S6, repeating steps S2 to S5 to perform the next training round, and when a preset number of training rounds are completed, updating the policy network parameters and value network parameters of the proximal policy optimization algorithm in batches using the experience data stored in each independent buffer area, emptying all independent buffer areas, returning to step S2 to reinitialize the population, and continuing training until a preset upper limit of training rounds is reached to complete the training of the path planning method; S7, applying the trained path planning method to the two-dimensional grid map environment to output the optimal path of the robot from the starting position to the target position.

2. The method of claim 1, wherein, The expression of the fitness function in step S1 is: wherein, represents the total fitness value of the th individual, represents the length fitness value of the th individual, represents the obstacle fitness value of the th individual; the length fitness value The formula is as follows: In the formula, This indicates the total number of nodes in the current path. Indicates the first in the current population The first of the individuals 1 node Indicates the first in the current population The first of the individuals One node; The disorder fitness value The formula is as follows: ; where, represents a constant, denotes the total number of obstacles the current path passes through.

3. The method of claim 1, wherein, The state characteristics in step S3 are as follows: ; where, represents a state characteristic, represents a global distribution degree, represents a local neighborhood distribution degree, represents an individual fitness improvement relative amplitude, represents an individual position continuous generations not updated, represents an iteration progress; the global distribution degree The formula is as follows: ; wherein represents the individual the degree of aggregation of the population, represents the population size; the local neighborhood distribution degree The formula is as follows: wherein, denotes the local neighborhood distribution of an individual , denotes the Euclidean distance between the individual closest to the individual ; The relative amplitude of the individual fitness improvement The formula is as follows: wherein denotes the fitness improvement of the individual relative to the average fitness improvement of the population, denotes the fitness improvement of the individual relative to the average fitness improvement of the population, denotes the fitness improvement of the individual relative to the average fitness improvement of the population; The individual location has not been updated for successive generations The formula is as follows: ; the stall constraint is as follows: ; wherein represents an individual position is not updated for successive generations, represents the i-th individual of the current population, represents the i-th individual of the current population, represents the i-th individual of the previous population, represents the i-th individual of the previous population, the iteration progress The formula is as follows: ; wherein, denotes the number of the current iteration, denotes the maximum number of iterations.

4. The method of claim 3, wherein, the degree of clustering of the individual i relative to the population The formula is as follows: wherein represents the individual with respect to the individual Euclidean distance, the calculation formula is as follows: wherein, denotes the i-th individual of the current population, and denote the dimension of the individual and the individual , respectively.​ The individual Fitness improvement The formula is as follows: wherein denotes the fitness value of the initial individual , denotes the fitness value of the current individual . The previous generation of individuals Fitness improvement The formula is as follows: ; where, represents the fitness value of the previous generation of individuals .

5. The method of claim 1, wherein, The construction of the path planning multi-mutation strategy action pool in step S3 includes: Four kinds of mutation operations are defined: crossover , mutation , deletion , improvement ; wherein the intersection The formula is as follows: wherein and denotes an individual selected according to the action policy and individual , denotes an individual selected according to the action policy and individual a number of indices randomly selected in the minimum dimension of the individual The mutations The formula is as follows: ; wherein denotes a randomly selected path node in the nodes not contained in the individual denotes a randomly selected path node in the nodes not contained in the individual denotes a randomly selected number of indices in the dimensionality of the individual denotes a randomly selected number of indices in the dimensionality of the individual The deletion The formula is as follows: wherein represents the fitness value of the individual after the deletion operation, represents the fitness value of the current individual . The improvements The formula is as follows: In the formula, express Corresponding to The optimal position within the 8-neighborhood of a location node. Indicates the individual after the improvement operation fitness value; Based on the above four mutation operations, an action pool is designed, which contains six actions in total, and the formula is as follows: ; where, represents a generated mutated solution, , respectively represent randomly selecting two different individuals in the population other than the current individual as parent 1 and parent 2, represents the best solution in the current population, represents an action value selected by the proximal policy optimization algorithm according to the state of the current individual.

6. The method of claim 1, wherein, The generation of offspring individuals by crossover and selection operations of the mutation vector and the corresponding current individual in step S4 includes: Actions selected by the policy network generate mutated individuals with the current individual are crossed to produce new individuals The formula is as follows: ; wherein denotes the cross-over of new individuals; New offspring are generated by selection operation, and the formula is as follows: ; where, represents the next generation of individuals , represents the fitness value of the individual resulting from crossover.

7. The method of claim 1, wherein, The formula for obtaining a reward based on the fitness value of the offspring individuals in step S4 is as follows: ; ; wherein, represents the fitness value of the offspring individual , is the ratio of the fitness value of the offspring individual to the initial fitness estimate, is the reward.

8. The method of claim 1, wherein, In step S5, the experience storage mechanism of the proximal policy optimization algorithm is improved as follows: An independent experience buffer area is opened and maintained for each individual in the population; The storage structure of each experience buffer area includes: a state sequence for storing the state feature vector extracted by the individual in each iteration; an action sequence for storing the action selected by the individual in each iteration based on the policy network, the action being the number of the mutation strategy in the path planning multi-mutation strategy action pool; a reward sequence for storing the immediate reward value calculated by the individual in each iteration according to the improvement of the fitness value of the offspring individual generated thereby; a termination flag sequence for recording whether the individual reaches the preset iteration termination condition in each iteration; In the iteration process, all individuals in the population synchronously perform state feature extraction, action selection, mutation and crossover operations, reward calculation and experience buffer updating operations, and the data storage and updating among the experience buffers are independent of each other.

9. The method of claim 1, wherein, The training process of the proximal policy optimization algorithm in step S6 includes: A complete single optimization process starting from an initialization population until a maximum number of iterations is reached is defined as one training round of the improved differential evolution algorithm ; within each training round, the old parameters of the current policy network are used to interact with the path planning environment to obtain a stable sequence of experience data; After the cumulative completion of M training rounds, integrate the experience data collected in all experience buffers as batch training samples for updating the parameters of the policy network and the value network; By optimizing a total loss function of a proximal policy optimization algorithm to update network parameters, the total loss function is defined as: where, represents the loss of the clipping strategy, represents the loss of the value function, represents the entropy reward of the strategy, and is a hyperparameter used to balance the weights of different loss functions; The cropping strategy loss The calculation method is: ; where, denotes the policy update magnitude, denotes the advantage function, controls the clipping magnitude as a hyperparameter; The policy update amplitude The calculation is as follows: ; where, represents the update magnitude of the policy, and represent the probabilities of the new and old policies taking action in state , respectively. The merit function is calculated as follows: wherein represents the value of taking action in state , represents the average value in state . The value function loss is calculated as: ; where, represents a state of the value function, represents a target value function; The total loss function is calculated Afterwards, the parameters of the policy network and the value network are updated by backpropagation; the loop process of repeatedly performing the training episode to accumulate experience and updating the network parameters in batches is executed until the upper limit of the training rounds is reached, and the optimal policy network parameters after training are saved.

10. The method of claim 1, wherein, Step S7 includes: loading the optimal policy network parameters saved after training into the proximal policy optimization algorithm; In the two-dimensional grid map environment, state feature extraction is performed on each individual in each generation of the population to obtain a current state feature vector; inputting the current state feature vector into the policy network loaded with the optimal policy network parameters, the policy network outputting the probability distribution of all selectable actions in the path planning multi-mutation strategy action pool; randomly selecting an action according to the probability distribution and performing a corresponding mutation operation to generate a new population individual; repeating the process of state feature extraction, action selection and population updating until the maximum number of iterations is reached; outputting the optimal path obtained at this time as the final planning path of the robot from the starting position to the target position.

Images