Patrol robot trajectory planning method suitable for compact obstacle environment of power distribution room

Abstract

The invention provides an inspection robot trajectory planning method suitable for a compact obstacle environment of a power distribution room, and solves the problems that keys are compact and an obstacle is close to a target key in the power distribution room environment. According to the method, a bee colony and a Q table are initialized, a fitness function is constructed, robot motion and environment constraints are introduced, and an exploration or development strategy is dynamically selected according to a population state. A particle swarm algorithm with control parameters is introduced in a scout bee stage, and the global search capability and convergence efficiency are enhanced. The optimal solution is updated through an elitist strategy and neighborhood search, and the Q value is adaptively adjusted by means of feedback, so that the strategy flexibility is improved. According to the method, the global search advantage of the particle swarm and the local development advantage of the artificial bee colony are integrated, the trajectory precision and the operation safety are guaranteed, the efficiency and the real-time performance are also improved, the success rate reaches 98.16%, the time is reduced by 10.11% compared with that of an artificial bee colony algorithm, and the method is suitable for popularization and application. The method is suitable for a patrol robot track planning task with a complex distribution room structure, a compact control panel arrangement and a closer obstacle to a target.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of track planning of inspection robots, and particularly to the field of robot track planning method based on power distribution room power environment, and relates to a track planning method for inspection robots in a compact obstacle environment of a power distribution room. BACKGROUND

[0002] In a modern power system, the operation state of a power distribution room is directly related to the safety and stability of the entire power grid. Although the traditional manual inspection method ensures the safety of equipment to a certain extent, it has problems such as high work intensity and high operation risk. With the development of intelligent technology, inspection robots represented by industrial robots, special operation robots, and service consumption robots, and automatic systems constructed relying on advanced technologies such as additive manufacturing equipment manufacturing and industrial automatic control system device manufacturing, have gradually become an important means to improve the operation efficiency and safety of power distribution rooms. The environment of a power distribution room is relatively dangerous, and the precision requirement for robot operation is also higher. The spatial structure is compact, and the switch cabinet panel usually has a large number of physical buttons and indicator lights with similar shapes and small distances. At the same time, there may be obstacles such as left-over maintenance cards with close distance keys during the inspection process. These obstacle positions are random and small in size, posing a severe challenge to the perception and operation ability of the inspection robot, and exposing the problems of insufficient real-time performance and speed. The breakthrough of such technical bottlenecks depends on the continuous improvement of industrial automatic control system device manufacturing, robot precision control, and intelligent integration capability, and has become a key link that needs to be broken through for current intelligent operation and maintenance.

[0003] At present, commonly used track planning methods include genetic algorithm, particle swarm algorithm, and A* algorithm, but these methods generally have problems such as low parameter sensitivity, slow convergence speed, and weak processing ability for complex constraints, and are difficult to meet the actual needs of high-precision and high-efficiency track planning. Therefore, the method proposes a method of fusing an artificial bee colony algorithm and a learning factor particle swarm algorithm with control parameters under 3-5-3 polynomial trajectory modeling and constraint conditions, and introduces a Q learning mechanism to dynamically adjust the search strategy. By introducing the learning factor particle swarm mechanism with control parameters in the scout bee stage, the information transmission ability and global search performance of individuals are enhanced, and the adaptability of the population to complex trajectory constraints is effectively improved. Experiments show that the method not only reduces the optimal fitness value of the trajectory, improves the search precision, but also has higher parameter sensitivity and shorter planning time, and exhibits superior comprehensive performance.

[0004] The current commonly used algorithms have the following problems in the research of track planning of inspection robots:

[0005] (1) The power distribution room belongs to an operation dangerous place, the internal environment is complex, there are temporary obstacles such as maintenance signs, and the obstacles are close to the operation button, which has certain requirements for the movement accuracy of the inspection robot, and the operation button also has certain requirements for time, the current algorithm iteration convergence speed is slow, and the real-time of the inspection robot cannot be guaranteed, although the document "Path planning method based on improved particle swarm algorithm and storage medium" uses particle swarm algorithm to fuse Q learning to enhance and improve the calculation efficiency of trajectory planning, but it is more suitable for the environment of moving obstacles, and the convergence speed is still insufficient, the Q learning is deeply integrated into the PSO algorithm, and the cosine function in the improved learning factor provides smooth change, avoids the shock caused by mutation, and provides faster convergence speed, and is more suitable for the compact power distribution room environment;

[0006] (2) Although the existing research attempts to introduce reinforcement learning mechanism into artificial bee colony algorithm to improve performance, such as the document "Improved artificial bee colony algorithm for three-dimensional unmanned aerial vehicle logistics path planning problem", but it is not mainly applied in the power distribution room environment, there are many small size and close static obstacles such as maintenance signs in the power distribution room, and the distance is compact, although the above research uses reinforcement learning to enhance the calculation efficiency of the algorithm, but the calculation complexity is relatively high, and the parameter adjustment is relatively complex, mainly suitable for macroscopic continuous loose three-dimensional space, and it is difficult to meet the requirements of the power distribution room environment, the learning factor can be automatically adjusted according to the iteration process, the self-adaptive mechanism is more refined and mathematical, so that the control accuracy of the mechanical arm is enhanced. SUMMARY

[0007] The application innovatively provides a trajectory planning method for a compact obstacle environment of a power distribution room inspection robot, in the power distribution room environment, the introduction of Q learning calculation reward significantly improves the quality of understanding and the directionality of search, solves the problem of insufficient search directionality in the traditional algorithm and easy to fall into local optimal solution, the scout bee stage uses the idea of particle swarm algorithm, combines the advantages of fast convergence speed of particle swarm algorithm, considers global search and local search, realizes more accurate obstacle avoidance, and significantly speeds up global convergence; meanwhile, 3-5-3 polynomial is used for constraint, ensures the continuity of speed acceleration, realizes smooth trajectory of the inspection robot and stable lifting mechanism, and the polynomial provides sufficient degrees of freedom to meet the trajectory constraint, the structure is relatively fixed, the solving coefficient is efficient, and technical support is provided for safe and efficient operation of the inspection robot in power distribution automation inspection, operation and other tasks.

[0008] In order to achieve the above purpose, the application adopts the following specific technical solutions to solve the above problems:

[0009] S1: The inspection robot is mathematically modeled, obstacle constraints are set, and a segmented polynomial interpolation composed of cubic and quintic polynomials is used as an inspection robot trajectory optimization mathematical model;

[0010] S1.1: Normalization of the overall planning time, time t ∈ [0, 1], the time normalization formula is:

[0011]

[0012] In the formula, t is the normalized time variable, T is the actual use time of the joint, T i is the actual use time of the end of the inspection robot from the starting point to the end of the i-th trajectory, i ∈ [1, 3];

[0013] S1.2: Establish the interpolation function of the j-th joint with respect to time in each time period as follows:

[0014]

[0015] In the formula, t1, t2 and t3 are three time periods, θ j1 , θ j2 and θ j3 are the angular displacements of the joint in the three time periods;

[0016] S1.3: First and second order derivatives of equation (2) are obtained to obtain the function expressions of the angular velocity and angular acceleration of the joint with respect to time in the three time periods, and the coefficients in the polynomial are obtained by a = A -1 b, the specific formula is as follows:

[0017]

[0018] b = [000000 X j3 00 X j0 00 X j2 X j1 ] T (4)

[0019] In the formula, X j0 , X j1 , X j2 , X j3 are the joint target positions of the starting point, the middle point 1, the middle point 2 and the end point of the j-th joint, respectively;

[0020] S2: Take the time t of 3-5-3 polynomial interpolation as the optimization variable, and the optimal objective function is represented as:

[0021] f(t) = min(t j1 +t j2 +t j3 ) (5)

[0022] In the formula, t j1 , t j2 , tj3 These represent the running time of the j-th joint of the inspection robot in each trajectory segment;

[0023] S3: Initialize parameters, set the total number of bees and particle swarm parameters, bee position, initial velocity and constraints, etc.

[0024] S3.1: Use the FR5 intelligent collaborative robot to perform mathematical modeling and formulate soft constraints;

[0025] S3.2: Initialize Q-learning, create a Q-table with four states and two actions, and set the learning rate discount factor and exploration rate;

[0026] S4: Calculate the population diversity in the current state and map it to the four states;

[0027] S4.1: Use an ε-greedy strategy to randomly select exploration or development actions. These actions affect the radius of the domain.

[0028] The formula for the British radius is as follows:

[0029] r1 = 0.4 * (1 - k / N) max )+0.1 (6)

[0030] r2 = 0.2 * (1 - k / N) max )+0.05 (7)

[0031] In the formula, r1 and r2 are the elite radii of the exploration and development actions, respectively, k is the current iteration number, and N is the elite radius of the development action. max This represents the maximum number of iterations.

[0032] S5: The bee colony is set to have 20 bees in the mercenary bee stage and 10 bees in the scout bee stage;

[0033] S5.1: In the bee-hiring phase, use 6 elite bees to generate 12 neighborhood solutions within the elite radius, and 14 suboptimal bees to generate 28 neighborhood solutions within the suboptimal radius. Each bee adopts one solution.

[0034] S5.2: While calculating the constraint degree of the new solution, check each constraint, and then update the individual to select the globally optimal neighborhood solution;

[0035] S5.3: In the scout bee phase, the position of unselected bees is updated using the particle swarm optimization algorithm. Based on the iteration progress and population diversity, different strategies are used to adaptively adjust the particle swarm parameters in the early and later stages of the algorithm's operation.

[0036]

[0037] Wherein, c1 and c2 are respectively pre and post learning factors, and alpha is a control parameter alpha belongs to (0, 1);

[0038] S5.4: evaluating the new position on the basis of the constraint and updating the optimal solution;

[0039] S6: calculating each reward and penalty according to the optimal position feedback optimization information obtained in the above two stages and updating the Q table, and the Q table updating equation and adaptive exploration rate are as follows:

[0040] Q (s, a) = Q (s, a) + alpha [r + gamma * maxQ (s', a') - Q (s, a)] (9)

[0041] Epsilon = epsilon0 * 0.95k (10)

[0042] Wherein, s and s' are respectively current state and next state, a and a' are executed action and possible action, alpha is a proportional parameter, r is an immediate reward, gamma is a discount factor, and epsilon and epsilon0 respectively indicate current exploration rate and initial exploration rate;

[0043] S7: reiterating the above steps, the iteration number is set to 100 times, and the best fitness and minimum time consumption trajectory are obtained after multiple iterations.

[0044] The present application has the following beneficial effects:

[0045] (1) The power distribution room inspection robot trajectory planning method has strong robustness in the face of complex and compact environment, adopts a model constructed by 3-5-3 segmented polynomial interpolation and uses the selection of Q learning guided strategy, the algorithm can dynamically respond to environmental changes, is suitable for complex static environment, optimizes the algorithm performance, significantly reduces the operation time, is more continuous in speed than the traditional algorithm, has strong time controllability, effectively improves the safety of the inspection robot in the complex environment by combining the soft and hard constraint processing mechanism, guarantees the operation of the robot in a small space, uses the particle swarm learning factor formula with a control parameter, enhances the control ability of diversity, effectively improves the success rate of trajectory planning, reduces the success rates of simulation, traditional particle swarm algorithm and traditional artificial bee colony algorithm and grey wolf algorithm to 79.31%, 84.83% and 94.25% respectively, and the success rate of the present application algorithm reaches 98.16%; in terms of trajectory length, the present application algorithm reduces 0.67m and 0.34m compared with the traditional particle swarm algorithm and the traditional grey wolf algorithm respectively;

[0046] (2) The traditional artificial bee colony algorithm has weak local development ability and is prone to optimal solution, but the algorithm fuses the particle swarm algorithm with a control parameter learning factor, the learning factor with the control parameter can better balance global and local search, and the adaptive learning factor changes smoothly with the iteration process, avoiding the shock caused by parameter mutation and accelerating the convergence process. Multiple planning iterations select the most suitable trajectory. Simulation proves that, in terms of running time, the algorithm has a more obvious improvement effect than the traditional artificial bee colony algorithm and the particle swarm algorithm, and in terms of running time, the algorithm reduces 27.36% and 10.11% compared with the traditional particle swarm algorithm and the artificial bee colony algorithm respectively. BRIEF DESCRIPTION OF DRAWINGS

[0047] Figure 1 is a general flowchart of the trajectory planning method of the inspection robot;

[0048] Figure 2 is the best fitness iterative convergence effect diagram of the algorithm;

[0049] Figure 3 is the best fitness iterative convergence effect diagram of the traditional particle swarm algorithm;

[0050] Figure 4 is the best fitness iterative convergence effect diagram of the traditional grey wolf algorithm;

[0051] Figure 5 is the best fitness iterative effect diagram of the traditional artificial bee colony algorithm;

[0052] Figure 6 is the actual inspection robot and the power distribution room environment diagram;

[0053] Figure 7 is the joint position, velocity and acceleration of the optimized inspection robot of the algorithm;

[0054] Figure 8 is the end trajectory curve diagram of the optimized inspection robot of the algorithm. DETAILED DESCRIPTION

[0055] The technical solutions in the embodiments of the application will be described clearly and completely below. The application relates to a trajectory planning method for a compact obstacle environment inspection robot in a power distribution room.

[0056] Embodiment: as Figure 1As shown, the overall flow chart of the trajectory planning of the robot based on the artificial bee colony algorithm and the particle swarm with the control parameter learning factor and the Q learning. First, initialize the bee colony information and the Q learning, set the fitness function and determine the constraint conditions according to the model parameters of the inspection robot and the high-precision requirements of the power distribution room; then, the population diversity and the iteration progress are mapped to four states based on the diversity and the iteration progress, and the action exploration or development is selected; then, the scout bee stage is carried out, the elite neighborhood solution is generated using the elite radius or the suboptimal neighborhood solution is generated using the suboptimal radius, the unselected bees perform the scout bee stage, the new position is evaluated using the particle swarm idea and the optimal solution is updated; then, the results of the two stages are calculated for rewards and the Q table is updated; finally, it is judged whether the maximum iteration number is reached, if not, the above steps are performed again to optimize the trajectory, and if the required number of times is reached, the cycle is ended, which specifically includes the following steps:

[0057] S1: The inspection robot is mathematically modeled, the obstacle constraints are set, and the piecewise polynomial interpolation composed of cubic and quintic polynomials is used as the mathematical model for trajectory optimization of the inspection robot;

[0058] S1.1: The overall planning time is normalized, so that the time t∈[0,1], and the time normalization formula is:

[0059]

[0060] In the formula, t is the normalized time variable, T is the actual use time of the joint, T i is the actual use time of the inspection robot end from the starting point to the i-th trajectory termination point, i∈[1,3];

[0061] S1.2: The interpolation function of the j-th joint with respect to time in each time period is established as follows:

[0062]

[0063] In the formula, t1, t2 and t3 are the three time periods, θ j1 , θ j2 and θ j3 are the angular displacements of the joints in the three time periods;

[0064] S1.3: The first and second derivatives of formula (2) are obtained, and the function expressions of the angular velocity and angular acceleration of the joint with respect to time in the three time periods are obtained, and the coefficients in the polynomial are obtained through a=A -1 b, and the specific formula is as follows:

[0065]

[0066] b=[000000 X j3 00 Xj0 00 X j2 X j1 ] T (4)

[0067] wherein X j0 , X j1 , X j2 , X j3 are the joint target positions of the start point, the middle point 1, the middle point 2 and the end point of the jth joint respectively;

[0068] S2: taking the time t of 3-5-3 polynomial interpolation as the optimization variable, the optimal objective function is expressed as:

[0069] f(t) = min(t j1 +t j2 +t j3 ) (5)

[0070] wherein t j1 , t j2 , t j3 represent the running time of the jth joint of the inspection robot in each segment of the trajectory respectively;

[0071] S3: initializing parameters, setting the total number of bees, the parameters of the particle swarm, the position and initial speed of the bees, the constraint conditions and other information;

[0072] S3.1: using the robot FR5-intelligent collaborative robot to perform mathematical modeling and formulate soft constraints;

[0073] S3.2: initializing Q learning, creating a Q table of four states and two actions, and setting the learning rate discount factor and the exploration rate;

[0074] S4: calculating the population diversity of the current state and mapping it to four states;

[0075] S4.1: using an ε-greedy strategy to randomly select an exploration or development action, which will affect the field radius, the precision

[0076] of the action is as follows:

[0077] r1 = 0.4 * (1-k / N max ) + 0.1 (6)

[0078] r2 = 0.2 * (1-k / N max ) + 0.05 (7)

[0079] wherein r1 and r2 are the elite radii of the exploration action and the development action respectively, k is the current iteration number, and N max is the maximum iteration number;

[0080] S5: The colony is set to the employed bee phase with 20 bees and the onlooker bee phase with 10 bees;

[0081] S5.1: In the employed bee phase, 6 elite bees are used to generate 12 neighborhood solutions under the elite radius, and 14 sub-optimal bees are used to generate 28 neighborhood solutions under the sub-optimal radius, and each bee adopts one solution;

[0082] S5.2: The constraints are checked while calculating the degree of constraint of the new solution, and then the global optimal neighborhood solution is updated by individual selection;

[0083] S5.3: In the onlooker bee phase, the position of the unselected bees is updated using the idea of the particle swarm algorithm, and according to the iteration progress and population diversity, different strategies are used to adaptively adjust the particle swarm parameters in the early and late stages of the algorithm running:

[0084]

[0085] wherein c1 and c2 are the learning factors in the early and late stages respectively, and α is a control parameter α ∈ (0, 1);

[0086] S5.4: The new position is evaluated on the basis of checking the constraints and the optimal solution is updated;

[0087] S6: The rewards and penalties are calculated according to the feedback optimization information obtained from the above two phases, and the Q table is updated, and the Q table update equation and the adaptive exploration rate are as follows:

[0088] Q(s, a) = Q(s, a) + α [r + γ * maxQ(s', a') - Q(s, a)] (9)

[0089] ε = ε0*0.95k (10)

[0090] wherein s and s' are the current state and the next state respectively, a and a' are the executed action and the possible action, α is the proportion parameter, r is the immediate reward, γ is the discount factor, and ε and ε0 are the current exploration rate and the initial exploration rate respectively;

[0091] S7: The above steps are iterated again, and the number of iterations is set to 100 times, and the best fitness and the minimum time-consuming trajectory are obtained after multiple iterations.

[0092] In order to further verify the actual effect of the foregoing scheme, the application uses FR5 robot as an experimental platform, and models the actual power distribution room environment to carry out simulation experiments in Python3.8. In order to verify the better adaptability of the algorithm, the traditional particle swarm algorithm, the grey wolf algorithm and the artificial bee colony algorithm are compared, and multiple iteration experiments are carried out, and the time, the trajectory length and the success rate are obtained respectively, and it is verified that the algorithm is better than the traditional method in the above three performance indicators, and the better adaptability and comprehensive performance are shown, and the specific simulation data is shown in Table 1.

[0093] Table 1 Four algorithm experiment data table

[0094]

[0095] From Table 1, the algorithm of the application has a significant improvement in the global optimal position of time compared with the particle swarm algorithm, the grey wolf algorithm and the artificial bee colony algorithm:

[0096] (1) In terms of running time and optimal fitness, because the optimal fitness function is set and the planning time is the same, the planning time of the application is only 1.82s, which is reduced by 0.21s compared with the traditional artificial bee colony algorithm, and is reduced by 0.69s and 0.36s respectively compared with the particle swarm algorithm and the grey wolf algorithm, and the running time is obviously reduced.

[0097] (2) In terms of success rate, the success rate of the traditional particle swarm algorithm is 79.31%, the success rate of the grey wolf algorithm is 84.83%, the success rate of the traditional artificial bee colony algorithm is 94.25%, and the success rate of the algorithm of the application is 98.16%.

[0098] (3) In terms of trajectory length, the algorithm of the application is reduced by 0.67m, 0.34m and 0.06m respectively compared with the traditional particle swarm algorithm, the grey wolf algorithm and the traditional artificial bee colony algorithm, and the trajectory length is improved.

[0099] When the trajectory planning is carried out in different ways, the simulation diagram of the optimal fitness iterative convergence curve of the algorithm of the application is as shown in Figure 2 , the simulation result of the optimal fitness iterative convergence curve of the traditional particle swarm is as shown in Figure 3 , the optimal fitness iterative convergence curve of the grey wolf algorithm is as shown in Figure 4 , the optimal fitness iterative convergence curve of the artificial bee colony algorithm is as shown in Figure 5 , the actual inspection robot adopted by the application is as shown in Figure 6 , the position, velocity and acceleration of each joint of the optimized inspection robot of the algorithm are as shown in Figure 7 , and the end trajectory curve of the optimized inspection robot of the algorithm is as shown in Figure 8As shown, the purple cuboid is a switch cabinet, the green cuboid is a maintenance card obstacle, and the red cylinder is a target button. The algorithm performs trajectory planning in an environment with accuracy requirements in the distribution room, reduces the time consumption of the trajectory planning of the inspection robot, greatly enhances the real-time performance of the robot, and improves the work efficiency.

[0100] Finally, it should be pointed out that the above examples are only used to illustrate the technical solutions of the present application and are not limiting. Although the technical solutions of the present application have been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or replaced equivalently without departing from the purpose and scope of the present application, and they should be covered in the protection scope of the present application.

Claims

1. A trajectory planning method for a robot inspecting compact obstacle environments in a power distribution room, characterized in that, Includes the following steps: S1: Mathematical modeling of the inspection robot is performed, obstacle constraints are set, and piecewise polynomial interpolation is composed of cubic and quintic polynomials as the mathematical model for optimizing the trajectory of the inspection robot. S1.1: Normalize the overall planning time so that time t∈[0,1]. The time normalization formula is: In the formula, t is the normalized time variable, T is the actual usage time of the joint, and T i The actual time required for the inspection robot's end effector to travel from the starting point to the end point of the i-th segment of the trajectory; S1.2: The interpolation function for the j-th joint with respect to time within each time interval is established as follows: In the formula, t1, t2, and t3 are three time periods, and θ j1 θ j2 With θ j3 The angular displacement of the joint over three time periods; S1.3: Taking the first and second derivatives of equation (2), we obtain the functional expressions of joint angular velocity and angular acceleration with respect to time in the three time intervals. The coefficients in the polynomial are obtained through a = A -1 b yields the following specific formula: b=[0 0 0 0 0 0 X j3 0 0 X j0 0 0 X j2 X j1 ] T (4) In the formula, X j0 X j1 X j2 X j3 These are the target joint positions of the starting point, intermediate point 1, intermediate point 2, and ending point of the j-th joint, respectively. S2: Using the time t of the 3-5-3 polynomial interpolation as the optimization variable, the optimal objective function is expressed as: f(t)=min(t j1 +t j2 +t j3 ) (5) In the formula, t j1 t j2 t j3 These represent the running time of the j-th joint of the inspection robot in each trajectory segment; S3: Initialize parameters, set the total number of bees and particle swarm parameters, bee position, initial velocity and constraints, etc. S3.1: Use the FR5 intelligent collaborative robot to perform mathematical modeling and formulate soft constraints; S3.2: Initialize Q-learning, create a Q-table with four states and two actions, and set the learning rate discount factor and exploration rate; S4: Calculate the population diversity in the current state and map it to the four states; S4.1: Use an ε-greedy strategy to randomly select exploration or development actions. The action will affect the territory radius. The elite radius is as follows: r1=0.4*(1-k / N max )+0.1 (6) r2=0.2*(1-k / N max )+0.05 (7) In the formula, r1 and r2 are the elite radii of the exploration and development actions, respectively, k is the current iteration number, and N is the elite radius of the development action. max This represents the maximum number of iterations. S5: The bee colony is set to have 20 bees in the mercenary bee stage and 10 bees in the scout bee stage; S5.1: In the bee-hiring phase, use 6 elite bees to generate 12 neighborhood solutions within the elite radius, and 14 suboptimal bees to generate 28 neighborhood solutions within the suboptimal radius. Each bee adopts one solution. S5.2: While calculating the constraint degree of the new solution, check each constraint, and then update the individual to select the globally optimal neighborhood solution; S5.3: In the scout bee phase, the position of unselected bees is updated using the particle swarm optimization algorithm. Based on the iteration progress and population diversity, different strategies are used to adaptively adjust the particle swarm parameters in the early and later stages of the algorithm's operation. In the formula, c1 and c2 are the learning factors before and after the period, respectively, and α is the control parameter α∈(0,1); S5.4: Evaluate the new position and update the optimal solution based on the constraints; S6: Calculate various rewards and penalties based on the optimal position feedback information obtained in the above two stages and update the Q-table. The Q-table update equation and adaptive exploration rate are as follows: Q(s,a)=Q(s,a)+α[r+γ*maxQ(s',a')-Q(s,a)] (9) ε=ε0*0.95k (10) In the formula, s and s' represent the current state and the next state, respectively; a and a' represent the action to be performed and the possible action, respectively; α is the proportional parameter; r is the reward obtained immediately; γ is the discount factor; and ε and ε0 represent the current exploration rate and the initial exploration rate, respectively. S7: Iterate the above steps again, setting the number of iterations to 100. After multiple iterations, obtain the optimal fitness and the trajectory with the minimum time consumption.

Images