TD3-mixed multi-mechanism collaborative adaptive evolutionary algorithm
By introducing population reconstruction and roulette to select individuals in the TD3 algorithm, the problem that the TD3 algorithm is prone to fall into local optimality in the high-dimensional multi-objective problem is solved, and higher optimization accuracy and speed are achieved.
Patent Information
- Application Number
- CN202411796507.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-09
- Publication Date
- 2025-04-08
AI Technical Summary
Traditional TD3 algorithms are prone to fall into the problem of local optimization during iterative optimization, resulting in poor performance on the high-dimensional multi-objective problem.
A multi-mechanical collaborative adaptive evolution algorithm with mixed TD3 is proposed, which is optimized by initializing populations, reconstructing operations, merging populations, selecting elite individuals and random individuals to enter the improved TD3, and selecting excellent individuals through roulette until the termination condition is reached.
It effectively avoids the problem of TD3 algorithm falling into local optimality, improves the accuracy and convergence speed of the algorithm solution, and significantly improves the optimization speed and accuracy.
Smart Images
Figure CN119783777A_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the technical field of evolutionary algorithms, and in particular to a hybrid TD3 multi-mechanism collaborative adaptive evolutionary algorithm. Background Art
[0002] With the rapid development of science and technology, the engineering problems people face are becoming more and more complex, and solving problems through optimization algorithms is becoming more and more popular. Optimization algorithms have the characteristics of fast speed, strong adaptability, and self-learning, and have been favored by most researchers.
[0003] However, with the increasing scale of problems and high-dimensional parameter space, traditional optimization methods can no longer perform well on high-dimensional multi-objective problems. Traditional optimization algorithms often have the problem of blind search in multi-objective problems, which greatly increases the running time of the algorithm and reduces the search accuracy. In this context, the concept of reinforcement learning came into being. Through the self-learning characteristics of reinforcement learning, it can find the target area faster than traditional algorithms. However, when solving actual optimization problems, the TD3 algorithm is very likely to fall into the problem of local optimality during the iterative optimization process. Summary of the invention
[0004] The purpose of this application is to provide a hybrid TD3 multi-mechanism collaborative adaptive evolutionary algorithm, which solves the problem that traditional TD3 is very easy to fall into local optimality and improves the accuracy and convergence speed of the algorithm solution.
[0005] In order to achieve the above objectives, this application provides the following technical solutions:
[0006] A hybrid TD3 multi-mechanism collaborative adaptive evolutionary algorithm, including:
[0007] Initialize the population to obtain multiple independent individuals, then reconstruct each individual in the population to obtain a new individual, and merge the new population with the old population, which is a population reconstruction operation based on multi-mechanism collaboration;
[0008] Select elite individuals and some random individuals from the merged population and enter the improved TD3 for optimization, and return the optimized individuals and their corresponding objective function values, and then add the optimized individuals to the merged population to obtain a new merged population;
[0009] The roulette method is used to select excellent individuals from the new merged population for the next cycle (this part uses the offspring population generation strategy) until the termination condition is reached to obtain the final optimization result.
[0010] The algorithm includes a population reconstruction strategy based on multi-mechanism collaboration, a balance optimization strategy, a local search based on improved TD3, a child population generation strategy and an algorithm termination condition.
[0011] In some embodiments, the hybrid TD3 multi-mechanism collaborative adaptive evolutionary algorithm specifically includes:
[0012] S1: The initial population of size N obtained through initialization operation;
[0013] S2: Reconstruct the individuals in the generated initial population to obtain a new population;
[0014] S3: Select the top Elite individuals and A population of size M is composed of random individuals and put into TD3 for optimization;
[0015] S4: Based on the original TD3, the state space, action space, reward mechanism, state update and action generation strategy are modified, which is the local search based on the improved TD3;
[0016] S5: After TD3 parallel optimization, M new solutions are output. The M new solutions and the reconstructed 4N solutions are combined into a new population. The new population is first sorted according to the loss, and the top 10% of individuals are selected. Then, the roulette wheel method is used to select the remaining 90% of solutions according to the target value loss to form a sub-generation population of size N, that is, the sub-generation population generation strategy;
[0017] S6: The generated offspring population undergoes the reconstruction operation again and enters the next round of iteration, repeating S1-S5 until the termination condition is reached.
[0018] In some embodiments, the reconstruction operation includes swapping, perturbing, and simulating binary crossover.
[0019] In some embodiments, the formula for simulating binary crossover is as follows:
[0020]
[0021] Among them, the parent and Randomly select from the initial population and obtain offspring through simulated binary crossover and β is dynamically and randomly determined by the distribution factor η according to the following formula:
[0022]
[0023] Among them, η is a custom parameter.
[0024] In some embodiments, the exchange operation includes selecting two parent individuals in the initial population and exchanging n elements therein, where the parameter n is a custom parameter.
[0025] In some embodiments, the perturbation operation includes changing n random elements in each individual in the initial population within a value range, and the parameter n is the same as the parameter in the exchange operation.
[0026] In some embodiments, the state space state is composed of the parameter vector x, the current value X t (i) With the previous value X t-1 (i) Make the difference value X t (i)-X t-1 (i) Previous generation value X t-1 (i) and its previous value X t-2 (i) Make the difference value X t-1 (i)-X t-2 (i) Composition, wherein X t (i)-X t-1 (i) X t-1 (i)-X t-2 The dimension of (i) is related to the dimension of x, and the parameters of state are defined as follows:
[0027]
[0028] Among them, the action space parameter is the new step length Δx, and the step length is the difference between the current generation x and the previous generation x multiplied by the action action. The formula is as follows:
[0029] Δx=[Δx1,Δx2,...,Δx n ]
[0030] Δx i =(x t (i)-x t-1 (i))*action i
[0031] Among them, x t (i) and x t-1 (i) is the i-th element in the t-th generation and the t-1-th generation, action i is the i-th element in action action, and its range is [0,1];
[0032] Then the action space parameter is action = [action1, action2, ...action n ].
[0033] In some embodiments, the reward value is determined based on the target loss, by comparing the current loss t Compared with the previous generation loss t-1, determine whether the reward value is positive or negative. If the current loss is less than the historical loss, the reward value is +1, otherwise it is -1;
[0034]
[0035] After the action is obtained through the TD3 network, the parameter vector x is updated, thereby updating the state space and obtaining a new state;
[0036] Among them, x i The calculation formula is as follows:
[0037] x i (t) = x i (t-1)+Δx i
[0038] Among them, t is the current iteration number, x i (t) is x at the current iteration number i Value, Δx i They are the corresponding step length Δx under the current number of iterations. i .
[0039] In some embodiments, the termination condition is: if the target value does not improve within a preset k generations, the iteration is stopped directly.
[0040] In some embodiments, the stopping criterion is adaptively changed, and is determined by the difference f2 between the minimum f1 of the previous iteration and the current minimum value divided by the minimum f1 of the previous iteration, and the formula is as follows:
[0041]
[0042] If δ is greater than 6%, it is determined to be a valid iteration, and the current minimum value is set as the current optimal target value to continue the next round of iteration. Otherwise, it is an invalid iteration. If there is an improvement result for more than k times, the program is terminated. This is the termination condition of the algorithm.
[0043] Compared with the prior art, the hybrid TD3 multi-mechanism collaborative adaptive evolutionary algorithm provided in this application has the following beneficial effects:
[0044] This application combines the advantages of population evolution and TD3 algorithms to help TD3 escape from local optimality and accelerate the convergence speed of population individuals, and greatly improves the optimization speed and accuracy on the original basis.
[0045] This application combines evolutionary algorithms with reinforcement learning, introduces concepts such as population, crossover, and mutation into the TD3 algorithm, and greatly increases the diversity of solutions through the operation of evolutionary algorithms. It solves the problem of TD3 algorithm relying on the initial solution. The original TD3 algorithm is also prone to falling into the local optimal value problem. Through solution and reconstruction operations, the diversity of solutions is increased, which plays a great role in helping TD3 jump out of the local optimal value.
[0046] During the TD3 training process, this algorithm uses an online training mode, that is, the strategy is updated when the sample size reaches the specified batch. This not only avoids the redundant operations of training first and then testing, but also greatly shortens the optimization time.
[0047] The same problem was selected for testing, and the time used by the algorithm based on this application and the final optimization result were calculated. The results showed that this application can find the optimal value faster than the original TD3 algorithm, and the final result is better than the original TD3 algorithm, with the effect improved by up to 80.95%. In addition, this algorithm greatly shortens the time compared to the original method.
[0048] Furthermore, by applying the algorithm of the present application to a nuclear power steam system, the iterative results under different parameter settings are as follows:
[0049] (1) A multi-mechanism collaborative adaptive evolutionary algorithm hybridized with TD3, with a population of 80, 40 iterations of the outer loop, 500 iterations in TD3, and a training + model application time of 296.8 seconds, with an average optimal value of 2.39169271;
[0050] (2) Particle swarm optimization (PSO), population 80, outer 1000 iterations, time 159 seconds, average optimal value 160.98788074;
[0051] (3) Twin Delayed Deep Deterministic Policy Gradient Adaptive Intelligent Optimization Algorithm (TD3PSO), TD3 offline training 1000 times, taking 249508 seconds; the pre-trained model is applied online, the PSO population is 80, the number of optimization iterations is 1000 times, taking 230 seconds, and the average optimal value is 6.51008127.
[0052] The algorithm of the present application is compared with the existing algorithms. The iteration curves of the above three algorithms in terms of the number of iterations and running time are shown as follows: Figure 3-6 As shown in the figure, it can be seen from the calculation results that the convergence speed and average optimal loss value of the adaptive algorithm are better than those of PSO and TD3PSO. BRIEF DESCRIPTION OF THE DRAWINGS
[0053] In order to more clearly illustrate the technical solution of the present application, the following is a brief introduction to the drawings required for the technical description.
[0054] Figure 1 Flow chart of the multi-mechanism collaborative adaptive evolutionary algorithm of hybrid TD3 provided in this application;
[0055] Figure 2 A schematic diagram of the optimization process of the multi-mechanism collaborative adaptive evolutionary algorithm of the hybrid TD3 provided in this application;
[0056] Figure 3 A schematic diagram of an iteration curve of the multi-mechanism collaborative adaptive evolutionary algorithm of the hybrid TD3 provided in the present application on the vertical axis 0-1000;
[0057] Figure 4 A schematic diagram of an iteration curve of the multi-mechanism collaborative adaptive evolutionary algorithm of the hybrid TD3 provided in this application on the vertical axis 0-400;
[0058] Figure 5 A schematic diagram of an iteration time curve of the multi-mechanism collaborative adaptive evolutionary algorithm of the hybrid TD3 provided in the present application on the vertical axis 0-1000;
[0059] Figure 6 A schematic diagram of an iteration time curve of the multi-mechanism collaborative adaptive evolutionary algorithm of the hybrid TD3 provided in this application on the vertical axis 0-200. DETAILED DESCRIPTION
[0060] The following is further explained in detail through specific implementation methods.
[0061] like Figure 1 and Figure 2 As shown, the present application provides a hybrid TD3 multi-mechanism collaborative adaptive evolutionary algorithm, which introduces a population based on the original TD3 (Twin Delayed Deep Deterministic policy gradient), including:
[0062] Initialize the population to obtain multiple independent individuals, then perform reconstruction operations on each individual in the population, including exchange, perturbation and SBX (simulated binary crossover) operations, to obtain new individuals, and merge the new population with the old population to improve the diversity of the population;
[0063] Randomly select some individuals from the merged population and enter them into the improved TD3 for optimization, and return the optimized individuals and their corresponding objective function values, and then add the optimized individuals to the merged population to obtain a new merged population;
[0064] The roulette method is used to select excellent individuals from the new merged population for the next cycle until the termination condition is reached to obtain the final optimization result.
[0065] This algorithm can greatly expand the search space based on the original TD3 algorithm, improve the search efficiency and accuracy, and can effectively avoid the blind search of traditional methods and the defects of the original TD3 algorithm that easily fall into local optimality. At the same time, it can also adjust the accuracy of the algorithm according to the size of the initial population and the number of individuals passed into the improved TD3.
[0066] In one embodiment, the specific implementation method of the algorithm is as follows:
[0067] S1: Through the initialization operation, an initial population of size N is obtained, which solves the problem that the original TD3 is very easy to fall into the local optimum.
[0068] Among them, N is adaptively generated according to the dimension of vector x. When the dimension of the decision variable is less than 60, the population size is equal to the dimension of the decision variable. Otherwise, the minimum value of 200 and 0.8 times the dimension of the decision variable is taken.
[0069] S2: Perform reconstruction operations on the individuals in the generated initial population to obtain a new population, which includes three operations: exchange, perturbation, and SBX (simulated binary crossover). The formula of SBX is as follows:
[0070]
[0071] Among them, the parent and Randomly select from the initial population and obtain offspring through SBX crossover and β is dynamically and randomly determined by the distribution factor η according to the following formula:
[0072]
[0073] Among them, η is a custom parameter. The larger the η value, the greater the probability that the offspring individuals are close to the parent individuals.
[0074] The swap operation is to select two parent individuals in the initial population and swap n elements between them. The parameter n is a custom parameter. The perturbation operation is to change the n random elements in each individual in the initial population within its value range. Here, the parameter n is the same as the parameter in the swap operation.
[0075] After these three operations, a new population will be obtained respectively. Then the three new populations will be merged with the initial population to obtain a large population with a size of 4N.
[0076] S3: Randomly select M solutions from the new population after the reconstruction operation and put them into TD3 for optimization. Among them, the first 10% of the individuals come from the optimal solution of the merged population, and the remaining 90% of the individuals are randomly selected from the remaining population using the roulette method.
[0077] S4: Based on the original TD3, its state space, action space, reward mechanism, state update and action generation strategy were modified.
[0078] In this algorithm, the state space state is composed of the parameter vector x, the current value X t (i) With the previous value X t-1 (i) Make the difference value X t (i)-X t-1 (i) Previous generation value X t-1 (i) and its previous value X t-2 (i) Make the difference value X t-1 (i)-X t-2 (i) Composition. Wherein, X t (i)-X t-1 (i) X t-1 (i)-X t-2 The dimension of (i) is related to the dimension of x. The parameters of state are defined as follows:
[0079]
[0080] Among them, the action space parameter is the new step size Δx, and the step size is the difference between the current generation x and the previous generation x multiplied by the action action (when updating for the first time, the step size is expressed by the difference between the initial solution and the lower or upper limit of each parameter multiplied by action), and the formula is as follows:
[0081] Δx=[Δx1,Δx2,...,Δx n ]
[0082] Δx i =(x t (i)-x t-1 (i))*action i
[0083] Among them, x t (i) and x t-1 (i) is the i-th element in the t-th generation and the t-1-th generation, action i is the i-th element in action action, and its range is [0,1].
[0084] Therefore, the action space parameter is action = [action1, action2, ...action n ].
[0085] In addition, when generating actions, in order to prevent falling into local optimality, a corresponding probability is set to decide whether to select actions randomly or use existing actions. At the beginning of training, the model has not yet fully learned the environment information, so it needs to actively try a variety of options. In the later stages of training, the model needs to tend to use existing knowledge and experience.
[0086] In addition, the reward value is determined according to the target value loss, by comparing the contemporary loss t Compared with the previous generation loss t-1 , to determine the positive or negative of the reward value. If the contemporary loss is less than the historical loss, the reward value is +1, otherwise it is -1.
[0087]
[0088] After the action is obtained through the TD3 network, the parameter vector x should also be updated accordingly, thereby updating the state space and obtaining a new state.
[0089] In this method, x i The value of is related to the independent variable step length Δx, and its calculation formula is as follows:
[0090] x i (t) = x i (t-1)+Δx i
[0091] Among them, t is the current iteration number, x i (t) is x at the current iteration number i Value, Δx i They are the corresponding step length Δx under the current number of iterations. i The optimization process of TD3 is as follows Figure 2 shown.
[0092] S5: After TD3 parallel optimization, M new solutions will be output. These M new solutions and the reconstructed 4N solutions will form a new population. The new population will be sorted by loss first, and the top 10% of individuals will be selected. Then, the roulette wheel method will be used to select the remaining 90% solutions according to the target value loss to form a sub-generation population of size N. The selection strategy of the roulette wheel is that the smaller the loss value, the greater the probability of being selected.
[0093] S6: The generated offspring population is reconstructed again and enters the next round of iteration, repeating steps S1-S5 until the termination condition is reached.
[0094] The termination condition in this algorithm is that if the target value does not become better within the preset k generations, the iteration is stopped directly. The stopping criterion is adaptive, which is determined by the difference f2 between the minimum f1 of the previous iteration and the current minimum value divided by the minimum f1 of the previous iteration. The formula is as follows:
[0095]
[0096] If δ is greater than 6%, it is considered as a valid iteration, and the current minimum value is set as the current optimal target value to continue the next round of iteration. Otherwise, it is considered as an invalid iteration. If there is an improvement result for more than k times, the program is terminated.
[0097] The advantage of this setting is that it can avoid invalid iterations and shorten the optimization time.
[0098] In one embodiment, the algorithm provided by the present application is applied to a nuclear power steam system, and the iterative results under different parameter settings are as follows:
[0099] (1) A multi-mechanism collaborative adaptive evolutionary algorithm hybridized with TD3, with a population of 80, 40 iterations of the outer loop, 500 iterations in TD3, and a training + model application time of 296.8 seconds, with an average optimal value of 2.39169271;
[0100] (2) Particle swarm optimization (PSO), population 80, outer 1000 iterations, time 159 seconds, average optimal value 160.98788074;
[0101] (3) Twin Delayed Deep Deterministic Policy Gradient Adaptive Intelligent Optimization Algorithm (TD3PSO), TD3 offline training 1000 times, taking 249508 seconds; the pre-trained model is applied online, the PSO population is 80, the number of optimization iterations is 1000 times, taking 230 seconds, and the average optimal value is 6.51008127.
[0102] The iteration curves of the above three algorithms in terms of the number of iterations and running time are as follows: Figure 3-6 As shown, the blue curve represents a hybrid TD3 multi-mechanism collaborative adaptive evolutionary algorithm, the red curve represents the particle swarm algorithm, and the green curve represents the adaptive intelligent optimization algorithm of the twin delayed deep deterministic policy gradient. By comparing the three algorithms, it can be seen from the calculation results that the convergence speed and average optimal loss value of the adaptive algorithm are better than PSO and TD3PSO.
[0103] The above description is only a specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed in the present application should be covered within the protection scope of the present application.
Claims
1. A multi-mechanism collaborative adaptive evolutionary algorithm hybrid TD3, characterized in that: include: Initialize the population to obtain multiple independent individuals, then reconstruct each individual in the population to obtain new individuals, and merge the new population with the old population; Randomly select some individuals from the merged population and enter them into the improved TD3 for optimization, and return the optimized individuals and their corresponding objective function values, and then add the optimized individuals to the merged population to obtain a new merged population; The roulette method is used to select excellent individuals from the new merged population for the next cycle until the termination condition is reached to obtain the final optimization result.
2. The multi-mechanism collaborative adaptive evolutionary algorithm of hybrid TD3 according to claim 1 is characterized in that: The algorithm specifically includes: S1: The initial population of size N obtained through initialization operation; S2: Reconstruct the individuals in the generated initial population to obtain a new population; S3: Randomly select M solutions from the new population after the reconstruction operation and put them into TD3 for optimization; S4: Based on the original TD3, modify the state space, action space, reward mechanism, state update and action generation strategy; S5: After TD3 parallel optimization, M new solutions are output. The M new solutions and the reconstructed 4N solutions are combined into a new population. The new population is first sorted according to the loss, and the top 10% of individuals are selected. Then, the roulette wheel method is used to select the remaining 90% solutions according to the target value loss to form a sub-generation population of size N. S6: The generated offspring population undergoes the reconstruction operation again and enters the next round of iteration, repeating S1-S5 until the termination condition is reached.
3. The multi-mechanism collaborative adaptive evolutionary algorithm of hybrid TD3 according to claim 2 is characterized in that: In S2, the reconstruction operations include swapping, perturbation, and,simulating binary crossover.
4. The multi-mechanism collaborative adaptive evolutionary algorithm of hybrid TD3 according to claim 3 is characterized in that: The formula for simulating binary crossover is as follows: Among them, the parent and Randomly select from the initial population and obtain offspring through simulated binary crossover and β is dynamically and randomly determined by the distribution factor η according to the following formula: Among them, η is a custom parameter.
5. The multi-mechanism collaborative adaptive evolutionary algorithm of hybrid TD3 according to claim 3 is characterized in that: The exchange operation involves selecting two parent individuals in the initial population and exchanging n elements therein, where the parameter n is a custom parameter.
6. The multi-mechanism collaborative adaptive evolutionary algorithm of hybrid TD3 according to claim 3 is characterized in that: The perturbation operation involves changing n random elements in each individual in the initial population within their value range, and the parameter n is the same as the parameter in the exchange operation.
7. The multi-mechanism collaborative adaptive evolutionary algorithm of hybrid TD3 according to claim 2 is characterized in that: In S4, the state space state consists of the parameter vector x, the current value X t (i) With the previous value X t-1 (i) Make the difference value X t (i)-X t-1 (i) Previous generation value X t-1 (i) and its previous value X t-2 (i) Make the difference value X t-1 (i)-X t-2 (i) Composition, wherein X t (i)-X t-1 (i) X t-1 (i)-X t-2 The dimension of (i) is related to the dimension of x, and the parameters of state are defined as follows: Among them, the action space parameter is the new step length Δx, and the step length is the difference between the current generation x and the previous generation x multiplied by the action action. The formula is as follows: Δx=[Δx1,Δx2,...,Δx n ] Δx i =(x t (i)-x t-1 (i))*action i Among them, x t (i) and x t-1 (i) is the i-th element in the t-th generation and the t-1-th generation, action i is the i-th element in action action, and its range is [0,1]; Then the action space parameter is action = [action1, action2, ...action n ].
8. The multi-mechanism collaborative adaptive evolutionary algorithm of hybrid TD3 according to claim 7 is characterized in that: The reward value is determined based on the target loss, and is calculated by comparing the current loss t Compared with the previous generation loss t-1 , determine whether the reward value is positive or negative. If the current loss is less than the historical loss, the reward value is +1, otherwise it is -1; After the action is obtained through the TD3 network, the parameter vector x is updated, thereby updating the state space and obtaining a new state; Among them, x i The calculation formula is as follows: x i (t)=x i (t-1)+Δx i Among them, t is the current iteration number, x i (t) is x at the current iteration number i Value, Δx i They are the corresponding step length Δx under the current number of iterations. i .
9. The multi-mechanism collaborative adaptive evolutionary algorithm of hybrid TD3 according to claim 2 is characterized in that: In S6, the termination condition is: if the target value does not improve within the preset k generations, the iteration is stopped directly.
10. The multi-mechanism collaborative adaptive evolutionary algorithm of hybrid TD3 according to claim 2 is characterized in that: In S6, the stopping criterion is adaptively changed, and is determined by the difference f2 between the minimum f1 of the previous iteration and the current minimum value divided by the minimum f1 of the previous iteration. The formula is as follows: If δ is greater than 6%, it is considered as a valid iteration, and the current minimum value is set as the current optimal target value to continue the next round of iteration. Otherwise, it is considered as an invalid iteration. If there is an improvement result for more than k times, the program is terminated.