A method for training intelligent agents based on differential evolution

Abstract

The present invention relates to an intelligent agent training method based on differential evolution, comprising: randomly initializing Q table parameters to obtain a plurality of Q tables, wherein the plurality of Q tables constitute an initial evolution population, wherein a Q table is used as an individual in the population; performing differential evolution operations on the initial evolution population to obtain a new generation of Q table population, and using the new generation of Q table population as the initial evolution population of the next generation to repeat the differential evolution operations until a preset evolution condition is reached to obtain a final Q table population; using a preset fitness function to evaluate each individual in the final Q table population, and outputting the optimal individual; performing reinforcement learning training on the optimal individual to obtain an intelligent agent. The intelligent agent training method of the present invention firstly uses an evolutionary algorithm to obtain an optimal Q table, and then performs reinforcement learning training based on the optimal Q table, thereby accelerating the convergence speed in the intelligent agent training process, reducing the time resources required for training, and improving the immediacy of the intelligent agent training.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and in particular relates to an intelligent agent training method based on differential evolution. Background Art

[0002] As an important branch of artificial intelligence, reinforcement learning (RL) has been highly recognized by scholars and has been applied to many fields. In solving sequential decision-making problems, reinforcement learning uses the Markov decision process to enable the agent to make decisions on the current state using the sample set obtained by interacting with the environment.

[0003] Maze problems have always been a classic problem that has been discussed in the fields of graphics and data structures. Solving complex maze problems can realize the reasonable path planning problem of intelligent agents. Generally speaking, a maze problem refers to an intelligent agent walking from the entrance of a maze to the exit of a maze in a certain way, and the steps and time consumed are as small as possible. The traditional methods for solving maze problems mainly include three categories: 1) depth-first traversal. 2) breadth-first traversal. 3) intelligent algorithm solution. Depth-first traversal and breadth-first traversal, as classic data structure solution methods, have good applicability for solving maze problems. However, when the maze problem becomes increasingly complex, sampling the above two methods will increase the consumption of time resources. The use of intelligent algorithms to solve the problem has improved the executability of the algorithm to a certain extent and effectively reduced the time complexity of the algorithm.

[0004] As the problem becomes more and more in-depth, the maze environment becomes more and more complex. Traditional methods may not be able to fully consider all situations, such as random changes in some areas of the maze, trap areas and other uncertain interference factors. Therefore, reinforcement learning is used to solve the maze problem. Q learning, as a classic model-free reinforcement learning method, is widely used in maze problems. However, the current Q learning method has a slow convergence speed, a long training time and poor timeliness during the training of intelligent agents. Summary of the invention

[0005] In order to solve the above problems existing in the prior art, the present invention provides an agent training method based on differential evolution. The technical problem to be solved by the present invention is achieved through the following technical solutions:

[0006] The present invention provides an intelligent agent training method based on differential evolution, comprising:

[0007] Randomly initialize the Q-table parameters to obtain several Q-tables, which constitute the initial evolution population, where one Q-table is an individual in the population;

[0008] Performing a differential evolution operation on the initial evolution population to obtain a new generation Q table population, and repeating the differential evolution operation with the new generation Q table population as the initial evolution population of the next generation until a preset evolution condition is reached to obtain a final Q table population;

[0009] Using a preset fitness function to evaluate each individual in the final Q-table population, and outputting the optimal individual;

[0010] The optimal individual is subjected to reinforcement learning training to obtain an intelligent agent.

[0011] In one embodiment of the present invention, the Q table parameters include state parameters and action parameters.

[0012] In one embodiment of the present invention, the differential evolution operation includes:

[0013] Using a mutation operator to mutate the initial population to obtain a mutant population;

[0014] Performing a crossover operation on the mutant population using a binomial crossover method to obtain a crossover population;

[0015] The fitness value of each individual in the crossover population is compared with the fitness value of the corresponding individual in its initial evolution population, and the better individual is selected according to the comparison result to form the new generation Q table population.

[0016] In one embodiment of the present invention, each individual in the initial evolution population is subjected to mutation processing according to the following formula to obtain a mutation population v:

[0017]

[0018] Where g represents the evolutionary generation of the population, i represents the individual index value in the population, i=1,2,...,NP, NP represents the number of individuals in the population, r1,r2,r3 represent the index values ​​randomly sampled from NP individuals, and r1≠r2≠r3, F represents the scaling factor, F∈[0,1], represents the r1th individual in the population x of the gth generation of evolution, Represents the i-th individual in the mutant population v of the evolutionary g-th generation.

[0019] In one embodiment of the present invention, a crossover operation is performed on each individual in the variant population according to the following formula to obtain a crossover population u:

[0020]

[0021] Where CR represents the crossover probability, CR∈[0,1], rand is a random number between [0,1], jrand is a random integer between [0,D], and D represents the dimension of the problem. represents the jth dimension of the i-th individual in the crossover population u in the g-th generation of evolution, represents the jth dimension of the i-th individual in the g-th generation population x, represents the jth dimension of the i-th individual in the mutant population v of the g-th generation of evolution.

[0022] In one embodiment of the present invention, the following selection method is used to select better individuals to form the new generation Q table population:

[0023]

[0024] Where f represents the fitness function, It represents the fitness value of the i-th individual in the initial evolution population x when the differential evolution reaches the g-th generation. It represents the fitness value of the i-th individual in the crossover population u when the differential evolution reaches the g-th generation. Represents the i-th individual in the new generation Q-table population.

[0025] In one embodiment of the present invention, the fitness function is based on the number of target state hits under a specified step size:

[0026]

[0027] in, Represents the fitness value of the i-th individual in the t-th generation.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] The intelligent agent training method based on differential evolution of the present invention first uses an evolutionary algorithm to obtain an optimal Q table, and then performs reinforcement learning training based on the optimal Q table. In the process of intelligent agent training, the convergence speed is accelerated, the time resources required for training are reduced, and the immediacy of intelligent agent training is improved.

[0030] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the following specifically cites a preferred embodiment and describes it in detail with the accompanying drawings as follows. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 is a schematic diagram of an intelligent agent training method based on differential evolution provided by an embodiment of the present invention;

[0032] Figure 2 It is a diagram of the creation process of an evolutionary individual provided by an embodiment of the present invention;

[0033] Figure 3 It is a Q learning optimization structure diagram based on DE provided by an embodiment of the present invention;

[0034] Figure 4 It is a schematic diagram of a complex maze environment provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0035] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined purpose of the invention, the following is a detailed description of an intelligent agent training method based on differential evolution proposed in accordance with the present invention in combination with the accompanying drawings and specific implementation methods.

[0036] The above and other technical contents, features and effects of the present invention are clearly presented in the following detailed description of the specific implementation modes in conjunction with the accompanying drawings. Through the description of the specific implementation modes, the technical means and effects adopted by the present invention to achieve the predetermined purpose can be more deeply and specifically understood. However, the attached drawings are only for reference and explanation purposes and are not used to limit the technical solutions of the present invention.

[0037] Embodiment 1

[0038] See also Figure 1 and Figure 3 , Figure 1 is a schematic diagram of an intelligent agent training method based on differential evolution provided by an embodiment of the present invention, Figure 3 : is a Q learning optimization structure diagram based on DE provided by an embodiment of the present invention. As shown in the figure, the intelligent agent training method based on differential evolution of this embodiment includes:

[0039] S1: Randomly initialize the Q-table parameters to obtain several Q-tables, which constitute the initial evolution population, where a Q-table is an individual in the population;

[0040] In this embodiment, the Q table parameters include state parameters and action parameters. Traditional Q table initialization is usually a full 0 matrix, so there is a large blindness in the early stage of model exploration, which leads to the failure of Q learning to converge quickly. In this embodiment, a Q table is regarded as an individual in the population, and several Q tables are randomly initialized to form an evolutionary population NP. During the evolution process, a specific evaluation method is used to retain excellent individuals. The best combination of state-action values ​​in the better Q table individuals will be found as the initialization information of the Q table, such as Figure 2 As shown, Figure 2 It is a diagram of the creation process of an evolutionary individual provided by an embodiment of the present invention.

[0041] Taking the complex maze problem to be solved in this embodiment as an example, the randomly initialized Q table parameters include: the Q value of the action taken by the agent in each state, where the state parameter includes the feasible area of ​​the maze problem to be solved, and each feasible area represents a state S. The action parameter is the state transition operation that the agent can implement in the current state, that is, up, down, left, and right.

[0042] It should be noted that in the random initialization process, the population P is composed of NP solution vectors randomly sampled from the feasible solution space, where the population of the gth generation can be expressed as Where NP represents the population size, D represents the dimension of the problem, represents the i-th individual in the g-th generation of the population. The way DE (Deferential evolution) generates individuals is as follows:

[0043]

[0044] Among them, ub and lb represent the upper bound and lower bound of the problem domain respectively, and NP individuals are generated in the feasible solution space by random initialization.

[0045] S2: Perform differential evolution operation on the initial evolution population to obtain a new generation of Q-table population, and repeat the differential evolution operation with the new generation of Q-table population as the initial evolution population of the next generation until the preset evolution conditions are met to obtain the final Q-table population;

[0046] Specifically, the differential evolution operation includes:

[0047] Step 1: Use the mutation operator to mutate the initial population to obtain the mutant population;

[0048] Specifically, in this embodiment, each individual in the initial evolution population is mutated according to formula (2) to obtain a mutant population v,

[0049]

[0050] Where g represents the evolutionary generation of the population, i represents the individual index value in the population, i=1,2,...,NP, NP represents the number of individuals in the population, r1,r2,r3 represent the index values ​​randomly sampled from NP individuals, and r1≠r2≠r3, F represents the scaling factor, F∈[0,1], represents the r1th individual in the population x of the gth generation of evolution, V i g Represents the i-th individual in the mutant population v of the evolutionary g-th generation.

[0051] Step 2: Use binomial crossover to perform crossover operation on the mutant population to obtain a crossover population;

[0052] Specifically, in this embodiment, a crossover operation is performed on each individual in the variant population according to formula (3) to obtain a crossover population u,

[0053]

[0054] Where CR represents the crossover probability, CR∈[0,1], rand is a random number between [0,1], jrand is a random integer between [0,D], and D represents the dimension of the problem. represents the jth dimension of the i-th individual in the crossover population u in the g-th generation of evolution, represents the jth dimension of the i-th individual in the g-th generation population x, represents the jth dimension of the i-th individual in the mutant population v of the g-th generation of evolution.

[0055] In this embodiment, jrand selects a random integer between [0, D] to ensure that the test vector At least one decision variable differs from the corresponding target vector.

[0056] Step 3: Compare the fitness value of each individual in the crossover population with the fitness value of the corresponding individual in its initial evolution population, and select better individuals based on the comparison results to form a new generation of Q table population.

[0057] Specifically, in this embodiment, according to the selection method shown in formula (4), better individuals are selected to form a new generation of Q table population.

[0058]

[0059] Where f represents the fitness function, It represents the fitness value of the i-th individual in the initial evolution population x when the differential evolution reaches the g-th generation. It represents the fitness value of the i-th individual in the crossover population u when the differential evolution reaches the g-th generation. Represents the i-th individual in the new generation Q-table population.

[0060] It should be noted that, in this embodiment, when the preset evolutionary number is reached, the evolution is stopped to obtain the final Q-table population.

[0061] S3: Use the preset fitness function to evaluate each individual in the final Q-table population and output the optimal individual;

[0062] In this embodiment, the optimal individual is the optimal Q table in the final Q table population.

[0063] For solving complex maze problems, the intuitive evaluation standard is the number of times the maze exit is found. If the number of times the agent reaches the terminal state is large under the specified step length, it means that the Q table at this time has good environmental information. Therefore, for complex maze problems, in this embodiment, the fitness function is based on the number of target state hits under the specified step length, that is:

[0064]

[0065] in, Represents the fitness value of the i-th individual in the t-th generation.

[0066] It should be noted that during the differential evolution operation, the fitness value of each individual in the comparison crossover population is also calculated using the above fitness function.

[0067] It is worth noting that in evolutionary algorithms, the setting of the fitness function depends on the goal of the optimization problem. For the setting of the fitness function of the evolutionary algorithm, specific settings should be made according to different problems to be solved. Q learning and DE (Deferential evolution) are only tools and methods for solving problems. How to evaluate the evolution of individuals in a population should be set according to the specific problems to be solved, and no restrictions are made here.

[0068] S4: Perform reinforcement learning training on the optimal individual to obtain an intelligent agent.

[0069] In this embodiment, the optimal individual is used as the initial model input of Q learning, and the optimal strategy is found through training. The specific training method is consistent with the existing Q learning training method and will not be repeated here.

[0070] The intelligent agent training method based on differential evolution of this embodiment first uses an evolutionary algorithm to obtain the optimal Q table, and then performs reinforcement learning training based on the optimal Q table. In the process of intelligent agent training, the convergence speed is accelerated, the time resources required for training are reduced, and the immediacy of intelligent agent training is improved.

[0071] It should be noted that the intelligent agent training method based on differential evolution in this embodiment can be applied not only to complex maze problems, but also to any model-free problem with a finite Q table. It has good robustness, and the setting of the evolutionary algorithm is not limited to the differential evolution algorithm.

[0072] Embodiment 2

[0073] This embodiment uses a specific case to verify the effect of the agent training method based on differential evolution in Embodiment 1. Figure 4 , Figure 4This is a schematic diagram of a complex maze environment provided by an embodiment of the present invention. As shown in the figure, the square frame is set as a wall, representing an unreachable state, the empty area in the lower left corner is the starting point of the maze, that is, the initial position of the agent, and the corresponding empty area in the upper right corner is the maze exit, representing the terminal state. There are 376 reachable states in the model. The reward value R of the agent in the remaining positions except the terminal state is 0. When reaching the terminal state, the reward R is set to 100.

[0074] The purpose of setting the feedback of the model to sparse rewards is to better highlight the efficiency of the present invention. When positive rewards only exist in the terminal state, the algorithm can only converge by repeatedly exploring the agent until the target state is reached and the Q table is continuously updated. In this embodiment, the proposed agent training method based on differential evolution is compared with the original Q learning and the other two improvements of Q learning (Double Q learning and SA-Q learning) in the test. The specific experimental parameters involved are as follows:

[0075] The parameters involved in the Q-learning algorithm are: learning rate α = 0.9; discount factor γ = 0.95, greed factor ε = 0.1. The main parameters involved in Double Q learning are the same as those in Q-learning. The main parameters involved in SA-Q learning are the same as those in Q-learning, and the cooling parameters used are set as follows: initial temperature Temperature = 100, attenuation parameter K = 0.95.

[0076] Table 1 gives the parameters and values ​​required for the DE-Q learning algorithm, where G represents the population evolution algebra; Table 2 shows the evaluation results of the convergence index, comparing the convergence performance of different algorithms through the mean and variance.

[0077] Table 1 Parameters involved in the DE-Q learning algorithm

[0078]

[0079]

[0080] Table 2 Convergence index test results

[0081]

[0082] From the results in 2, it can be seen that the DE-Q-learning algorithm has the best convergence when NP and G are 100, but overall, the convergence performance of DE-Q-learning is greatly improved compared with Q-learning, Double-Q-learning and SA-Q-learning, and DE-Q-learning also has a certain improvement in convergence performance as G and NP increase.

[0083] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants are intended to cover non-exclusive inclusion, so that an article or device including a series of elements includes not only those elements, but also other elements that are not explicitly listed. In the absence of more restrictions, the elements defined by the statement "including one..." do not exclude the existence of other identical elements in the article or device including the elements.

[0084] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. An agent training method based on differential evolution, characterized in that: include: Randomly initialize the Q table parameters to obtain several Q tables, which constitute the initial evolution population, wherein one Q table is an individual in the population; the Q table parameters include state parameters and action parameters. For complex maze problems, the randomly initialized Q table parameters include: the Q value of the action taken by the agent in each state, wherein the state parameters include the feasible area of ​​the maze problem to be solved, each feasible area represents a state S, and the action parameters are the state transfer operations that the agent can implement in the current state, including up, down, left and right; Performing a differential evolution operation on the initial evolution population to obtain a new generation Q table population, and repeating the differential evolution operation with the new generation Q table population as the initial evolution population of the next generation until a preset evolution condition is reached to obtain a final Q table population; Using a preset fitness function to evaluate each individual in the final Q-table population, and outputting the optimal individual; The optimal individual is subjected to reinforcement learning training to obtain an intelligent agent, and the intelligent agent is used to realize reasonable path planning for complex maze problems.

2. The agent training method based on differential evolution according to claim 1, characterized in that: The differential evolution operation includes: Using a mutation operator to mutate the initial population to obtain a mutant population; Performing a crossover operation on the mutant population using a binomial crossover method to obtain a crossover population; The fitness value of each individual in the crossover population is compared with the fitness value of the corresponding individual in its initial evolution population, and the better individual is selected according to the comparison result to form the new generation Q table population.

3. The agent training method based on differential evolution according to claim 1, characterized in that: Each individual in the initial evolution population is mutated according to the following formula to obtain a mutant population v: Among them, g represents the evolutionary generation of the population, i represents the individual index value in the population, i = 1, 2, ..., NP, NP represents the number of individuals in the population, r1, r 2， r3 represents the index value randomly sampled from NP individuals, and r1≠r2≠r3, F represents the scaling factor, F∈[0,1], represents the r1th individual in the population x of the gth generation of evolution, V i g Represents the i-th individual in the mutant population v of the evolutionary g-th generation.

4. The agent training method based on differential evolution according to claim 3, characterized in that: Perform a crossover operation on each individual in the variant population according to the following formula to obtain a crossover population u: Where CR represents the crossover probability, CR∈[0,1], rand is a random number between [0,1], jrand is a random integer between [0,D], and D represents the dimension of the problem. represents the jth dimension of the i-th individual in the crossover population u in the g-th generation of evolution, represents the jth dimension of the i-th individual in the g-th generation population x, represents the jth dimension of the i-th individual in the mutant population v of the g-th generation of evolution.

5. The agent training method based on differential evolution according to claim 4, characterized in that: According to the following selection method, better individuals are selected to form the new generation of Q table population. Where f represents the fitness function, It represents the fitness value of the i-th individual in the initial evolution population x when the differential evolution reaches the g-th generation. It represents the fitness value of the i-th individual in the crossover population u when the differential evolution reaches the g-th generation. Represents the i-th individual in the new generation Q-table population.

6. The agent training method based on differential evolution according to claim 3, characterized in that: The fitness function is based on the number of target state hits under a specified step size: in, i∈{1,2,...,NP} , Represents the fitness value of the i-th individual in the t-th generation.

Images