Reinforcement learning hyper-parameter optimization method and device and computer readable storage medium

Abstract

The invention discloses a reinforcement learning hyper-parameter optimization method and device and a computer readable storage medium. The method comprises the following steps: initializing a group; in the evolutionary process of any generation, performing reinforcement learning training on each individual in the group of the generation, and then calculating the fitness value of the generation of each individual; adjusting the hyper-parameter data corresponding to the elite individuals of the generation by adopting the hyper-parameter data corresponding to the optimal fitness value of the inferior individuals of the generation in the evolutionary process of the generation, and taking the adjusted hyper-parameter data as the hyper-parameter data of the inferior individuals of the generation after the evolution of the generation; and after performing multi-generation evolution on each individual in the group, determining a target model from each individual according to the fitness values of the individuals in the group. According to the reinforcement learning hyper-parameter optimization scheme provided by the invention, the condition that the model pays attention to the performance in the short-term training and ignores the performance in the long-term training is avoided, so that the reinforcement learning hyper-parameter optimization effect is improved.

Description

Technical Field

[0001] The present invention relates to the field of reinforcement learning technology, and in particular to a reinforcement learning hyperparameter optimization method, device and computer-readable storage medium. Background Art

[0002] Reinforcement learning is one of the methods and methodologies of machine learning, which is used to describe and solve the problem of how an agent can maximize its rewards or achieve specific goals by learning strategies during its interaction with the environment. The PBT (Population Based Training) algorithm proposes a group-based automatic hyperparameter optimization solution. It optimizes the hyperparameter optimization by iteratively using the mutation methods of Exploit and Explore, and optimizes the learning rate hyperparameter in the reinforcement learning algorithm. However, in the process of using this algorithm, the hyperparameters of elite individuals are directly copied to replace the hyperparameters of inferior individuals to evolve the inferior individuals, which causes the model to focus on the performance in the short term of training and ignore the performance in the long term of training, resulting in poor optimization of the hyperparameters of reinforcement learning. For example, when reinforcement learning is applied to task planning and scheduling, the accuracy of the planning and scheduling will be low. Summary of the invention

[0003] The main purpose of the present invention is to provide a reinforcement learning hyperparameter optimization method, device and computer-readable storage medium, aiming to propose a reinforcement learning hyperparameter optimization scheme to avoid the model focusing on short-term training performance and ignoring long-term training performance, thereby improving the effect of reinforcement learning hyperparameter optimization.

[0004] To achieve the above object, the present invention provides a method for optimizing hyperparameters of reinforcement learning, the method comprising the following steps:

[0005] Initialize a group, each individual in the group is a reinforcement learning model with different hyperparameter data, and the hyperparameter data includes a parameter value corresponding to at least one hyperparameter;

[0006] In any generation of evolution, each individual in the group of this generation is subjected to reinforcement learning training, and the fitness value of each individual in this generation is calculated;

[0007] Determine the elite individuals and inferior individuals of this generation from each individual according to the fitness value of this generation;

[0008] Using the hyperparameter data corresponding to the optimal fitness value of the inferior individual of this generation in the evolution process of successive generations to adjust the hyperparameter data corresponding to the elite individual of this generation, and using the adjusted hyperparameter data as the hyperparameter data of the inferior individual of this generation after evolution in this generation;

[0009] After multiple generations of evolution for each individual in the group, a target model is determined from each individual according to the fitness value of the individual in the group.

[0010] Optionally, the reinforcement learning model uses task data as model input data and a task scheduling result as model output data, and the target model is used to perform planning and scheduling based on the task data to obtain the task scheduling result. The step of performing reinforcement learning training on each individual in the group of this generation includes:

[0011] For any target individual among the individuals, the task data of the target task in the preset training data set is input into the reinforcement learning model corresponding to the target individual for planning and scheduling, and the scheduling result of the target task is obtained;

[0012] Performing simulation on the target task according to the scheduling result to obtain the task achievement status of the target task;

[0013] An environmental reward is generated according to the task achievement status, and a reinforcement learning model corresponding to the target individual is optimized according to the environmental reward.

[0014] Optionally, the step of calculating the fitness value of each individual in this generation includes:

[0015] For any target individual among the individuals, the target individual is interacted with the environment multiple times to obtain multiple cumulative environmental reward expectations;

[0016] An average value of the multiple accumulated environmental reward expectations is determined as the current generation fitness value of the target individual, or a variance of the multiple accumulated environmental reward expectations is determined as the current generation fitness value of the target individual.

[0017] Optionally, after the step of determining elite individuals and inferior individuals of this generation from each individual according to the fitness value of this generation of each individual, the step further includes:

[0018] Calculating the utilization strategy probability corresponding to the disadvantaged individual of the current generation according to the current generation fitness value of the disadvantaged individual of the current generation, wherein the higher the current generation fitness value of the disadvantaged individual of the current generation, the higher the utilization strategy probability corresponding to the disadvantaged individual of the current generation;

[0019] Selecting a target strategy from at least two preset variant strategies according to the utilization strategy probability, wherein the at least two variant strategies include the utilization strategy, and the higher the utilization strategy probability, the higher the possibility that the utilization strategy is selected;

[0020] When the target strategy is the utilization strategy, the step of adjusting the hyperparameter data corresponding to the elite individuals of this generation by using the hyperparameter data corresponding to the optimal fitness values ​​of the inferior individuals of this generation in the evolutionary process of successive generations is performed.

[0021] Optionally, the at least two mutation strategies further include an exploration strategy, and after the step of selecting a target strategy from the at least two preset mutation strategies according to the utilization strategy probability, the step further includes:

[0022] When the target strategy is the exploration strategy, the probability distribution of the current generation corresponding to the at least one hyperparameter in the current generation evolution is obtained, and the hyperparameter data is obtained by sampling according to the current generation probability distribution, and the sampled hyperparameter data is used as the hyperparameter data of the disadvantaged individuals of the current generation after the evolution of the current generation.

[0023] Optionally, the step of initializing the group includes:

[0024] Initializing an initial probability distribution corresponding to the at least one hyperparameter;

[0025] According to the initial probability distribution, a plurality of different groups of the hyperparameter data are sampled, and each group of the hyperparameter data is used as the hyperparameter data of each reinforcement learning model in the group.

[0026] Optionally, the step of obtaining the probability distribution of the current generation corresponding to the at least one hyperparameter in the current generation evolution includes:

[0027] Obtaining a previous generation probability distribution corresponding to the at least one hyperparameter, wherein the previous generation probability distribution corresponding to the at least one hyperparameter in the first generation evolution process is the initial probability distribution;

[0028] The hyperparameter data of the elite individuals of the current generation are used to update the probability distribution of the previous generation, so as to obtain the probability distribution of the current generation corresponding to the at least one hyperparameter in the evolution of the current generation.

[0029] Optionally, when there are multiple elite individuals in the current generation, the step of using the hyperparameter data of the elite individuals in the current generation to update the probability distribution of the previous generation to obtain the probability distribution of the current generation corresponding to the at least one hyperparameter in the current generation evolution includes:

[0030] Normalizing the fitness values ​​of the current generation of each elite individual of the current generation to obtain the weights corresponding to each elite individual of the current generation;

[0031] For any target hyperparameter among the at least one hyperparameter, weighted summing is performed on the parameter values ​​corresponding to the target hyperparameter in the hyperparameter data of each elite individual of the current generation using the weight, and the mean parameter in the probability distribution of the current generation corresponding to the target hyperparameter in the current generation evolution is obtained according to the calculation result;

[0032] Subtract the old mean parameter from the parameter value corresponding to the target hyperparameter in the hyperparameter data of each elite individual of the current generation, and then calculate the square to obtain the intermediate calculation results corresponding to each elite individual of the current generation, wherein the old mean parameter is the mean parameter in the probability distribution of the previous generation corresponding to the target hyperparameter;

[0033] The intermediate calculation results are weightedly summed using the weights, and the variance parameter of the probability distribution of the current generation corresponding to the target hyperparameter in the current generation evolution is obtained according to the calculation results.

[0034] Optionally, the hyperparameter data includes at least parameter values ​​corresponding to one or more hyperparameters of the number of neural network layers, the number of neurons in each layer, the learning rate, the activation function type, the discount factor, and the size of the experience replay pool.

[0035] To achieve the above object, the present invention also provides a reinforcement learning hyperparameter optimization device, the device comprising:

[0036] An initialization module, used to initialize a group, each individual in the group is a reinforcement learning model with different hyperparameter data, and the hyperparameter data includes a parameter value corresponding to at least one hyperparameter;

[0037] A calculation module is used to calculate the fitness value of each individual in the group of the current generation after performing reinforcement learning training on each individual in the current generation during the evolution of any generation;

[0038] The first determination module is used to determine the elite individuals and the inferior individuals of this generation from each individual according to the fitness value of this generation of each individual;

[0039] An adjustment module, used to adjust the hyperparameter data corresponding to the elite individuals of the current generation by using the hyperparameter data corresponding to the optimal fitness value of the inferior individuals of the current generation in the evolution process of successive generations, and use the adjusted hyperparameter data as the hyperparameter data of the inferior individuals of the current generation after evolution in the current generation;

[0040] The second determination module is used to determine the target model from each individual in the group according to the fitness value of the individual in the group after performing multiple generations of evolution on each individual in the group.

[0041] To achieve the above-mentioned purpose, the present invention also provides a reinforcement learning hyperparameter optimization device, which includes: a memory, a processor, and a reinforcement learning hyperparameter optimization program stored in the memory and executable on the processor, and the reinforcement learning hyperparameter optimization program implements the steps of the reinforcement learning hyperparameter optimization method as described above when executed by the processor.

[0042] In addition, to achieve the above-mentioned purpose, the present invention also proposes a computer-readable storage medium, on which a reinforcement learning hyperparameter optimization program is stored. When the reinforcement learning hyperparameter optimization program is executed by a processor, the steps of the reinforcement learning hyperparameter optimization method as described above are implemented.

[0043] In the embodiment of the present invention, the hyperparameter data of the elite individuals are adjusted by using the hyperparameter data corresponding to the optimal fitness value of the disadvantaged individuals in the evolution process of successive generations, and the adjusted hyperparameter data are used as the hyperparameter data of the disadvantaged individuals after the evolution of the current generation, so that the hyperparameter data of the next generation of individuals are optimized towards the hyperparameter data of the elite individuals of the current generation, but at the same time, some information of the hyperparameter data corresponding to the optimal fitness value of the disadvantaged individuals of the current generation is retained, and the short-term performance is utilized, but at the same time, the exploration space of the hyperparameter data is expanded, so as to avoid falling into the local optimum by only paying attention to the short-term performance of each individual, and the hyperparameter data corresponding to the optimal fitness value of the disadvantaged individuals of the current generation in the evolution of successive generations is utilized, and the long-term performance of the individuals is utilized, which is more conducive to finding the global optimal hyperparameter data, thereby improving the effect of the hyperparameter optimization of reinforcement learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 This is a flow chart of the first embodiment of the method for optimizing hyperparameters of reinforcement learning according to the present invention;

[0045] Figure 2 A schematic diagram of an architecture for implementing a reinforcement learning hyperparameter optimization solution based on a distributed system according to an embodiment of the present invention;

[0046] Figure 3 This is a flow chart of a second embodiment of the method for optimizing hyperparameters of reinforcement learning according to the present invention;

[0047] Figure 4 This is a flow chart of a third embodiment of the method for optimizing hyperparameters of reinforcement learning according to the present invention;

[0048] Figure 5 A schematic diagram of a reinforcement learning hyperparameter optimization algorithm architecture involved in an embodiment of the present invention;

[0049] Figure 6 It is a schematic diagram of the structure of the hardware operating environment involved in the embodiment of the present invention.

[0050] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0051] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0052] The first embodiment of the method for optimizing hyperparameters of reinforcement learning of the present invention is proposed:

[0053] Reinforcement learning is one of the methods and methodologies of machine learning, which is used to describe and solve the problem of how an agent can maximize its rewards or achieve specific goals through learning strategies during its interaction with the environment. The PBT (Population Based Training) algorithm proposes a group-based automatic hyperparameter optimization solution. It optimizes the hyperparameters of the learning rate in the reinforcement learning algorithm by iteratively optimizing the hyperparameters using the mutation methods of Exploit and Explore. However, in the process of using this algorithm, the hyperparameters of the elite individuals are directly copied to replace the hyperparameters of the disadvantaged individuals to evolve the disadvantaged individuals, which causes the model to focus on the performance in the short term of training and ignore the performance in the long term of training, resulting in poor optimization of the hyperparameters of reinforcement learning. In specific application scenarios, this will affect the accuracy of the output results of the reinforcement learning model obtained by the final training.

[0054] For example, planning and scheduling is based on limited production capacity, combined with the impact of orders, materials, inventory, production capacity, etc. on production, to optimize and obtain a reasonable production plan. In the application scenario of planning and scheduling, there are problems such as large sales order data volume, limited factory production capacity, different order priorities, and limited inventory. For such multi-objective optimization problems, reinforcement learning algorithms can be used for digital modeling. In this scenario, in addition to a large number of parameters related to business logic during the modeling process, there are also many hyperparameters involved in the reinforcement learning model. Especially in a big data environment, the time required for a round of algorithm training has increased significantly. It is difficult for algorithm engineers to rely on manual experience to test a set of appropriate hyperparameters, which is difficult to meet the performance requirements of the algorithm. When the reinforcement learning model obtained by the final training is used to plan and schedule tasks, the accuracy of planning and scheduling is low.

[0055] For example, in the application scenario of intelligent replenishment of regional warehouses, in the product supply and demand link, in order to alleviate the storage capacity pressure of the headquarters warehouse and balance the fluctuations of regions and production capacity, multi-regional warehouses will be built to flexibly supply, quickly respond to order delivery, reduce logistics arrival risks, and improve logistics timeliness. In order to improve the accuracy of stocking in each regional warehouse and thus improve logistics timeliness, it is necessary to calculate the transfer volume, where the transfer volume includes the transfer quantity and transfer time between the general warehouse and the regional warehouse. While improving the accuracy of regional warehouse stocking, it is necessary to pay attention to the operating costs of the regional warehouse, such as storage costs, manual operation costs, logistics costs from the general warehouse to the regional warehouse, and logistics costs from the regional warehouse to the customer. Reinforcement learning can be used to solve this optimization problem, but the reinforcement learning model involves many hyperparameters. It is difficult for algorithm engineers to test a set of appropriate hyperparameters based on manual experience, so it is difficult to meet the algorithm performance requirements, which will result in low transfer accuracy when the reinforcement learning model obtained by the final training is used to transfer and prepare stocks for the regional warehouse.

[0056] In view of the above problems, this embodiment proposes a reinforcement learning hyperparameter optimization scheme, which adjusts the hyperparameter data of elite individuals by using the hyperparameter data corresponding to the optimal fitness value of the disadvantaged individuals in the evolution process of successive generations, and uses the adjusted hyperparameter data as the hyperparameter data of the disadvantaged individuals after evolution in this generation, so that the hyperparameter data of the next generation of individuals are optimized in the direction of the hyperparameter data of the elite individuals of this generation, but at the same time retains part of the information of the hyperparameter data corresponding to the optimal fitness value of the disadvantaged individuals of this generation, and utilizes the performance in the short term, but at the same time expands the exploration space of the hyperparameter data, so as to avoid the model focusing on the performance in the short term of training and ignoring the performance in the long term of training, thereby improving the effect of reinforcement learning hyperparameter optimization. In this embodiment, the reinforcement learning hyperparameter optimization scheme can be used in application scenarios such as planning and scheduling, intelligent allocation and replenishment of regional warehouses, etc., to solve the problems of low scheduling accuracy and low allocation accuracy caused by poor reinforcement learning hyperparameter optimization in specific application scenarios.

[0057] In this embodiment, the execution subject of the reinforcement learning hyperparameter optimization method can be a server, a server cluster, a personal computer, a smart phone and other devices, which are not limited in this embodiment. In a feasible implementation, considering that the amount of data in real business is very large and the computing power of a single machine system is insufficient, a distributed system can be used to execute the reinforcement learning hyperparameter optimization solution in this embodiment.

[0058] Reference Figure 1 , Figure 1The flowchart of the first embodiment of the method for optimizing hyperparameters of reinforcement learning of the present invention is shown in the flowchart. The embodiment of the present invention provides an embodiment of the method for optimizing hyperparameters of reinforcement learning. It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here. In this embodiment, the method for optimizing hyperparameters of reinforcement learning includes the following steps S10 to S50:

[0059] Step S10, initializing a group, each individual in the group is a reinforcement learning model with different hyperparameter data, and the hyperparameter data includes a parameter value corresponding to at least one hyperparameter.

[0060] Each individual in the group is a reinforcement learning model, and a reinforcement learning model corresponds to a set of hyperparameter data. A set of hyperparameter data includes at least one parameter value corresponding to a hyperparameter, which is the hyperparameter that needs to be optimized. The number of individuals in the group is not limited in this embodiment and can be set as needed. In this embodiment, it is not limited to which type of reinforcement learning model is specifically adopted by the reinforcement learning model. For example, it can be a deep reinforcement learning model, and the hyperparameter that needs to be optimized is the hyperparameter of the neural network as an intelligent agent in the deep reinforcement learning. In this embodiment, it is not limited to which hyperparameters need to be optimized, and they can be set specifically as needed. For example, in a feasible implementation, the hyperparameter data includes at least the parameter values ​​corresponding to one or more hyperparameters in the number of neural network layers, the number of neurons in each layer, the learning rate, the type of active function, the discount factor, and the size of the experience replay pool.

[0061] The input data and output data of the reinforcement learning model can be set according to the specific application scenario. For example, in the application scenario of planning and scheduling, the task data of the task can be used as the model input data of the reinforcement learning model, and the scheduling result of the task can be used as the model output data. The task can be a sales order, a production order, etc. that needs to be completed within a certain period of time and has a scheduling requirement. The task data can be data that affects the scheduling result of the task, which can be selected according to the needs. In this embodiment, there is no limitation, and the specific data items contained in the scheduling result can also be set according to the needs in the specific application scenario. For example, for a sales order task, the task data may include field information such as order priority, expected delivery date, order type, inventory information of the material warehouse, material table of materials required for the order, work center information, etc., wherein the work center information may include product information and capacity information that each work center can produce, etc.; the scheduling result may include the planned delivery date of the sales order, the material requirement quantity, the capacity requirement quantity, the work center requirement quantity, the material deduction date, etc.

[0062] In the application scenario of intelligent replenishment of regional warehouses, the warehouse data of the regional warehouse can be used as the model input data of the reinforcement learning model, and the transfer results of the regional warehouse can be used as the model output data. The warehouse data of the regional warehouse can include data such as sales order table, sales delivery table, sales office and regional warehouse correspondence table, inventory snapshot table, product life cycle table, etc.; the transfer results can include the inventory transfer quantity table of the regional warehouse for the current month, and the inventory transfer quantity table can include the regional warehouse ID, the material number of the material to be transferred, the forecast value of the regional warehouse's monthly inventory, the forecast value of the regional warehouse's daily inventory, the inventory value of the general warehouse, the transfer quantity of the material to be transferred, the minimum packing quantity, the minimum packing volume, the regional warehouse backlog mark, the transfer time, and other data.

[0063] For example, suppose the initialization size is N, each individual in the group is a reinforcement learning model, and the reinforcement learning model of the i-th individual is represented by M i , M i It can include one or more agents. When it includes n agents, it can be expressed as π 1 , π 2 ,…,π n , the hyperparameter data of each individual is represented by θ i , thus, the g-th generation group pop g It can be expressed as:

[0064] pop g =({M1,θ1} g ,{M2,θ2} g ,…,{M N ,θ N} g )

[0065] There are many ways to initialize a group, which are not limited in this embodiment. For example, in one feasible implementation, multiple sets of hyperparameter data can be set based on experience, each set of hyperparameter data is used as hyperparameter data corresponding to a reinforcement learning model, and the model parameters in the reinforcement learning model (such as the weights of neurons in a neural network) can be initialized based on experience or randomly initialized.

[0066] The hyperparameter data of each individual in the initialized group is different. Two sets of hyperparameter data are different if at least one hyperparameter value in the two sets of hyperparameter data is different.

[0067] In a feasible implementation, a distributed system may be used to execute the reinforcement learning hyperparameter optimization solution in this embodiment. Figure 2As shown, the distributed system includes a central node and multiple computing nodes. The group can be initialized through the central node. During each generation of evolution, the central node distributes each individual in the group (that is, the reinforcement learning model) and the corresponding hyperparameter data to each computing node. Each computing node is distributed with at least one individual and the corresponding hyperparameter data. In practical applications, the number of computing nodes can be set according to the requirements of IO (Input Output) cost and computing cost.

[0068] Step S20, in any generation of evolution, after performing reinforcement learning training on each individual in the group of the current generation, the fitness value of each individual of the current generation is calculated.

[0069] It should be noted that each hyperparameter of the reinforcement learning model has multiple possible values. The purpose of optimizing the hyperparameters of the reinforcement learning model is to find appropriate parameter values ​​for each hyperparameter so that the reinforcement learning model can be more suitable for the reinforcement learning tasks in specific application scenarios.

[0070] The hyperparameter data of each individual in the group is not necessarily optimal. It is necessary to optimize the hyperparameter data of each individual by performing multiple generations of evolution on each individual in the group. After multiple generations of evolution, the final model is determined from each individual. The reinforcement learning hyperparameter optimization scheme provided in this embodiment belongs to an automated reinforcement learning scheme (Automated Reinforcement Learning, AutoRL), which uses a hyperparameter optimization method to automatically find the optimal hyperparameter combination of the reinforcement learning algorithm. It also belongs to an evolutionary reinforcement learning scheme (Evolutionary Reinforcement Learning), which combines evolutionary learning and reinforcement learning. The intelligent agent can perform multiple tasks in multiple complex environments.

[0071] The evolution process of each generation of individuals in the group is the same, so the following is an example of the evolution of one generation. It should be noted that each individual in the initialized group is called the first generation individual. After the first generation of individuals are evolved for the first generation, the second generation individuals are obtained. After the second generation of individuals are evolved for the second generation, the third generation individuals are obtained, and so on.

[0072] After each individual in the group of this generation is subjected to reinforcement learning training, the individual fitness value corresponding to each individual is calculated (called the fitness value of this generation for distinction). The method of performing reinforcement learning training on individuals is not limited in this embodiment, and the existing reinforcement learning training method can be used.

[0073] For example, in a feasible implementation, the reinforcement learning model can use task data as model input data and the task scheduling result as model output data. Accordingly, the trained target model is used to perform planning and scheduling based on the task data to obtain the task scheduling result. The step of performing reinforcement learning training on each individual in the group of this generation in step S20 includes:

[0074] Step S201, for any target individual among the individuals, the task data of the target task in the preset training data set is input into the reinforcement learning model corresponding to the target individual for planning and scheduling, and a scheduling result of the target task is obtained.

[0075] The method of performing reinforcement learning training on each individual is the same, so an individual is used as an example for explanation and is referred to as a target individual for distinction. The preset training data set can be set in advance as needed, and the preset training data set includes task data of multiple tasks (hereinafter referred to as target tasks for distinction). By inputting the task data of the target task into the reinforcement learning model corresponding to the target individual, the scheduling result is obtained by predicting the intelligent agent in the reinforcement learning model based on the task data.

[0076] Step S202: simulate the execution of the target task according to the scheduling result to obtain the task achievement status of the target task.

[0077] The target task can be simulated and executed by a simulation environment designed in advance according to the actual execution process of the task to obtain the task achievement status. The task achievement status can be, for example, whether the task is achieved on time or not, which can be determined according to the execution result of the simulation environment. For example, the task data is equivalent to the state of the simulation environment at time t. The task data is input into the reinforcement learning model for planning and scheduling. The obtained scheduling result is the action taken by the reinforcement learning model at time t. The scheduling result is input into the simulation environment. For example, the planned delivery date of the sales order, the material requirement quantity, the capacity requirement quantity, the work center requirement quantity, the material deduction date and other scheduling results are input into the simulation environment. The planned delivery date of the sales order is compared with the expected delivery date through the simulation environment, and whether the sales order can be achieved on time is determined according to the comparison result. The simulation environment deducts materials, capacity, etc. according to the scheduling results to obtain the state of the simulation environment after deduction, that is, the state of the simulation environment at time t+1.

[0078] Step S203, generating an environmental reward according to the task achievement status, and optimizing the reinforcement learning model corresponding to the target individual according to the environmental reward.

[0079] It can be that the environment reward is generated according to the task completion status by using a preset reward calculation method, and the environment reward is the reward of the simulated environment feedback after the scheduling action is made at time t. For example, the calculation method of the environment reward can be set to 1 if the order can be completed on schedule, and -1 if it cannot be completed. For a batch of outstanding sales order data input, the more orders are completed, the higher the environment reward, and vice versa. Optimizing the reinforcement learning model according to the environment reward is to update the model parameters in the reinforcement learning model. The specific optimization method can refer to the method of optimizing the reinforcement learning model according to the environment reward in the related technology. For example, when the reinforcement learning hyperparameter optimization scheme in this embodiment is applied to optimize the hyperparameters of the policy network of the DDPG (deep deterministic policy gradient) reinforcement learning algorithm, the experience fragment can be used to optimize the reinforcement learning model; wherein the experience fragment includes the simulation environment state at time t, the scheduling action at time t (that is, the scheduling result), the environmental reward fed back by the simulation environment at time t, and the simulation environment state at time t+1. In order to maximize the environmental reward at time T, the negative Q value can be used as the loss function to perform gradient descent training on the policy network to optimize the policy network; the Q value is input into the Critic (evaluation) network in the DDPG algorithm using the simulation environment state at time t and the scheduling action at time t, and is calculated by the Critic network.

[0080] For another example, in a feasible implementation, the reinforcement learning model can use the warehouse data of the regional warehouse as the model input data and the allocation result of the regional warehouse as the model output data. Accordingly, the trained target model is used to allocate and prepare goods based on the warehouse data of the regional warehouse to obtain the allocation result of the regional warehouse. For any target individual among the individuals, the warehouse data of the target regional warehouse in the preset training data set can be input into the reinforcement learning model corresponding to the target individual, and the allocation result of the target regional warehouse can be obtained after model processing.

[0081] The method of reinforcement learning training for each individual is the same, so an individual is used as an example for explanation, and is called the target individual for distinction. The preset training data set can be set in advance as needed, and the preset training data set includes warehouse data of multiple regional warehouses (hereinafter referred to as target regional warehouses for distinction), for example, sales order tables, sales delivery tables, sales office and regional warehouse correspondence tables, inventory snapshot tables, product life cycle tables and other data. By inputting the warehouse data of the target regional warehouse into the reinforcement learning model corresponding to the target individual, the intelligent agent in the reinforcement learning model predicts the allocation result based on the warehouse data, such as the inventory allocation quantity table of the target regional warehouse for the current month. The inventory allocation quantity table of the target regional warehouse for the current month can be input into the simulation environment, and the simulation environment is used to simulate the allocation to obtain the result of whether the inventory of the regional warehouse meets the next-day or next-day delivery of the sales order. The simulation environment can be set according to the actual allocation process of the regional warehouse. For example, the warehouse data of the regional warehouse is equivalent to the state of the simulation environment at time t. The warehouse data is input into the reinforcement learning model for prediction. The transfer result obtained is the action made by the reinforcement learning model at time t. The transfer result is input into the simulation environment. For example, the sales order table, sales delivery table, sales office and regional warehouse correspondence table, inventory snapshot table, and product life cycle table are input into the simulation environment. The predicted value of the daily inventory required by the regional warehouse is compared with the demand quantity in the sales order table through the simulation environment. According to the comparison result, it is determined whether the inventory of the target regional warehouse can meet the sales order for the next day or the next day. The simulation environment can deduct the inventory transfer quantity of the target regional warehouse in the current month according to the transfer result, combined with the regional warehouse inventory, the total warehouse inventory, the regional warehouse outstanding orders, and the total warehouse outstanding orders, to obtain the state of the simulation environment after the deduction, that is, the state of the simulation environment at time t+1. The environment reward can be calculated according to the judgment result using a preset reward calculation method. The environment reward is the reward fed back by the simulation environment after the transfer action is made at time t. The method of calculating the environmental reward can be set in advance according to needs. For example, it can be set that if the inventory of the regional warehouse can meet the sales order for the next day or the next day, the environmental reward is 1, and if it does not meet the requirements, the environmental reward is -1. Optimizing the reinforcement learning model according to the environmental reward is to update the model parameters in the reinforcement learning model. The specific optimization method can refer to the method of optimizing the reinforcement learning model according to the environmental reward in the relevant technology.For example, when the reinforcement learning hyperparameter optimization scheme in this embodiment is applied to optimize the hyperparameters of the policy network of the DDPG reinforcement learning algorithm, the experience fragment can be used to optimize the reinforcement learning model; wherein the experience fragment includes the simulation environment state at time t, the allocation action at time t (that is, the allocation result), the environmental reward fed back by the simulation environment at time t, and the simulation environment state at time t+1. In order to maximize the environmental reward at time T, the negative Q value can be used as the loss function to perform gradient descent training on the policy network to optimize the policy network; the Q value is input into the Critic (evaluation) network in the DDPG algorithm using the simulation environment state at time t and the allocation action at time t, and is calculated by the Critic network.

[0082] The fitness value represents the quality of an individual. In this embodiment, the quality of an individual is evaluated by the training effect of the reinforcement learning model. The fitness value of each individual in this generation can be calculated based on the effect of the reinforcement learning training. In this embodiment, the specific method for calculating the fitness value of this generation is not limited.

[0083] For example, in one feasible implementation, the step of calculating the fitness value of each individual in this generation in step S20 includes:

[0084] Step S211: for any target individual among the individuals, the target individual is made to interact with the environment multiple times to obtain multiple accumulated environmental reward expectations.

[0085] The method for calculating the fitness value of each individual in this generation is the same, so an individual is used as an example for explanation, and it is called the target individual for distinction. After the target individual is subjected to reinforcement learning training, the target individual can be interacted with the environment multiple times to obtain multiple cumulative environmental reward expectations, and the cumulative environmental reward expectation is the expectation of the cumulative environmental reward. The number of interactions is not limited in this embodiment and can be set as needed. The environment refers to the external environment set in a specific application scenario for interacting with the intelligent agent. For example, in the application scenario of planning and scheduling, the environment can be a simulation environment designed according to the actual execution process of the task. Accordingly, the specific process of the interaction between the individual and the environment and the calculation method of the cumulative environmental reward expectation are not limited in this embodiment. For example, when the reinforcement learning hyperparameter optimization scheme in this embodiment is applied to optimize the hyperparameters of the policy network of the DDPG reinforcement learning algorithm, the Critic network in the DDPG reinforcement learning algorithm can be used to calculate the expectation of the cumulative environmental reward obtained by the interaction between the individual and the environment.

[0086] In one feasible implementation, the process of performing multiple interactions between the target individual and the environment to obtain multiple cumulative environmental reward expectations can be: after reinforcement learning training is performed on the target individual, the task data (equivalent to the state of the simulation environment at time t) of the task in the preset training data set (referred to as the target task for distinction) can be input into the reinforcement learning model corresponding to the target individual for planning and scheduling, and the scheduling result of the target task (equivalent to the action taken at time t) is obtained, and the task data and the scheduling result are input into the Critic network, and the cumulative environmental reward expectations are calculated through the Critic network; in accordance with the above method, multiple cumulative environmental reward expectations can be obtained based on multiple target tasks in the preset training data set.

[0087] Step S212: determining an average value of the multiple accumulated environmental reward expectations as the current generation fitness value of the target individual, or determining a variance of the multiple accumulated environmental reward expectations as the current generation fitness value of the target individual.

[0088] In one feasible implementation, the average value of multiple cumulative environmental reward expectations can be determined as the fitness value of the target individual in this generation. Since the higher the cumulative environmental reward expectation, the better the effect of the individual's reinforcement learning training, the average value of multiple cumulative environmental reward expectations obtained by multiple interactions between the target individual and the environment is determined as the fitness value of the target individual in this generation, which can accurately characterize the quality of each individual, thereby improving the accuracy of the evolutionary direction of each individual, thereby facilitating the search for the optimal hyperparameter data.

[0089] In one feasible implementation, the variance of multiple cumulative environmental reward expectations can be determined as the fitness value of the target individual in this generation. Since the more stable the cumulative environmental reward expectations are, the better the effect of reinforcement learning training for the individual is, the variance of multiple cumulative environmental reward expectations obtained by multiple interactions between the target individual and the environment is determined as the fitness value of the target individual in this generation, which can accurately characterize the quality of each individual, thereby improving the accuracy of the evolutionary direction of each individual, thereby facilitating the search for the optimal hyperparameter data.

[0090] In one feasible implementation, for any computing node in a distributed system, after the computing node receives the individual (i.e., reinforcement learning model) and its corresponding hyperparameter data from the central node, the reinforcement learning model can be subjected to reinforcement learning training, and the model parameters of the trained reinforcement learning model are sent to the central node. The central node evaluates the trained reinforcement learning model to obtain the fitness value of the individual in this generation. In the process of training the reinforcement learning model in the computing node, the reinforcement learning model interacts with the environment, and the obtained experience fragments can be stored in the experience replay pool, and the model parameters of the reinforcement learning model are optimized through the experience fragments in the experience replay pool. The experience fragments are the results obtained by the interaction between the reinforcement learning model and the environment. For example, the experience fragments may include: [environmental state at time t, reinforcement learning model output result at time t, environmental reward at time t, and environmental state at time t+1]. The experience replay pool can be stored locally in the computing node or in a storage node in the distributed system.

[0091] Step S30, determining the elite individuals and the inferior individuals of this generation from each individual according to the fitness value of each individual of this generation.

[0092] The number of elite individuals and inferior individuals of this generation can be set as needed, and is not limited in this embodiment. The fitness value of the elite individuals of this generation is higher than the fitness value of the inferior individuals of this generation, but in this embodiment, the specific determination method of the elite individuals of this generation and the inferior individuals of this generation is not limited. For example, in a feasible implementation, the fitness values ​​of the current generation of each individual can be sorted in descending order, and the individuals in the top 20% are regarded as the elite individuals of this generation, and the individuals in the bottom 20% are regarded as the inferior individuals of this generation.

[0093] Step S40, using the hyperparameter data corresponding to the optimal fitness value of the inferior individuals of this generation in the evolution process of successive generations to adjust the hyperparameter data corresponding to the elite individuals of this generation, and using the adjusted hyperparameter data as the hyperparameter data of the inferior individuals of this generation after evolution in this generation.

[0094] After calculating the fitness value corresponding to each individual in each generation of evolution, the optimal fitness value of each individual in the evolutionary process and the hyperparameter data corresponding to the optimal fitness value can be recorded.

[0095] For example, f i g Represents individual M i The fitness value of the evolution to the gth generation, f i best Represents individual M i The optimal fitness value in the evolutionary process of successive generations, θ i gRepresents individual M i Evolved to the hyperparameter data of the gth generation, θ i best Represents individual M i The hyperparameter data corresponding to the optimal fitness value in the evolutionary process of successive generations. i best =max{f i 1 , f i 2 , …, f i g}, θ i best = argmaxf i g (θ i g ).

[0096] For any inferior individual of this generation, the hyperparameter data corresponding to the optimal fitness value of the inferior individual of this generation in the evolution process of successive generations can be used to adjust the hyperparameter data corresponding to the elite individual of this generation, and the adjusted hyperparameter data can be used as the hyperparameter data of the inferior individual of this generation after evolution in this generation. In a specific implementation, when there are multiple elite individuals of this generation, for each inferior individual of this generation, uniform sampling can be performed from multiple elite individuals of this generation to obtain an elite individual of this generation, and the hyperparameter data corresponding to the elite individual of this generation is adjusted using the hyperparameter data corresponding to the optimal fitness value of the inferior individual of this generation.

[0097] In this embodiment, there is no restriction on the way of adjusting the hyperparameter data corresponding to the elite individuals of this generation using the hyperparameter data corresponding to the optimal fitness value, and it can be set as needed. The purpose is to optimize the hyperparameter data of the individuals of the next generation towards the hyperparameter data of the elite individuals of this generation, but at the same time retain some information of the hyperparameter data corresponding to the optimal fitness value of the inferior individuals of this generation.

[0098] For example, in one feasible implementation, the parameter values ​​corresponding to the same hyperparameter in the hyperparameter data of the elite individuals of this generation and the hyperparameter data of the inferior individuals of this generation can be weighted averaged to obtain the weighted average parameter values ​​corresponding to each hyperparameter, forming a group of hyperparameter data, and using this group of hyperparameter data as the adjusted hyperparameter data. The weights used in the weighted average can be set as needed and are not limited here. In one feasible implementation, for integer hyperparameters, the result obtained by the weighted average corresponding to the hyperparameter can be rounded to ensure the legitimacy of the parameter.

[0099] In a specific implementation, for each individual in the group other than the disadvantaged individual of this generation, the hyperparameter data of each individual can be kept unchanged and directly enter the next generation of evolution, or other methods can be used for evolution. The specific settings can be made as needed and are not limited in this embodiment.

[0100] In one feasible implementation, after calculating the fitness value of each individual in this generation, the central node can determine the elite individuals and the inferior individuals of this generation from each individual, and use the hyperparameter data corresponding to the optimal fitness value of the inferior individuals of this generation in the evolution process of previous generations to adjust the hyperparameter data corresponding to the elite individuals of this generation, and use the adjusted hyperparameter data as the hyperparameter data of the inferior individuals of this generation after evolution in this generation.

[0101] Step S50, after performing multiple generations of evolution on each individual in the group, a target model is determined from each individual according to the fitness value of the individual in the group.

[0102] There is no limit on the number of times the group is evolved, which can be set as needed; or a stop condition can be set, and the evolution is stopped after the stop condition is met. The stop condition can be, for example, that the individual fitness values ​​of more than half of the individuals exceed a preset threshold.

[0103] The target model refers to a determined reinforcement learning model, and the hyperparameter data of the reinforcement learning model is the hyperparameter data optimized by the group evolution method. There are many ways to determine the target model from each individual according to the fitness value of each individual in the group, which is not limited in this embodiment. For example, the individual with the highest individual fitness value can be used as the target model.

[0104] After obtaining the target model, the target model can be applied to the prediction tasks in specific application scenarios. For example, in the application scenario of planning and scheduling, after obtaining the target model, for tasks that need to be planned and scheduled, the target model can be used to plan and schedule based on the task data of the task, and the scheduling result of the task can be obtained, so that the execution subject of the task can execute the task according to the scheduling result, ensuring that the task can be completed on schedule. For another example, in the application scenario of intelligent allocation and replenishment of regional warehouses, after obtaining the target model, for regional warehouses that need to be allocated and stocked, the target model can be used to allocate and stocked based on the warehouse data of the regional warehouse, and the allocation result of the regional warehouse can be obtained, so that the regional warehouse can be allocated according to the allocation result, ensuring that the regional warehouse can meet the inventory demand.

[0105] In a feasible implementation, after multiple generations of evolution of each individual in the group through a central node, a target model can be determined from each individual according to the fitness value of the individual in the group.

[0106] In the related art, the hyperparameter data of elite individuals are used to directly replace the hyperparameter data of disadvantaged individuals to evolve the disadvantaged individuals. Since elite individuals are only individuals with excellent performance in the short term, directly using the hyperparameter data of elite individuals to replace the hyperparameter data of disadvantaged individuals will cause the hyperparameter optimization process of the reinforcement learning model to only focus on the short-term performance of each individual and ignore the long-term performance, thus falling into the local optimum and making it difficult to find the global optimum, resulting in unsatisfactory hyperparameter optimization effect of reinforcement learning. In this embodiment, the hyperparameter data of the elite individuals are adjusted by using the hyperparameter data corresponding to the optimal fitness value of the disadvantaged individuals in the evolution process of successive generations, and the adjusted hyperparameter data are used as the hyperparameter data of the disadvantaged individuals after evolution in this generation, so that the hyperparameter data of the next generation of individuals are optimized towards the hyperparameter data of the elite individuals of this generation, but at the same time, some information of the hyperparameter data corresponding to the optimal fitness value of the disadvantaged individuals of this generation is retained, and the short-term performance is utilized, but at the same time, the exploration space of the hyperparameter data is expanded, so as to avoid falling into the local optimum by only paying attention to the short-term performance of each individual, and the hyperparameter data corresponding to the optimal fitness value of the disadvantaged individuals of this generation in the evolution of successive generations are utilized, and the long-term performance of the individuals is utilized, which is more conducive to finding the global optimal hyperparameter data, thereby improving the effect of the hyperparameter optimization of reinforcement learning. For example, in the application scenario of planning and scheduling, a higher scheduling accuracy can be obtained when the trained target model is used to plan and schedule tasks, and in the application scenario of intelligent allocation and replenishment of regional warehouses, a higher allocation accuracy can be obtained when the trained target model is used to allocate and prepare stocks for regional warehouses.

[0107] Based on the above first embodiment, a second embodiment of the method for optimizing hyperparameters of reinforcement learning of the present invention is proposed. In this embodiment, referring to Figure 3 After step S30, steps S60 to S70 are also included. In a feasible implementation, steps S60 to S70 in this embodiment can be executed by a central node in a distributed system.

[0108] Step S60, calculating the utilization strategy probability corresponding to the disadvantaged individual of this generation according to the current generation fitness value of the disadvantaged individual of this generation, wherein the higher the current generation fitness value of the disadvantaged individual of this generation, the higher the utilization strategy probability corresponding to the disadvantaged individual of this generation.

[0109] After determining the inferior individuals and elite individuals of the current generation among the individuals of the current generation, for any inferior individual of the current generation, the utilization strategy probability corresponding to the inferior individual of the current generation can be calculated according to the fitness value of the inferior individual of the current generation. In this embodiment, the specific calculation method for calculating the utilization strategy probability according to the fitness value of the inferior individual of the current generation is not limited, and can be set as needed, and the calculation method meets the requirement that the utilization strategy probability corresponding to the inferior individual of the current generation is higher when the fitness value of the inferior individual of the current generation is higher.

[0110] For example, in a feasible implementation, the mean of the fitness values ​​of each elite individual of this generation can be calculated, and then the mean value is added to the fitness value of the inferior individual of this generation to obtain a sum value, and the fitness value of the inferior individual of this generation is divided by the sum value, and the result is used as the utilization strategy probability corresponding to the inferior individual of this generation. It can be understood that this calculation method meets the requirement that the higher the fitness value of the inferior individual of this generation, the higher the utilization strategy probability corresponding to the inferior individual of this generation.

[0111] Step S70, selecting a target strategy from at least two preset variant strategies according to the utilization strategy probability, wherein the at least two variant strategies include the utilization strategy, and the higher the utilization strategy probability, the higher the possibility that the utilization strategy is selected.

[0112] At least two optional mutation strategies can be set in advance, including the utilization strategy, which is the strategy described in step S40, that is, using the hyperparameter data corresponding to the optimal fitness value of the inferior individuals of this generation in the evolution process of previous generations to adjust the hyperparameter data corresponding to the elite individuals of this generation, and using the adjusted hyperparameter data as the hyperparameter data of the inferior individuals of this generation after evolution in this generation. In addition to the utilization strategy, other mutation strategies can also be included, which can be set specifically as needed and are not limited in this embodiment.

[0113] After calculating the probability of utilizing the strategy corresponding to the disadvantaged individual of this generation, a strategy can be selected from at least two variant strategies according to the probability of utilizing the strategy, and the selected strategy is called the target strategy for distinction. In this embodiment, there is no restriction on the specific selection method for selecting the target strategy from at least two variant strategies according to the probability of utilizing the strategy, and it can be set as needed. The selection method can meet the requirement that the higher the probability of utilizing the strategy, the higher the probability of utilizing the strategy being selected.

[0114] For example, in a feasible implementation, when two optional mutation strategies are set, including the utilization strategy, a random number between 0 and 1 can be randomly generated. If the random number is less than or equal to the utilization strategy probability corresponding to the disadvantaged individual of this generation, the utilization strategy is determined as the target strategy. If the random number is greater than the utilization strategy probability corresponding to the disadvantaged individual of this generation, the other mutation strategy other than the utilization strategy in the two mutation strategies is determined as the target strategy. It can be understood that when the utilization strategy probability is large, the randomly generated random number is more likely to be less than or equal to the utilization strategy probability, so that the utilization strategy is more likely to be selected as the target strategy. On the contrary, when the utilization strategy probability is small, the randomly generated random number is more likely to be greater than the utilization strategy probability, so that the utilization strategy is less likely to be selected as the target strategy. That is, the selection method proposed in this implementation meets the requirement that the utilization strategy is more likely to be selected when the utilization strategy probability is higher.

[0115] When the target policy is the utilization policy, step S40 is performed.

[0116] When the target strategy corresponding to the disadvantaged individuals of this generation is the utilization strategy, the disadvantaged individuals of this generation may be mutated according to the strategy described in step S40 to evolve the disadvantaged individuals of this generation.

[0117] In a feasible implementation, the parameter values ​​corresponding to the same hyperparameter in the hyperparameter data of the elite individuals of this generation and the hyperparameter data of the inferior individuals of this generation can be weighted averaged to obtain the weighted average parameter values ​​corresponding to each hyperparameter, forming a group of hyperparameter data, and using this group of hyperparameter data as the adjusted hyperparameter data. Among them, when weighted averaging, the weight corresponding to the hyperparameter data of the elite individuals of this generation can be the probability of using the strategy, and the weight corresponding to the hyperparameter data of the inferior individuals of this generation can be the result obtained by subtracting the probability of using the strategy from 1. It can be seen from the calculation formula that the individuals with poor performance have less influence on the new hyperparameter data obtained by adjustment due to their smaller weights, which can be regarded as adjustments made on the basis of the hyperparameter data of the elite individuals to expand the search space.

[0118] In this embodiment, at least two optional mutation strategies are set, which further expands the search space, thereby being more conducive to finding the global optimal hyperparameter data. The probability of using the strategy is calculated according to the current generation fitness value of the inferior individual of the current generation, and the target strategy is selected from at least two mutation strategies according to the probability of using the strategy. When the current generation fitness value of the inferior individual of the current generation is higher, the target strategy corresponding to the inferior individual of the current generation is more likely to be the utilization strategy, so that when the hyperparameter data of the inferior individual of the current generation in the current generation evolution is good, the possibility of using the best hyperparameter data of the inferior individual of the current generation to adjust the hyperparameter data of the elite individual of the current generation is higher, so that while ensuring the optimization in the direction of the hyperparameter data of the elite individual of the current generation, some information of the best hyperparameter data of the inferior individual of the current generation is also retained, expanding the search space while also ensuring the accuracy of the search direction.

[0119] Based on the above second embodiment, a third embodiment of the method for optimizing hyperparameters of reinforcement learning of the present invention is proposed. In this embodiment, referring to Figure 4 After step S70, step S80 is further included. In a feasible implementation, step S80 in this embodiment can be performed by a central node in the distributed system.

[0120] Step S80, when the target strategy is the exploration strategy, obtain the current generation probability distribution corresponding to the at least one hyperparameter in the current generation evolution, sample the hyperparameter data according to the current generation probability distribution, and use the sampled hyperparameter data as the hyperparameter data of the disadvantaged individual of the current generation after the evolution of the current generation.

[0121] In this embodiment, at least two mutation strategies may also include an exploration strategy, which may be to obtain new hyperparameter data by sampling from a probability distribution, and directly use the sampled new hyperparameter data as the hyperparameter data after the evolution of the inferior individuals of this generation.

[0122] The probability distribution corresponding to the hyperparameter in the current generation of evolution is called the current generation probability distribution for distinction. Each hyperparameter corresponds to a current generation probability distribution, that is, when there are multiple hyperparameters that need to be optimized, the current generation probability distribution corresponding to each hyperparameter in the current generation of evolution is obtained. It should be noted that the probability distribution corresponding to the same hyperparameter in each generation of evolution can be the same, or it can be different, which is not limited in this embodiment. The probability distribution of the hyperparameter in the gth generation can be expressed as N(μ g , σ g 2 ).

[0123] Taking the example of multiple hyperparameters that need to be optimized, for each disadvantaged individual of this generation, if the target strategy corresponding to the disadvantaged individual of this generation is the exploration strategy, then after obtaining the probability distribution of this generation corresponding to each hyperparameter, a parameter value can be sampled according to each probability distribution of this generation to obtain multiple parameter values. The multiple parameter values ​​constitute a set of hyperparameter data, and the set of hyperparameter data is used as the hyperparameter data of the disadvantaged individual of this generation after evolution in this generation.

[0124] For example, f i g represents the fitness value of the i-th disadvantaged individual in the g-th generation, f j g represents the fitness value of the jth elite individual in the gth generation, p i represents the probability of the utilization strategy corresponding to the i-th disadvantaged individual, and K represents the number of elite individuals in the g-th generation, which can be calculated as follows:

[0125]

[0126] From p i It can be seen from the calculation formula that the probability of disadvantaged individuals adopting the strategy mutation is determined by their own fitness value and the mean fitness value of elite individuals. i The fitness value f i g The higher it is, the better its hyperparameter data is, and the probability p of individuals adopting strategy mutation is higher. i The larger it is, the more disadvantaged individuals can use their own hyperparameter data for fine-tuning in the next round of updates. i The lower the fitness value, the more unreasonable the current hyperparameter data is, and the probability p of the individual adopting the strategy mutation is i The smaller it is, the higher the probability of using the exploration strategy for resampling.

[0127] In this embodiment, two mutation strategies, namely the utilization strategy and the exploration strategy, are set, and the utilization strategy probability is calculated according to the current-generation fitness value of the disadvantaged individual of this generation. A target strategy is selected from at least two mutation strategies according to the utilization strategy probability. When the current-generation fitness value of the disadvantaged individual of this generation is lower, the target strategy corresponding to the disadvantaged individual of this generation is more likely to be the exploration strategy. Therefore, when the hyperparameter data of the disadvantaged individual of this generation in the current-generation evolution is poor, it is more likely to evolve the disadvantaged individual of this generation by resampling, and individuals with poor performance are eliminated, thereby expanding the search space while also ensuring the accuracy of the search direction.

[0128] In one feasible implementation, step S10 includes:

[0129] Step S101, initializing an initial probability distribution corresponding to the at least one hyperparameter.

[0130] Initializing the group is to initialize the hyperparameter data of each individual in the group (that is, each reinforcement learning model). In this embodiment, a feasible group initialization method is proposed.

[0131] Initialize the probability distribution corresponding to at least one hyperparameter (hereinafter referred to as the initial probability distribution for distinction), which can be expressed as N(μ0, σ0 2 ). It should be noted that when there is only one hyperparameter to be optimized, an initial probability distribution corresponding to the hyperparameter is initialized. When there are multiple hyperparameters to be optimized, an initial probability distribution corresponding to each hyperparameter is initialized, that is, multiple initial probability distributions are initialized. The initial probability distribution may include a mean parameter and a variance parameter, which may be set based on experience or randomly selected, and are not limited in this embodiment.

[0132] Step S102: sampling and obtaining a plurality of different groups of the hyperparameter data according to the initial probability distribution, and using each group of the hyperparameter data as the hyperparameter data of each reinforcement learning model in the group.

[0133] According to the initial probability distribution corresponding to each hyperparameter, a parameter value corresponding to the hyperparameter can be sampled, and the parameter values ​​obtained by sampling each hyperparameter can be combined to obtain a set of hyperparameter data. Multiple samplings are performed to obtain multiple sets of hyperparameter data, and each set of hyperparameter data can be used as the hyperparameter data of each reinforcement learning model in the initialized group.

[0134] In one feasible implementation, for an integer hyperparameter, if a non-integer is obtained by sampling according to the probability distribution corresponding to the hyperparameter, the sampling result can be rounded up and the obtained integer is used as the parameter value corresponding to the hyperparameter.

[0135] In a feasible implementation manner, the step of obtaining the probability distribution of the current generation corresponding to the at least one hyperparameter in the current generation evolution in step S80 includes:

[0136] Step S801, obtaining the previous generation probability distribution corresponding to the at least one hyperparameter, wherein the previous generation probability distribution corresponding to the at least one hyperparameter in the first generation evolution process is the initial probability distribution.

[0137] In this embodiment, a feasible implementation method for obtaining the probability distribution of the current generation corresponding to the hyperparameter in the current generation of evolution is proposed. For any hyperparameter that needs to be optimized, the probability distribution of the previous generation corresponding to the hyperparameter can be obtained in any generation of evolution. It should be noted that in the first generation of evolution, the initial probability distribution corresponding to the hyperparameter can be used as the probability distribution of the previous generation corresponding to the hyperparameter, and in the second and subsequent generations of evolution, the probability distribution of the hyperparameter in the previous generation of evolution can be used as the probability distribution of the previous generation corresponding to the hyperparameter.

[0138] Step S802, using the hyperparameter data of the elite individuals of the current generation to update the probability distribution of the previous generation, to obtain the probability distribution of the current generation corresponding to the at least one hyperparameter in the current generation evolution.

[0139] The hyperparameter data of the elite individuals of this generation can be used to update the probability distribution of the previous generation corresponding to at least one hyperparameter that needs to be optimized, so as to obtain the probability distribution of this generation corresponding to the hyperparameter in the evolution of this generation. That is, in this embodiment, in each generation of evolution, the probability distribution corresponding to the hyperparameter that needs to be updated can be updated using the hyperparameter data of each generation of elite individuals, so as to optimize the probability distribution corresponding to each hyperparameter, so that in the process of evolving the inferior individuals, it can evolve towards the hyperparameter data of the elite individuals, which is conducive to finding the optimal hyperparameter data and eliminating the influence of the human factors of the prior distribution.

[0140] In this embodiment, the method of updating the probability distribution of the previous generation according to the hyperparameter data of the elite individuals of this generation is not limited, and can be set as needed. For example, in a feasible embodiment, when there are multiple elite individuals of this generation, for any hyperparameter that needs to be optimized, the parameter value corresponding to the hyperparameter in the hyperparameter data of the elite individuals of this generation can be averaged, and the calculation result can be used as the mean parameter of the probability distribution of this generation in the evolution of this generation. The parameter value corresponding to the hyperparameter in the hyperparameter data of each elite individual of this generation is deducted from the mean parameter in the probability distribution of the previous generation corresponding to the hyperparameter, and then the square is calculated to obtain the intermediate calculation results corresponding to each elite individual of this generation, and each intermediate calculation result is averaged to obtain the variance parameter of the probability distribution of this generation in the evolution of this generation.

[0141] In one feasible implementation, step S802 includes:

[0142] Step S8021, normalizing the fitness value of each elite individual of the current generation to obtain the weight corresponding to each elite individual of the current generation.

[0143] In this embodiment, a feasible implementation method is proposed to update the probability distribution of the previous generation according to the hyperparameter data of the elite individuals of the current generation.

[0144] In the case where there are multiple elite individuals of this generation, the fitness values ​​of each elite individual of this generation can be normalized to obtain the weights corresponding to each elite individual of this generation. In a feasible implementation, the normalization method can be to sum the fitness values ​​of each elite individual of this generation to obtain the total fitness value, and for any elite individual of this generation, the fitness value of the elite individual of this generation is divided by the total fitness value to obtain the weight corresponding to the elite individual of this generation.

[0145] Step S8022: For any target hyperparameter among the at least one hyperparameter, the parameter values ​​corresponding to the target hyperparameter in the hyperparameter data of each elite individual of the current generation are weightedly summed using the weight, and the mean parameter in the probability distribution of the current generation corresponding to the target hyperparameter in the current generation evolution is obtained according to the calculation result.

[0146] An arbitrary hyperparameter of at least one hyperparameter to be optimized is used as an example for explanation, and the hyperparameter is referred to as a target hyperparameter for distinction. The parameter value corresponding to the target hyperparameter in the hyperparameter data of each elite individual of this generation can be weighted summed using the weights corresponding to each elite individual of this generation, and the mean parameter in the probability distribution of this generation corresponding to the target hyperparameter in the evolution of this generation is obtained according to the calculation result. In a specific implementation, the calculation result can be directly used as the mean parameter in the probability distribution of this generation corresponding to the target hyperparameter in the evolution of this generation, or other processing can be performed on the basis of the calculation result to obtain the mean parameter in the probability distribution of this generation corresponding to the target hyperparameter in the evolution of this generation. It can be specifically set as needed and is not limited here.

[0147] Step S8023, subtract the old mean parameter from the parameter value corresponding to the target hyperparameter in the hyperparameter data of each elite individual of the current generation, and then calculate the square to obtain the intermediate calculation results corresponding to each elite individual of the current generation, wherein the old mean parameter is the mean parameter in the probability distribution of the previous generation corresponding to the target hyperparameter.

[0148] The mean parameter in the probability distribution of the previous generation corresponding to the target hyperparameter can be called the old mean parameter for distinction. The parameter value distribution corresponding to the target hyperparameter in the hyperparameter data of each elite individual of this generation is subtracted from the old mean parameter and then squared to obtain a calculation result corresponding to each elite individual of this generation (called the intermediate calculation result for distinction).

[0149] Step S8024, performing weighted summation on each of the intermediate calculation results using the weights, and obtaining the variance parameter of the probability distribution of the current generation corresponding to the target hyperparameter in the current generation evolution according to the calculation results.

[0150] The intermediate calculation results corresponding to each elite individual of this generation are weighted summed using the weights corresponding to each elite individual of this generation, and the variance parameters of the probability distribution of this generation corresponding to the target hyperparameter in the evolution of this generation are obtained according to the calculation results. In a specific implementation, the calculation results can be directly used as the variance parameters of the probability distribution of this generation corresponding to the target hyperparameter in the evolution of this generation, or other processing can be performed on the basis of the calculation results to obtain the variance parameters of the probability distribution of this generation corresponding to the target hyperparameter in the evolution of this generation. The specific settings can be made as needed and are not limited here. For example, a preset error value can be added to the calculation results to obtain the variance parameters of the probability distribution of this generation corresponding to the target hyperparameter in the evolution of this generation.

[0151] For example, assuming there are K elite individuals of this generation, the parameter value of the target hyperparameter in the hyperparameter data of the i-th elite individual of this generation is expressed as T i , the previous generation probability distribution of the target hyperparameter is expressed as N(μ old , σ old 2 ), ε i represents the weight corresponding to the i-th elite individual of this generation, then the mean parameter μ in the probability distribution of the target hyperparameter of this generation new and variance parameter σ new 2 The calculation method can be:

[0152]

[0153]

[0154] Wherein, ∈Δ represents a preset error value.

[0155] In this embodiment, the fitness value of each elite individual of this generation is different, and the weight of the hyperparameter data of each elite individual of this generation when used to update the probability distribution of the previous generation is also different. The hyperparameter data of the elite individuals of this generation are used to update the probability distribution of the previous generation by sampling and weighted averaging according to the weights of each elite individual of this generation. The hyperparameter data sampled according to the updated probability distribution of this generation is used as the hyperparameter data after the evolution of the disadvantaged individuals, so that the disadvantaged individuals can evolve towards the global optimal direction, which is more conducive to finding the optimal hyperparameter data and eliminating the influence of artificial factors of the prior distribution.

[0156] In one feasible implementation, the reinforcement learning hyperparameter optimization scheme provided by the embodiment of the present invention can be used to optimize the hyperparameter data in the policy network of the MERL model. MERL (Multi-Agent Evolutionary Reinforcement Learning) is a typical multi-agent evolutionary reinforcement algorithm. Most of the current MERL hyperparameter optimization work fixes the network structure and optimizes learning rate hyperparameters. However, the neural network structure determines the upper limit of the optimization of learning rate hyperparameters. The neural network structure is fixed only by artificial experience, ignoring the search space of network structure hyperparameters, and cannot reflect the impact of network structure on model performance. Therefore, it is necessary to use hyperparameter optimization methods to optimize the model network structure and explore the unexplored part of the hyperparameter exploration space. In the reinforcement learning hyperparameter optimization scheme provided by the embodiment of the present invention, the hyperparameter data may include at least the parameter values ​​corresponding to one or more hyperparameters in the number of neural network layers, the number of neurons in each layer, the learning rate, the type of active function, the discount factor, and the size of the experience replay pool, so that it can be used to optimize the network structure of the MERL model policy network. The algorithm architecture can refer to Figure 5 As shown, Figure 5 The MERL algorithm framework includes three agents, so each individual in the group includes three agents respectively. Each individual in the group trains the policy network by interacting with the environment. The hyperparameter optimization scheme provided in the embodiment of the present invention uses group screening and variation to explore the search space of neural network structure hyperparameters and optimize the network structure of the policy network.

[0157] In addition, an embodiment of the present invention further provides a device for optimizing hyperparameters of reinforcement learning, the device comprising:

[0158] An initialization module, used to initialize a group, each individual in the group is a reinforcement learning model with different hyperparameter data, and the hyperparameter data includes a parameter value corresponding to at least one hyperparameter;

[0159] A calculation module is used to calculate the fitness value of each individual in the group of the current generation after performing reinforcement learning training on each individual in the current generation during the evolution of any generation;

[0160] The first determination module is used to determine the elite individuals and the inferior individuals of this generation from each individual according to the fitness value of this generation of each individual;

[0161] An adjustment module, used to adjust the hyperparameter data corresponding to the elite individuals of the current generation by using the hyperparameter data corresponding to the optimal fitness value of the inferior individuals of the current generation in the evolution process of successive generations, and use the adjusted hyperparameter data as the hyperparameter data of the inferior individuals of the current generation after evolution in the current generation;

[0162] The second determination module is used to determine the target model from each individual in the group according to the fitness value of the individual in the group after performing multiple generations of evolution on each individual in the group.

[0163] In one feasible implementation, the reinforcement learning model uses task data as model input data and the scheduling result of the task as model output data. The target model is used to perform planning and scheduling based on the task data to obtain the scheduling result of the task. The calculation module is also used to:

[0164] For any target individual among the individuals, the task data of the target task in the preset training data set is input into the reinforcement learning model corresponding to the target individual for planning and scheduling, and the scheduling result of the target task is obtained;

[0165] Performing simulation on the target task according to the scheduling result to obtain the task achievement status of the target task;

[0166] An environmental reward is generated according to the task achievement status, and a reinforcement learning model corresponding to the target individual is optimized according to the environmental reward.

[0167] In one feasible implementation, the calculation module is further used for:

[0168] For any target individual among the individuals, the target individual is interacted with the environment multiple times to obtain multiple cumulative environmental reward expectations;

[0169] An average value of the multiple accumulated environmental reward expectations is determined as the current generation fitness value of the target individual, or a variance of the multiple accumulated environmental reward expectations is determined as the current generation fitness value of the target individual.

[0170] In a feasible implementation manner, the calculation module is further used to calculate the utilization strategy probability corresponding to the disadvantaged individual of the current generation according to the current generation fitness value of the disadvantaged individual of the current generation, wherein the higher the current generation fitness value of the disadvantaged individual of the current generation, the higher the utilization strategy probability corresponding to the disadvantaged individual of the current generation;

[0171] The reinforcement learning hyperparameter optimization device also includes:

[0172] A selection module, configured to select a target strategy from at least two preset variant strategies according to the utilization strategy probability, wherein the at least two variant strategies include the utilization strategy, and the higher the utilization strategy probability, the higher the possibility that the utilization strategy is selected;

[0173] The adjustment module is also used for: when the target strategy is the utilization strategy, executing the operation of adjusting the hyperparameter data corresponding to the elite individuals of this generation by using the hyperparameter data corresponding to the optimal fitness value of the disadvantaged individuals of this generation in the evolution process of successive generations.

[0174] In a feasible implementation, the at least two mutation strategies also include an exploration strategy, and the reinforcement learning hyperparameter optimization device also includes:

[0175] A sampling module is used to obtain the current generation probability distribution corresponding to the at least one hyperparameter in the current generation evolution when the target strategy is the exploration strategy, obtain hyperparameter data according to the current generation probability distribution sampling, and use the sampled hyperparameter data as the hyperparameter data of the disadvantaged individual of the current generation after the evolution of the current generation.

[0176] In a feasible implementation manner, the initialization module is further used for:

[0177] Initializing an initial probability distribution corresponding to the at least one hyperparameter;

[0178] According to the initial probability distribution, a plurality of different groups of the hyperparameter data are sampled, and each group of the hyperparameter data is used as the hyperparameter data of each reinforcement learning model in the group.

[0179] In one feasible implementation, the sampling module is further used for:

[0180] Obtaining a previous generation probability distribution corresponding to the at least one hyperparameter, wherein the previous generation probability distribution corresponding to the at least one hyperparameter in the first generation evolution process is the initial probability distribution;

[0181] The hyperparameter data of the elite individuals of the current generation are used to update the probability distribution of the previous generation, so as to obtain the probability distribution of the current generation corresponding to the at least one hyperparameter in the evolution of the current generation.

[0182] In a feasible implementation manner, when there are multiple elite individuals in the current generation, the sampling module is further used to:

[0183] Normalizing the fitness values ​​of the current generation of each elite individual of the current generation to obtain the weights corresponding to each elite individual of the current generation;

[0184] For any target hyperparameter among the at least one hyperparameter, weighted summing is performed on the parameter values ​​corresponding to the target hyperparameter in the hyperparameter data of each elite individual of the current generation using the weight, and the mean parameter in the probability distribution of the current generation corresponding to the target hyperparameter in the current generation evolution is obtained according to the calculation result;

[0185] Subtract the old mean parameter from the parameter value corresponding to the target hyperparameter in the hyperparameter data of each elite individual of the current generation, and then calculate the square to obtain the intermediate calculation results corresponding to each elite individual of the current generation, wherein the old mean parameter is the mean parameter in the probability distribution of the previous generation corresponding to the target hyperparameter;

[0186] The intermediate calculation results are weightedly summed using the weights, and the variance parameter of the probability distribution of the current generation corresponding to the target hyperparameter in the current generation evolution is obtained according to the calculation results.

[0187] In one feasible implementation, the hyperparameter data includes at least parameter values ​​corresponding to one or more hyperparameters of the number of neural network layers, the number of neurons in each layer, the learning rate, the activation function type, the discount factor, and the size of the experience replay pool.

[0188] In addition, the embodiment of the present invention also proposes a reinforcement learning hyperparameter optimization device, such as Figure 6 As shown, Figure 6 It is a schematic diagram of the device structure of the hardware operating environment involved in the embodiment of the present invention. It should be noted that the reinforcement learning hyperparameter optimization device in the embodiment of the present invention can be a smart phone, a personal computer, a server and other devices, and no specific limitation is made here.

[0189] like Figure 6 As shown, the reinforcement learning hyperparameter optimization device may include: a processor 1001, such as a CPU, a network interface 1004, a user interface 1003, a memory 1005, and a communication bus 1002. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0190] Those skilled in the art will understand that Figure 6 The device structure shown in the figure does not constitute a limitation on the reinforcement learning hyperparameter optimization device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0191] like Figure 6As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a reinforcement learning hyperparameter optimization program. The operating system is a program that manages and controls the hardware and software resources of the device, and supports the operation of the reinforcement learning hyperparameter optimization program and other software or programs. Figure 6 In the device shown, the user interface 1003 is mainly used for data communication with the client; the network interface 1004 is mainly used for establishing a communication connection with the server; and the processor 1001 can be used to call the reinforcement learning hyperparameter optimization program stored in the memory 1005 and perform the following operations:

[0192] Initialize a group, each individual in the group is a reinforcement learning model with different hyperparameter data, and the hyperparameter data includes a parameter value corresponding to at least one hyperparameter;

[0193] In any generation of evolution, each individual in the group of this generation is subjected to reinforcement learning training, and the fitness value of each individual in this generation is calculated;

[0194] Determine the elite individuals and inferior individuals of this generation from each individual according to the fitness value of this generation;

[0195] Using the hyperparameter data corresponding to the optimal fitness value of the inferior individual of this generation in the evolution process of successive generations to adjust the hyperparameter data corresponding to the elite individual of this generation, and using the adjusted hyperparameter data as the hyperparameter data of the inferior individual of this generation after evolution in this generation;

[0196] After multiple generations of evolution of each individual in the group, a target model is determined from each individual according to the fitness value of the individual in the group, and the target model is used to perform planning and scheduling based on the task data to obtain a scheduling result of the task.

[0197] In one feasible implementation, the reinforcement learning model uses task data as model input data and the scheduling result of the task as model output data. The target model is used to perform planning and scheduling based on the task data to obtain the scheduling result of the task. The operation of performing reinforcement learning training on each individual in the group of this generation includes:

[0198] For any target individual among the individuals, the task data of the target task in the preset training data set is input into the reinforcement learning model corresponding to the target individual for planning and scheduling, and the scheduling result of the target task is obtained;

[0199] Performing simulation on the target task according to the scheduling result to obtain the task achievement status of the target task;

[0200] An environmental reward is generated according to the task achievement status, and a reinforcement learning model corresponding to the target individual is optimized according to the environmental reward.

[0201] In one feasible implementation, the operation of calculating the fitness value of each individual in this generation includes:

[0202] For any target individual among the individuals, the target individual is interacted with the environment multiple times to obtain multiple cumulative environmental reward expectations;

[0203] An average value of the multiple accumulated environmental reward expectations is determined as the current generation fitness value of the target individual, or a variance of the multiple accumulated environmental reward expectations is determined as the current generation fitness value of the target individual.

[0204] In a feasible implementation manner, after the operation of determining the elite individuals and the inferior individuals of this generation from each individual according to the fitness value of this generation of each individual, the processor 1001 may also be used to call the reinforcement learning hyperparameter optimization program stored in the memory 1005 to perform the following operations:

[0205] Calculating the utilization strategy probability corresponding to the disadvantaged individual of the current generation according to the current generation fitness value of the disadvantaged individual of the current generation, wherein the higher the current generation fitness value of the disadvantaged individual of the current generation, the higher the utilization strategy probability corresponding to the disadvantaged individual of the current generation;

[0206] Selecting a target strategy from at least two preset variant strategies according to the utilization strategy probability, wherein the at least two variant strategies include the utilization strategy, and the higher the utilization strategy probability, the higher the possibility that the utilization strategy is selected;

[0207] When the target strategy is the utilization strategy, the operation of adjusting the hyperparameter data corresponding to the elite individuals of this generation by using the hyperparameter data corresponding to the optimal fitness values ​​of the inferior individuals of this generation in the evolutionary process of successive generations is performed.

[0208] In a feasible implementation manner, the at least two mutation strategies also include an exploration strategy. After the operation of selecting a target strategy from the at least two preset mutation strategies according to the strategy probability, the processor 1001 may also be used to call a reinforcement learning hyperparameter optimization program stored in the memory 1005 to perform the following operations:

[0209] When the target strategy is the exploration strategy, the probability distribution of the current generation corresponding to the at least one hyperparameter in the current generation evolution is obtained, and the hyperparameter data is obtained by sampling according to the current generation probability distribution, and the sampled hyperparameter data is used as the hyperparameter data of the disadvantaged individuals of the current generation after the evolution of the current generation.

[0210] In one feasible implementation, the operation of initializing the group includes:

[0211] Initializing an initial probability distribution corresponding to the at least one hyperparameter;

[0212] According to the initial probability distribution, a plurality of different groups of the hyperparameter data are sampled, and each group of the hyperparameter data is used as the hyperparameter data of each reinforcement learning model in the group.

[0213] In a feasible implementation manner, the operation of obtaining the probability distribution of the current generation corresponding to the at least one hyperparameter in the current generation evolution includes:

[0214] Obtaining a previous generation probability distribution corresponding to the at least one hyperparameter, wherein the previous generation probability distribution corresponding to the at least one hyperparameter in the first generation evolution process is the initial probability distribution;

[0215] The hyperparameter data of the elite individuals of the current generation are used to update the probability distribution of the previous generation, so as to obtain the probability distribution of the current generation corresponding to the at least one hyperparameter in the evolution of the current generation.

[0216] In a feasible implementation manner, when there are multiple elite individuals in the current generation, the operation of updating the previous generation probability distribution using the hyperparameter data of the elite individuals in the current generation to obtain the current generation probability distribution corresponding to the at least one hyperparameter in the current generation evolution includes:

[0217] Normalizing the fitness values ​​of the current generation of each elite individual of the current generation to obtain the weights corresponding to each elite individual of the current generation;

[0218] For any target hyperparameter among the at least one hyperparameter, weighted summing is performed on the parameter values ​​corresponding to the target hyperparameter in the hyperparameter data of each elite individual of the current generation using the weight, and the mean parameter in the probability distribution of the current generation corresponding to the target hyperparameter in the current generation evolution is obtained according to the calculation result;

[0219] Subtract the old mean parameter from the parameter value corresponding to the target hyperparameter in the hyperparameter data of each elite individual of the current generation, and then calculate the square to obtain the intermediate calculation results corresponding to each elite individual of the current generation, wherein the old mean parameter is the mean parameter in the probability distribution of the previous generation corresponding to the target hyperparameter;

[0220] The intermediate calculation results are weightedly summed using the weights, and the variance parameter of the probability distribution of the current generation corresponding to the target hyperparameter in the current generation evolution is obtained according to the calculation results.

[0221] In one feasible implementation, the hyperparameter data includes at least parameter values ​​corresponding to one or more hyperparameters of the number of neural network layers, the number of neurons in each layer, the learning rate, the activation function type, the discount factor, and the size of the experience replay pool.

[0222] In addition, an embodiment of the present invention also proposes a computer-readable storage medium, on which a reinforcement learning hyperparameter optimization program is stored. When the reinforcement learning hyperparameter optimization program is executed by a processor, the steps of the reinforcement learning hyperparameter optimization method as described above are implemented.

[0223] The various embodiments of the reinforcement learning hyperparameter optimization device and computer-readable storage medium of the present invention can all refer to the various embodiments of the reinforcement learning hyperparameter optimization method of the present invention, and will not be repeated here.

[0224] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0225] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0226] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0227] The above are only preferred embodiments of the present invention, and the protection scope of the present invention is not limited thereto. Any equivalent structure or equivalent process transformation made by using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the protection scope of the present invention.

Claims

1. A method for optimizing hyperparameters of reinforcement learning, characterized in that: The reinforcement learning hyperparameter optimization method comprises the following steps: Initialize a group, each individual in the group is a reinforcement learning model with different hyperparameter data, and the hyperparameter data includes a parameter value corresponding to at least one hyperparameter; In any generation of evolution, each individual in the group of this generation is subjected to reinforcement learning training, and the fitness value of each individual in this generation is calculated; Determine the elite individuals and inferior individuals of this generation from each individual according to the fitness value of this generation; Using the hyperparameter data corresponding to the optimal fitness value of the inferior individual of this generation in the evolution process of successive generations to adjust the hyperparameter data corresponding to the elite individual of this generation, and using the adjusted hyperparameter data as the hyperparameter data of the inferior individual of this generation after evolution in this generation; After multiple generations of evolution for each individual in the group, a target model is determined from each individual according to the fitness value of the individual in the group.

2. The method for optimizing hyperparameters of reinforcement learning according to claim 1, wherein: The reinforcement learning model uses task data as model input data and task scheduling results as model output data. The target model is used to perform planning and scheduling based on the task data to obtain the task scheduling results. The step of performing reinforcement learning training on each individual in the group of this generation includes: For any target individual among the individuals, the task data of the target task in the preset training data set is input into the reinforcement learning model corresponding to the target individual for planning and scheduling, and the scheduling result of the target task is obtained; Performing simulation on the target task according to the scheduling result to obtain the task achievement status of the target task; An environmental reward is generated according to the task achievement status, and a reinforcement learning model corresponding to the target individual is optimized according to the environmental reward.

3. The method for optimizing hyperparameters of reinforcement learning according to claim 1, wherein: The step of calculating the fitness value of each individual in this generation includes: For any target individual among the individuals, the target individual is interacted with the environment multiple times to obtain multiple cumulative environmental reward expectations; An average value of the multiple accumulated environmental reward expectations is determined as the current generation fitness value of the target individual, or a variance of the multiple accumulated environmental reward expectations is determined as the current generation fitness value of the target individual.

4. The method for optimizing hyperparameters of reinforcement learning according to claim 1, wherein: After the step of determining the elite individuals and the inferior individuals of this generation from each individual according to the fitness value of this generation of each individual, the method further includes: Calculating the utilization strategy probability corresponding to the disadvantaged individual of the current generation according to the current generation fitness value of the disadvantaged individual of the current generation, wherein the higher the current generation fitness value of the disadvantaged individual of the current generation, the higher the utilization strategy probability corresponding to the disadvantaged individual of the current generation; Selecting a target strategy from at least two preset variant strategies according to the utilization strategy probability, wherein the at least two variant strategies include the utilization strategy, and the higher the utilization strategy probability, the higher the possibility that the utilization strategy is selected; When the target strategy is the utilization strategy, the step of adjusting the hyperparameter data corresponding to the elite individuals of this generation by using the hyperparameter data corresponding to the optimal fitness values ​​of the inferior individuals of this generation in the evolutionary process of successive generations is performed.

5. The method for optimizing hyperparameters of reinforcement learning according to claim 4, wherein: The at least two mutation strategies also include an exploration strategy. After the step of selecting a target strategy from the at least two preset mutation strategies according to the strategy probability, the method further includes: When the target strategy is the exploration strategy, the probability distribution of the current generation corresponding to the at least one hyperparameter in the current generation evolution is obtained, and the hyperparameter data is obtained by sampling according to the current generation probability distribution, and the sampled hyperparameter data is used as the hyperparameter data of the disadvantaged individuals of the current generation after the evolution of the current generation.

6. The method for optimizing hyperparameters of reinforcement learning according to claim 5, wherein: The steps of initializing the group include: Initializing an initial probability distribution corresponding to the at least one hyperparameter; According to the initial probability distribution, a plurality of different groups of the hyperparameter data are sampled, and each group of the hyperparameter data is used as the hyperparameter data of each reinforcement learning model in the group.

7. The method for optimizing hyperparameters of reinforcement learning according to claim 6, wherein: The step of obtaining the probability distribution of the current generation corresponding to the at least one hyperparameter in the current generation evolution comprises: Obtaining a previous generation probability distribution corresponding to the at least one hyperparameter, wherein the previous generation probability distribution corresponding to the at least one hyperparameter in the first generation evolution process is the initial probability distribution; The hyperparameter data of the elite individuals of the current generation are used to update the probability distribution of the previous generation, so as to obtain the probability distribution of the current generation corresponding to the at least one hyperparameter in the evolution of the current generation.

8. The method for optimizing hyperparameters of reinforcement learning according to claim 7, wherein: In the case where there are multiple elite individuals in the current generation, the step of using the hyperparameter data of the elite individuals in the current generation to update the probability distribution of the previous generation to obtain the probability distribution of the current generation corresponding to the at least one hyperparameter in the current generation evolution includes: Normalizing the fitness values ​​of the current generation of each elite individual of the current generation to obtain the weights corresponding to each elite individual of the current generation; For any target hyperparameter among the at least one hyperparameter, weighted summing is performed on the parameter values ​​corresponding to the target hyperparameter in the hyperparameter data of each elite individual of the current generation using the weight, and the mean parameter in the probability distribution of the current generation corresponding to the target hyperparameter in the current generation evolution is obtained according to the calculation result; Subtract the old mean parameter from the parameter value corresponding to the target hyperparameter in the hyperparameter data of each elite individual of the current generation, and then calculate the square to obtain the intermediate calculation results corresponding to each elite individual of the current generation, wherein the old mean parameter is the mean parameter in the probability distribution of the previous generation corresponding to the target hyperparameter; The intermediate calculation results are weightedly summed using the weights, and the variance parameter of the probability distribution of the current generation corresponding to the target hyperparameter in the current generation evolution is obtained according to the calculation results.

9. The method for optimizing hyperparameters of reinforcement learning according to any one of claims 1 to 8, characterized in that: The hyperparameter data includes at least parameter values ​​corresponding to one or more hyperparameters of the number of neural network layers, the number of neurons in each layer, the learning rate, the activation function type, the discount factor, and the size of the experience replay pool.

10. A reinforcement learning hyperparameter optimization device, characterized in that: The reinforcement learning hyperparameter optimization device includes: a memory, a processor, and a reinforcement learning hyperparameter optimization program stored in the memory and executable on the processor. When the reinforcement learning hyperparameter optimization program is executed by the processor, the steps of the reinforcement learning hyperparameter optimization method according to any one of claims 1 to 9 are implemented.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a reinforcement learning hyperparameter optimization program, which, when executed by a processor, implements the steps of the reinforcement learning hyperparameter optimization method according to any one of claims 1 to 9.

Images