A priority hierarchical learning method

Abstract

This invention proposes a priority-based hierarchical learning method. For N tasks with different priorities, the evaluation function for task i is denoted as Q. i and maintain a prediction model π i For task i with priority, and all higher priority tasks j < i, prepare a predetermined threshold ε. ij and maintain a slack variable λ ij For any i > j, if Q j (π i )+ε ij <Q j (π j ), then represents π. i Performance on task j was better than π. j If the difference is too large, then increase λ. ij Conversely, λ decreases. ij But keep λ ij >0. With λ ij j < i is used as the weight to optimize π i Repeat the previous steps until convergence, and finally obtain π. N This is the desired model. In this invention, constraints are used to describe priorities, solving the problem of traditional multi-objective optimization methods lacking a priority order. Slack variables are introduced to automatically adjust the weights of each optimization objective. The dual variable is adaptively adjusted, resulting in zero duality with the primal problem, making it a convex optimization problem that can be solved quickly by existing solvers.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of machine learning, and particularly relates to a priority hierarchical learning method. BACKGROUND

[0002] Current machine learning methods are good at optimizing a single objective. However, when a task needs to optimize multiple objectives simultaneously, the effect is poor. Moreover, the trade-off relationship between different objectives is generally determined by a weight coefficient, and the effect of this method depends on the selection of hyperparameters, which brings more difficulties to training.

[0003] A multi-objective optimization problem refers to finding an output y that maximizes multiple evaluation functions Q i (x, y), i = 1, 2, …, N for an input x, where Q i is larger, the better the task i is completed. SUMMARY

[0004] To solve the above problems, the present application proposes a priority hierarchical learning method. This method can complete low-priority tasks as well as possible under the premise of ensuring the completion of high-priority tasks for multiple tasks with priority order. If low-priority tasks and high-priority tasks cannot be completed simultaneously, the low-priority tasks will be abandoned and the high-priority tasks will be completed as much as possible.

[0005] The present application contains a principle for solving a multi-objective optimization problem hierarchically according to priority, which contains:

[0006] Suppose there are N tasks with different priorities. If j < i, it means that the priority of task j is higher than that of task i. For a task with priority i, its evaluation function is denoted as Q i . For a task with priority i, a model to be trained π i is maintained.

[0007] For a task with priority i and all higher priority tasks j < i, a predetermined threshold ε ij is prepared.

[0008] After the optimization of all higher priority tasks is completed, for the model corresponding to priority j, its optimized result is denoted as The model with priority i is optimized using the following optimization problem:

[0009]

[0010]

[0011] Based on the above principle, the present application proposes a priority hierarchical learning method, the steps of which are as follows:

[0012] Step 1: Suppose there are N different priority tasks, for the priority i task, let its evaluation function be Q i , and maintain a prediction model π i .

[0013] Wherein, the evaluation function Q i is a mapping from model output to real number, for a model output value, the greater the value obtained by the evaluation function indicates the better the output completes the task. The evaluation function has various implementations, such as: using a deterministic function or using a neural network fitting, etc. The prediction model π i is a mapping from the input space to the output space, which receives the input quantity and returns the output value. The prediction model can be implemented by a neural network.

[0014] Step 2: For the priority i task, and all higher priority tasks j < i, a predetermined threshold ε ij is prepared, and a relaxation variable λ ij is maintained. Wherein, the relaxation variable is a non-negative real number.

[0015] Step 3: For any i > j, if Q j (π i )+ε ij <Q j (π j ), it means that π i performs worse than π j on task j, then increase λ ij ; otherwise, decrease λ ij , but keep λ ij > 0.

[0016] Step 4: Optimize π ij with λ i , j < i as weights:

[0017] Step 5: Repeat steps 3 and 4 until convergence, and the final π N is the model to be solved.

[0018] The beneficial effects of the present application are that the priority is described by constraint, which solves the problem that the traditional multi-objective optimization method has no priority order. The relaxation variable is introduced to automatically adjust the weights of each optimization target. The dual variable is adaptively adjusted. There is zero dual gap with the original problem, which is a convex optimization problem, and existing solvers can quickly solve it. DETAILED DESCRIPTION

[0019] The present application will be described in detail below with examples.

[0020] The above method is implemented in a reinforcement learning problem. This reinforcement learning problem requires the predictive model to control a robot to move as fast as possible in a simulation environment, while ensuring that the robot's body remains vertical and its energy consumption is low.

[0021] Therefore, this problem involves three tasks. The first task is to ensure the robot's body is upright; the second task is to control the robot to move forward as quickly as possible; and the third task is to reduce the robot's energy consumption.

[0022] The first task has the highest priority, and the third task has the lowest priority.

[0023] The steps of this method are as follows:

[0024] For task of priority i, initialize a policy model π to be trained. i and an evaluation function Q i It should be noted that in reinforcement learning problems, the evaluation function Q... i It also needs to be obtained through training, and the training method can be fitting the Bellman equation. To train Q... i The learning rate α of the evaluation function needs to be determined in advance. Qi and the rate of decay of returns γ i ∈(0,1).

[0025] For task of priority i, prepare a pre-determined policy model learning rate α. i and slack variable learning rate β i For task i with priority, and all higher priority tasks j < i, prepare a predetermined threshold ε. ij And maintain a slack variable λ ij .

[0026] During each training session, the policy model and evaluation function for each task are processed as follows:

[0027] 1. For a task of priority i, a set of input s is sampled. t Output a t , return r t .

[0028] 2. Update the evaluation function Q using conventional methods. i :

[0029] 3. Update the slack variable λ ij ←max(λ ij +β i (-Q j (π i (s t))-ε ij +Q j (π j (s t ))),

[0030]

[0031] 4. Updating the policy model

[0032] The above calculation is performed for each task. This is repeated until convergence. At this point, the lowest priority model π3 is the policy sought.

[0033] This embodiment is only a preferred implementation of the present application, but the protection scope of the present application is not limited to this, any person skilled in the art can easily think of changes or replacements within the technical scope disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A priority-based hierarchical learning method, characterized in that, The steps are as follows: Step 1: For N tasks with different priorities, for task i, let its evaluation function be Q. i and maintain a strategy model π i ; Strategy Model π i Control a robot to move as fast as possible in a simulation environment, while ensuring that the robot's body is upright and its energy consumption is low; In step 1, N is 3; the first task is to ensure the robot's body is upright; the second task is to control the robot to move forward as quickly as possible; the third task is to reduce the robot's energy consumption; the first task has the highest priority; the third task has the lowest priority. Step 2: For a task of priority i, and all higher priority tasks j < i, prepare a predetermined threshold ε ij , and maintain a slack variable λ ij ; Step 3: For any i>j, if Q j (π i )+ε ij j (π j ), then represents π. i Performance on task j was better than π. j If the difference is too large, then increase λ. ij Conversely, λ decreases. ij But keep λ ij >0;​ Step 4: Optimize π with λ ij , j < i as the weight i : For task of priority i, a set of inputs s is obtained by sampling. t Output a t , return r t ; Update the evaluation function Q using conventional methods. i : α Qi To evaluate the learning rate of the function, γ i For the rate of decay of the reward; Update slack variables β i The learning rate is a slack variable. Update strategy model α i The learning rate of the policy model; Step 5: Repeat steps 3 and 4 until convergence, and finally obtain π. N That is the model we are looking for.

2. The priority-based hierarchical learning method according to claim 1, characterized in that: The evaluation function Q i Use a defined function or fit using a neural network.

3. The priority-based hierarchical learning method according to claim 1, characterized in that: The strategy model π i Implemented using neural networks.

4. The priority-based hierarchical learning method according to claim 1, characterized in that: The slack variable λ ij It is a non-negative real number.

Images