Hyper-parameter determination method and device for allocation task solver, medium and product

Abstract

One or more embodiments of the invention provide a hyper-parameter determination method and device for an allocation task solver, a medium and a product. The method comprises the steps that a strategy function used for determining hyper-parameters of a distribution task solver serves as an intelligent agent in reinforcement learning, and the distribution task solver serves as an environment in reinforcement learning; and taking a hyper-parameter determined by the strategy function as an action in reinforcement learning, so that a distribution task solver solves a plurality of distribution tasks in the first set based on the hyper-parameter to obtain a solving result, calculates a state and an award corresponding to reinforcement learning based on the solving result, and maximizes the award as an optimization target. Carrying out reinforcement learning training on the strategy function; and based on the trained strategy function, determining a target hyper-parameter of the allocation task solver.

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the field of computer software technology, and in particular, to a method for determining hyperparameters of a distribution task solver, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] Decisions on task allocation are ubiquitous in social life. For example, in a logistics supply chain scenario, when there are multiple logistics centers that store or produce goods and multiple sales points, it is necessary to decide how to distribute the goods from the logistics center (such as a warehouse or production site) to each sales point (such as a retail store or customer) to minimize transportation costs and time. Another example is in a resource allocation scenario. In the manufacturing industry, resources such as production equipment, raw materials, or labor need to be allocated to different production lines to ensure a smooth production process and maximize output. By allocating resources in the most efficient way, overhead can be reduced and resource utilization can be maximized, thereby increasing profits and customer satisfaction.

[0003] In the related art, the distribution task solver can be used to solve the distribution task so as to obtain the distribution result of the distribution task. Before using the distribution task solver to solve the distribution task, it is necessary to first determine the hyperparameters in the distribution task solver. At present, the hyperparameters in the distribution task solver are usually manually set by the staff based on experience, which requires a lot of time for experience tuning, and the efficiency of hyperparameter determination is low. Summary of the invention

[0004] In view of this, one or more embodiments of the present specification provide a method for determining hyperparameters of a distribution task solver, an electronic device, a computer-readable storage medium, and a computer program product.

[0005] To achieve the above objectives, one or more embodiments of this specification provide the following technical solutions:

[0006] According to a first aspect of one or more embodiments of this specification, a method for determining hyperparameters of a distribution task solver is proposed, comprising:

[0007] Using a policy function for determining hyperparameters of an assignment task solver as an agent in reinforcement learning, and using the assignment task solver as an environment in reinforcement learning; wherein the assignment task solver is used to solve a plurality of assignment tasks in a first set to determine a target item to be assigned to a target object indicated by each of the assignment tasks from a plurality of candidate items;

[0008] Using the hyperparameters determined by the policy function as actions in the reinforcement learning, so that the assigned task solver solves the multiple assigned tasks in the first set based on the hyperparameters to obtain solution results, and calculates the state and reward corresponding to the reinforcement learning based on the solution results, and performs reinforcement learning training on the policy function with maximizing the reward as the optimization goal;

[0009] Based on the policy function after training, target hyperparameters of the assigned task solver are determined.

[0010] According to a second aspect of the embodiments of this specification, an electronic device is provided, including:

[0011] processor;

[0012] a memory for storing processor-executable instructions;

[0013] When the processor executes the executable instructions, it is used to implement the method described in the first aspect.

[0014] According to a third aspect of the embodiments of this specification, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the steps of any of the above methods are implemented.

[0015] According to a fourth aspect of the embodiments of this specification, a computer program product is provided, including a computer program, which implements the steps of any of the above methods when executed by a processor.

[0016] The technical solutions provided by the embodiments of this specification may have the following beneficial effects:

[0017] In the embodiments of the present specification, the hyperparameter selection problem is introduced into the reinforcement learning framework, the policy function used to determine the hyperparameters of the allocation task solver is regarded as the intelligent agent in reinforcement learning, and the allocation task solver is regarded as the environment. The intelligent agent is trained to automatically select reasonable hyperparameters, which improves the efficiency of determining the hyperparameters. Training the intelligent agent to maximize the reward is also conducive to improving the accuracy of the hyperparameters.

[0018] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present specification. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a flowchart of a method for determining hyperparameters of a distribution task solver provided by an exemplary embodiment.

[0020] Figure 2 is a schematic diagram of a reinforcement learning framework provided by an exemplary embodiment.

[0021] Figure 3 It is a flowchart of a reinforcement learning training process provided by an exemplary embodiment.

[0022] Figure 4 is a schematic diagram of another reinforcement learning framework provided by an exemplary embodiment.

[0023] Figure 5 It is a schematic structural diagram of an electronic device provided by an exemplary embodiment. DETAILED DESCRIPTION

[0024] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with one or more embodiments of this specification. Instead, they are merely examples of devices and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0025] It should be noted that: in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this specification may be combined into a single step for description in other embodiments.

[0026] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this manual are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0027] The following is an explanation of the relevant terms in the embodiments of this specification:

[0028] 1. Reinforcement Learning (RL) is a machine learning method, the core of which is to enable the agent to optimize its behavior strategy in a specific environment through continuous trial and learning, in order to maximize the cumulative reward.

[0029] An agent is an entity that makes decisions in a reinforcement learning scenario. It can be a robot, a software algorithm, or any system that can perceive the environment and take actions.

[0030] The environment is the external world or situation in which the agent is located, which gives feedback (usually rewards or penalties) based on the agent's actions.

[0031] State (S) is a feature or information that describes the environment at any moment, and the agent decides actions based on the current state.

[0032] Action (A) is an operation or decision that an agent can perform in a given state. Actions affect the environment and lead to state transitions.

[0033] Reward (R) is the feedback given by the environment to the agent after it takes an action. It is a direct indicator of the quality of the action. Rewards can be given immediately or delayed to a certain point in the future.

[0034] A policy (π) is a rule or strategy for an agent to choose actions, which defines what action the agent should choose to achieve its goal in a given state. A policy can be deterministic or stochastic (i.e., policy π(s) = a, meaning take action a in state s).

[0035] 2. ADMM (Alternating Direction Method of Multipliers) is an optimization algorithm that is widely used to solve large-scale and distributed optimization problems. Its basic idea is to decompose a complex optimization problem into several simpler sub-problems, calculate each sub-problem separately, and then summarize and evaluate it, broadcast the updated status to each sub-problem, and repeat these steps to iterate until the optimal solution is reached.

[0036] 3. PJ-ADMM (Proximal Jacobian Alternating Direction Method of Multipliers) is an improved alternating direction multiplier method, which is mainly used to solve optimization problems with non-smooth terms. It combines proximal operations and Jacobi methods, making it more flexible in dealing with complex constraints.

[0037] Linear Assignment Problems (LAPs) such as resource allocation, task scheduling, and vehicle routing are frequently encountered in various industrial scenarios. Solving LAPs efficiently is crucial for optimizing resources and enhancing decision-making processes in complex systems. By allocating resources in the most efficient way, indirect costs can be reduced and resource utilization can be maximized, thereby improving profitability and customer satisfaction.

[0038] For example, in a logistics supply chain scenario, when there are multiple logistics centers that store or produce goods and multiple sales points, it is necessary to decide how to distribute the goods from the logistics center (such as a warehouse or production site) to each sales point (such as a retail store or customer) to minimize transportation costs and time.

[0039] In resource allocation scenarios, (1) for example, in cloud computing platforms, server resources such as CPU, memory, storage, etc. need to be allocated to different tasks or users; the linear allocation problem can be used to determine how to allocate limited computing resources to users to ensure that resource utilization is maximized while meeting user needs. (2) For example, in manufacturing, resources such as production equipment, raw materials, or labor need to be allocated to different production lines to ensure a smooth production process and maximize output.

[0040] In route planning scenarios, (1) for example, a logistics company needs to design optimal delivery routes for multiple packages to ensure that the goods are delivered on time and reduce transportation costs; the linear assignment problem can help determine which areas or customers each delivery vehicle should be responsible for in order to achieve optimal route planning. (2) For example, in urban traffic management, taxis need to be reasonably dispatched to the passenger's requested location; the linear assignment problem can be used to determine which taxi will pick up which passenger, so as to meet passenger needs in the shortest time and improve taxi utilization.

[0041] As can be seen, linear assignment problems (LAPs) involve optimizing the assignment of users (I) to a set of candidate projects (J). In these problems, each user must be assigned to a project while taking into account certain constraints and objectives. These constraints usually involve limited resources such as machines, vehicles, time periods, or budgets. The goal of LAPs can be to maximize profit or minimize risk. For example, the basic form of the linear assignment problem can be represented by the decision variables x ij To express, x ij represents the amount of user i assigned to project j, and its value is between [0,1]. The constraint is that each user i must be completely assigned to a certain project j, that is, the domain of x can be expressed as In addition, considering the complexity of industrial-scale problems, the problem is decomposed into K sub-problems, where Each subproblem k contains n decision variables. k , the optimization objective of the problem can be expressed as:

[0042]

[0043] Among them, c k is the linear weight vector of subproblem k; A k is the constraint coefficient of subproblem k; b is a constant vector representing resource constraints; x kis a subset of decision variables, and the optimization objective of this problem is to minimize the risk.

[0044] It can be understood that the above basic form of the linear assignment problem is only an example and does not constitute a limitation on the representation of the linear assignment problem. The relevant linear assignment problem can be specifically defined according to the actual application scenario.

[0045] Exemplarily, the PJ-ADMM solver can be used to solve the above-mentioned block linear optimization problem. PJ-ADMM (Proximal Jacobian Alternating Direction Method of Multipliers) provides a distributed computing method, which uses an augmented Lagrangian function and increases proximal terms to ensure convergence and improve the efficiency of parallel computing.

[0046] The augmented Lagrangian function has the form:

[0047]

[0048] Among them, λ is the dual variable; ρ is the hyperparameter of the PJ-ADMM solver, that is, the penalty parameter, ρ>0, which is used to enhance the penalty strength of the constraint.

[0049] To ensure stability and accelerated convergence during each iteration, the PJ-ADMM solver introduces a proximal term in the augmented Lagrangian function. The function of this proximal term is to limit the change of the solution in each iteration and prevent unstable large jumps.

[0050] The iterative process of the PJ-ADMM solver in solving the above linear assignment problem is as follows:

[0051] 1. Original solution x k Parallel update: For each subproblem k=1,2,…,K, update the original solution x in parallel k :

[0052]

[0053] The purpose of this step is to minimize the objective function of each subproblem when the current dual variables and the solutions of other subproblems are known. To limit the change of each solution step, prevent unstable large jumps, and thus ensure the stability of the algorithm. Among them, τ is the hyperparameter of the PJ-ADMM solver, that is, the proximal term parameter, τ>0. It represents the solution obtained by the current round of iteration. Represents the solution obtained by the previous iteration process.

[0054] 2. Update of the dual solution λ:

[0055] λ t+1 represents the solution obtained by the current round of iteration, λ t Represents the solution obtained by the previous iteration. This step is to update the dual variables to ensure that the final solution gradually satisfies the constraint function.

[0056] In the related art, the allocation task solver can be used to solve the allocation task in order to obtain the allocation result of the allocation task. Before using the allocation task solver to solve the allocation task, it is necessary to determine the hyperparameters in the allocation task solver. At present, the hyperparameters in the allocation task solver are usually set manually by the staff based on experience, which requires a lot of time for experience tuning and has low determination efficiency. The setting of hyperparameters has a great influence on the convergence efficiency of the allocation task solver. As in the above-mentioned PJ-ADMM solver, the selection of the penalty parameter ρ and the proximal term parameter τ has a significant effect on the convergence of the allocation task. Reasonable selection of these hyperparameters can greatly improve the efficiency and stability of the algorithm.

[0057] Based on this, see Figure 1 and Figure 2 The embodiment of this specification provides a method for determining hyperparameters of a task allocation solver based on reinforcement learning, which can be performed by an electronic device, including but not limited to a physical server, a server cluster, a cloud server, a smart phone / mobile phone, a tablet computer, a personal digital assistant (PDA), a laptop computer, and a desktop computer. The method includes:

[0058] In S101, a policy function for determining hyperparameters of an assignment task solver is used as an agent in reinforcement learning, and the assignment task solver is used as an environment in reinforcement learning; wherein the assignment task solver is used to solve multiple assignment tasks in a first set to determine a target item to be assigned to a target object indicated by each assignment task from multiple candidate items.

[0059] For example, in the reinforcement learning framework, the policy function is equivalent to an agent, which is responsible for taking actions (i.e., selecting hyperparameters) in different states. The assignment task solver is modeled as an environment, and its function is to solve a series of assignment tasks based on the input hyperparameters and output the corresponding solution results. The assignment task solver needs to find the preferred target project for each target object indicated by the assignment task from multiple candidate projects. This process involves the optimization of multiple constraints and objective functions, such as maximizing benefits or minimizing risks.

[0060] In S102, the hyperparameters determined by the policy function are used as actions in reinforcement learning, so that the assigned task solver solves the multiple assigned tasks in the first set based on the hyperparameters to obtain solution results, and calculates the state and reward corresponding to the reinforcement learning based on the solution results, and performs reinforcement learning training on the policy function with maximizing the reward as the optimization goal.

[0061] In this step, in reinforcement learning, the agent selects hyperparameters (e.g., penalty parameter ρ and proximal term parameter τ in the PJ-ADMM solver) as actions based on the policy function. Each selected hyperparameter affects the convergence performance and solution efficiency of the allocation task solver. The allocation task solver solves multiple allocation tasks based on the selected hyperparameters and outputs the solution results, which may include the preferred solution for resource allocation. Then, the electronic device calculates the state and reward in reinforcement learning based on the solution results; the state may include the current convergence of the solver, while the reward measures the performance of the solver, such as the solution speed, convergence quality, etc. By adjusting the hyperparameters to maximize the reward, the agent can optimize its policy function.

[0062] The key to reinforcement learning is to gradually optimize the policy function through iterative learning, so that the agent can select the optimal hyperparameter combination, thereby improving the performance of the distribution task solver. Compared with traditional methods, reinforcement learning can adaptively adjust parameters, improve convergence speed and enhance the stability of the distribution task solver.

[0063] In S103, based on the trained policy function, target hyperparameters of the allocation task solver are determined.

[0064] In this step, after the training is completed, the policy function can be used to determine the target hyperparameters of the solver, that is, to automatically select appropriate hyperparameters in the actual allocation task solving process.

[0065] In this embodiment, the hyperparameter selection problem is introduced into the reinforcement learning framework, and reasonable hyperparameters are automatically selected by training the intelligent agent. The traditional manual parameter adjustment process is time-consuming and inefficient because it relies on expert experience. This method reduces this dependence through reinforcement learning, improves the degree of automation of hyperparameter selection, and improves the efficiency of hyperparameter determination. This method is suitable for industrial scenarios that require efficient and stable solutions, and can significantly improve the intelligence level of the distribution task solver.

[0066] In some embodiments, the assigned tasks in the first set are obtained by sampling the assigned tasks in the second set; through mathematical derivation, it can be known that the hyperparameters selected for the first set training can be generalized to the second set after being scaled by the quantitative ratio between the first set and the second set. That is to say, after the policy function training is completed, the electronic device can scale the hyperparameters determined by the policy function after the training is completed based on the quantitative ratio between the first set and the second set to obtain the target hyperparameters of the assigned task solver for solving the second set. This embodiment saves computing resources and time and improves overall efficiency by performing reinforcement learning training on the first set.

[0067] Exemplarily, the allocation task solver includes a PJ-ADMM solver; the hyperparameters of the PJ-ADMM solver include a penalty parameter and a proximal term parameter; the electronic device can determine the target penalty parameter of the allocation solver for solving the second set based on the product of the penalty parameter determined by the policy function after training and the scaling ratio; and determine the proximal term parameter determined by the policy function after training as the target proximal term parameter of the allocation solver for solving the second set.

[0068] Exemplarily, the allocation task solver includes an ADMM solver; the hyperparameters of the ADMM solver include a penalty parameter; the electronic device can determine the target penalty parameter of the allocation solver for solving the second set based on the product of the penalty parameter determined by the strategy function after training and the scaling ratio.

[0069] Of course, the assigned task solver may also be other types of solvers, and this embodiment does not impose any limitation on this.

[0070] In some embodiments, the reinforcement learning training process is exemplarily described below:

[0071] See also Figure 3 and Figure 4 , Figure 4 In the figure, the distribution task solver is used as the PJ-ADMM solver for exemplary explanation, i represents the i-th round of training process, i>0, and the electronic device can cycle the following training process to train the policy function with the optimization goal of maximizing the reward in reinforcement learning:

[0072] In S301, the hyperparameters determined by the policy function are obtained and fed back to the allocation task solver.

[0073] In the reinforcement learning framework, the agent (i.e., the policy function, Policyπ θ )According to the current state i )Choose hyperparameters (e.g., Figure 4The penalty parameter ρ and the proximal term parameter τ in the PJ-ADMM solver shown in FIG. 4 are shown in FIG. 4 ). These hyperparameters will directly affect the solution effect of the allocation task solver. The selected hyperparameters will be input into the allocation task solver to guide it to solve multiple allocation tasks.

[0074] It is understandable that in the first round of reinforcement learning iterations, since the policy function has not been trained, it is impossible to make reasonable hyperparameter selections based on past experience. Therefore, the preset hyperparameters can be fed back to the allocation task solver. The preset hyperparameters can be manually set experience values ​​or some default initial parameters. The allocation task solver will run according to these hyperparameters, generate the corresponding solution results, and then determine the first state feedback data in the reinforcement learning process based on this solution result for subsequent policy function training. Using preset hyperparameters allows the reinforcement learning framework to quickly enter the training state, and through subsequent iterative optimization of hyperparameters, gradually replace the work of manual parameter adjustment, and finally find a hyperparameter configuration suitable for a specific allocation task.

[0075] In S302, a solution result obtained by the allocation task solver according to the hyperparameters for solving the multiple allocation tasks in the first set is obtained.

[0076] The allocation task solver solves multiple allocation tasks based on the hyperparameters in step S301 to obtain a solution result. Exemplarily, the allocation task includes an objective function and a constraint function for a decision variable; the solution result is a solution of the decision variable in the allocation task, indicating the allocation amount of the target object allocated to each candidate project.

[0077] Objective functions are usually used to optimize certain aspects of decision variables. For example, the optimization goal of the objective function is to maximize the benefits that the target object can bring after being assigned to the target project and / or minimize the risks that the target object needs to bear after being assigned to the target project. Constraint functions are used to describe the constraints that decision variables should satisfy. In order to ensure reasonable allocation, the constraints must be met. For example, resources cannot exceed the available quantity, or each task can only be assigned once. These constraints are used to limit the solution space of the task allocation solver to ensure the feasibility of the solution.

[0078] The allocation task solver runs the algorithm based on the hyperparameters and finally generates a solution for the decision variables. These solutions represent specific allocation schemes, that is, how the target objects are allocated to each candidate project (for example, the amount of resources allocated or the number of tasks assigned). These solutions must not only meet the constraints, but also optimize the benefits and / or risks under the given objective function. Therefore, these solution results can provide a basis for practical decision-making and help make good choices in complex scenarios.

[0079] The task allocation solver in this step provides real-time feedback on the solution results, allowing the reinforcement learning system to dynamically evaluate the effect of hyperparameter selection and provide data support for subsequent training.

[0080] Among them, in order to prevent the convergence of the assigned task solver from taking too long and affecting the overall training efficiency, a maximum number of iterations of the assigned task solver in the solution process can be set. When starting the assigned task solver, a maximum iteration threshold is set. When the iteration number of the assigned task solver reaches the maximum iteration threshold, even if it has not fully converged, the electronic device will force the current solution process to end and return the current solution result. For example, the maximum number of iterations is set to 1000 times, which means that the assigned task solver stops solving after a maximum of 1000 iterations. Of course, if during the iteration process, the assigned task solver reaches the convergence standard (such as the relative residual and relative target error have reached the preset accuracy requirements), the solution can be terminated in advance and the subsequent redundant iteration steps can be skipped. By controlling the number of iterations of a single solution, training can be performed on more assigned tasks, thereby speeding up the overall efficiency of the reinforcement learning process and improving the optimization speed of hyperparameter adjustment.

[0081] In S303, quality assessment is performed on the solution result to obtain a quality assessment result.

[0082] In this step, the quality assessment objectively analyzes the solution results of the task allocation solver to help the electronic device understand the pros and cons of the current hyperparameter settings. The quality assessment results provide a basis for subsequent reinforcement learning. Accurate quality assessment ensures that the electronic device can effectively judge the performance of the current policy function and provide precise direction for subsequent optimization.

[0083] Exemplarily, the allocation task includes an objective function and a constraint function for the decision variables. The electronic device can determine the relative residual of the allocation task solver in this solution process based on the constraint function and the solution result. The relative residual is an indicator to measure the constraint satisfaction in the solution process. In addition, the electronic device can determine the relative target error of the allocation task solver in this solution process based on the preset optimal solution of the objective function and the solution result. The relative target error is used to measure the degree of deviation of the solution result relative to the optimal solution of the objective function. Among them, the preset optimal solution of the objective function is obtained by solving multiple allocation tasks in the first set in advance based on other solvers.

[0084] The quality assessment results include at least one of the relative residual and the relative target error. If the relative residual of the solution result is large, it means that the allocation task solver fails to meet the constraints well; if the relative target error is large, it means that the allocation task solver does not find a solution close to the optimal solution. Therefore, quality assessment can help further improve and adjust the hyperparameters to obtain better results.

[0085] For the relative residual, the electronic device processes the solution result based on the constraint function to obtain the residual of the assigned task solver in this solution process. If the residual is zero, it means that the solution result fully meets the requirements of the constraint function; if the residual is not zero, it means that the solution result fails to strictly meet the constraint conditions. Finally, the electronic device determines the ratio between the residual of the assigned task solver in this solution process and the constraint value indicated by the constraint function as the relative residual of the assigned task solver in this solution process. In this embodiment, by calculating the relative residual, the electronic device can quantitatively evaluate the extent to which the solution result deviates from the constraint conditions, which is not only an important basis for judging the quality of the solution, but also helps to find out whether the solver has problems under certain constraints.

[0086] Continuing with the above example, assume that the constraint function for assigning tasks is The residual is used to measure the degree of deviation between the current solution and the constraints. The residual is defined as: primal =|Ax-b|; where ξ primal represents the residual, x is the solution result of the task solver in this solution process, A is the constraint coefficient of the constraint function, and b is the constraint value indicated by the constraint function. primal When it is zero, it means that the current solution completely satisfies the constraints; the larger the residual, the more the current solution deviates from the constraints. In order to maintain consistency in problems of different sizes, the residual is standardized to the relative residual ξ rel , expressed as: Among them, ||ξ primal || ∞ is the residual ξ primak The infinite norm of ||b|| ∞ is the infinity norm of b, which is used to scale the residual values ​​so that the residuals of different problem sizes are comparable.

[0087] For the relative target error, the electronic device can process the solution result based on the objective function to obtain the target value of the distribution task solver in this solution process, which represents the performance of the current solution in terms of maximizing benefits or minimizing risks. And based on the objective function, the preset optimal solution is processed to obtain the optimal target value, which represents the maximum benefit or minimum risk that the objective function can achieve under ideal conditions, that is, the target value corresponding to the optimal solution of the problem. Then the difference between the target value and the optimal target value of the distribution task solver in this solution process is taken as the target error. If the target error is zero, it means that the current solution result has reached the preset optimal target value; if the target error is not zero, it means that the current solution result has a certain deviation from the optimal solution. Then the ratio between the target error and the optimal target value of the distribution task solver in this solution process is taken as the relative target error of the distribution task solver in this solution process. In this embodiment, through the relative target error, the electronic device can evaluate the degree of deviation of the current solution result from the optimal solution in terms of target optimization, thereby judging the quality of the solution.

[0088] Continuing with the above example, assume that the objective function of assigning tasks is Then the relative target error can be expressed as: Among them, f acc represents the relative target error, f(x) represents the target value of the task solver in this solution process, and f * (x) represents the optimal target value. The closer the target error is to zero, the closer the target value of the current solution is to the optimal target value; the larger the target error is, the greater the degree to which the current solution deviates from the optimal solution.

[0089] In S304, the state in the current training process corresponding to the reinforcement learning is determined based on the quality evaluation result.

[0090] In the reinforcement learning framework, relative residual and relative target error can be used to construct part of the state of the task assignment solver. These error information helps the agent understand the quality of the current solution and select appropriate hyperparameters based on this information to accelerate convergence. The relative residual reflects whether the solution satisfies the constraints. The agent can ensure that the solution gradually approaches the constraints by reducing the residual. The relative target error reflects whether the solution is close to the optimal solution. The agent can minimize the target error by adjusting the strategy.

[0091] For example, considering that bad hyperparameters will cause the iterative process of the task allocation solver to diverge, the embodiment of this specification sets a rollback mechanism. Once the specified evaluation criteria are exceeded, the environment will reset the current update and give the agent another chance to calculate a new action. Specifically, please refer to Figure 4When determining the state corresponding to the reinforcement learning in this training process, the electronic device evaluates whether the quality evaluation result meets the preset first quality condition. If the quality evaluation result meets the preset first quality condition, the electronic device will consider that the current hyperparameter configuration is reasonable and the convergence process is stable, and then determine the state corresponding to the reinforcement learning in this training process based on the quality evaluation result (s i+1 ) and use it for the next step of reinforcement learning training. If the quality evaluation result does not meet the first quality condition, the electronic device determines that the current solution process may have diverged or the hyperparameter configuration is not ideal, and then executes the rollback strategy. The rollback strategy indicates that the state in the previous training process is used as the state in the current training process corresponding to the reinforcement learning (s i+1 =s i ). In this embodiment, the rollback mechanism can detect the divergence caused by improper hyperparameters in time, and avoid further expansion of the divergence by resetting the state, ensuring that the task solver is iterated within a reasonable convergence range as much as possible. Whenever a rollback occurs, the intelligent body will readjust the hyperparameters based on the last valid state. This mechanism speeds up the exploration and adjustment of hyperparameters, reduces the number of invalid solutions, and thus improves the overall training efficiency.

[0092] Among them, the first quality condition can be specifically set according to the actual application scenario, and this embodiment does not impose any restrictions on this. For example, the first quality condition includes at least one of the following: the relative residual is lower than a preset threshold and the relative target error is less than a preset error.

[0093] In order to prevent excessive rollbacks from affecting the overall training efficiency, the number of consecutive rollbacks can be limited. For example, a rollback counter can be introduced to record the number of consecutive rollbacks and set a preset maximum number of rollbacks. When the number of rollbacks reaches the upper limit, the electronic device will no longer execute the rollback strategy and will be forced to proceed to the next step to avoid falling into a rollback cycle.

[0094] The function of the rollback counter is to record the number of continuous rollbacks. When the electronic device determines that the solution of the current training process does not meet the preset first quality condition, the value of the counter will be accumulated after each rollback strategy is executed, and the counter will increase by 1 until the preset number is reached. By limiting the number of rollbacks, the electronic device can tolerate poor selection of hyperparameters within a certain range and correct errors through rollbacks. At the same time, limiting the number of rollbacks can avoid over-correction and ensure the stability of the training process.

[0095] If the quality assessment result meets the first quality condition, it means that the current hyperparameter setting is valid and the solution process is carried out within a reasonable range. In this case, the electronic device will update the state according to the solution result and reset the rollback counter to 0. If the quality assessment result does not meet the first quality condition and the number of continuous rollbacks does not exceed the preset number, the electronic device will continue to execute the rollback strategy, and after the rollback is completed, the count value of the rollback counter will be increased by one. If the quality assessment result does not meet the first quality condition and the number of continuous rollbacks exceeds the preset number, that is, when the value of the rollback counter reaches the predetermined threshold, the electronic device determines the state corresponding to the reinforcement learning in this training process based on the quality assessment result, that is, it forces the state to be updated and enter the next round of training, which prevents the system from falling into a loop due to continuous rollbacks and ensures the overall advancement of the training. In this embodiment, under the premise of not exceeding the upper limit of the number of rollbacks, the electronic device can flexibly adjust the hyperparameters and continuously optimize the performance of the solver based on the feedback results, which gives the agent more opportunities to explore suitable hyperparameters. Once the number of rollbacks reaches the upper limit, the electronic device will no longer correct the current step, but continue to move forward to ensure that the reinforcement learning training is not interrupted by excessive rollbacks, thereby maintaining the overall training progress.

[0096] The following is an exemplary explanation of determining the state in the current training process corresponding to reinforcement learning based on the quality evaluation results: the electronic device can determine the evaluation information for the current solution process based on the hyperparameters, relative residuals and relative target errors in the current training process; and then determine the evaluation information for the current solution process and the evaluation information for at least one historical solution process as the state in the current training process corresponding to reinforcement learning.

[0097] Exemplarily, the hyperparameters, relative residuals and relative target errors in this training process can be directly used as evaluation information for this solution process, or the hyperparameters, relative residuals and relative target errors in this training process can be further processed to obtain evaluation information for this solution process. This implementation does not impose any restrictions on this.

[0098] When a rollback mechanism is set, the state corresponding to the reinforcement learning in this training process also includes the number of consecutive rollbacks corresponding to this training process.

[0099] For example, the hyperparameters in the PJ-ADMM solver include the penalty parameter ρ and the proximal term parameter τ. After the i-th round of training, the evaluation information h i+1 It can be expressed as:

[0100] in, x represents ρ i and τ irepresents the two hyperparameters used in the i-th round of training process, are the relative residual error and relative target error calculated in the i-th round of training process respectively.

[0101] Then the state s in this training process corresponding to reinforcement learning is i+1 =[T i h i+1 h i … h i-l ]; where T i Indicates the number of consecutive rollbacks corresponding to this training process, h i …h i-l is the evaluation information for at least one historical solution process, l≥1.

[0102] In S305, the quality assessment result is processed using a preset reward function to obtain a reward corresponding to the reinforcement learning in this training process (such as Figure 4 Reward i ).

[0103] In this step, the quality assessment results are processed according to the preset reward function to generate corresponding rewards. The size of the reward depends on the quality of the solution. Excellent solution results will receive higher rewards, while poor results will receive lower or even negative rewards. The reward function can guide the agent to adjust the policy function in the right direction, prompting it to gradually find the optimal hyperparameter combination and improve the overall efficiency of the solver.

[0104] Exemplarily, the preset reward function is the product of the reward coefficient and the reward sub-function, which can be expressed as p=r*g, where p represents the reward value, r represents the reward coefficient, and g represents the reward sub-function. The reward sub-function is used to describe the processing method for the quality evaluation result.

[0105] The reward coefficient includes the first reward coefficient, the second reward coefficient and the third reward coefficient; among them, the first reward coefficient is a positive number, indicating a positive reward, which is used to encourage the agent to make the correct hyperparameter selection when the preset conditions are met. The second reward coefficient and the third reward coefficient are both negative numbers, and the second reward coefficient is greater than the third reward coefficient; the second reward coefficient indicates a slight penalty, which is used to give a slight penalty to the agent when the hyperparameter selection is poor but does not cause a rollback; the third reward coefficient indicates a heavy penalty, which gives a larger penalty to the agent when the hyperparameter selection is poor and causes a rollback.

[0106] In the process of calculating the reward corresponding to reinforcement learning in this training process, the electronic device first processes the quality assessment result based on the reward sub-function to obtain an intermediate result. Specifically, the reward sub-function is a piecewise function including a first sub-function and a second sub-function; the first sub-function is used to process the situation where the quality assessment result is within the range of an exact solution (i.e., a very high solution accuracy). The second sub-function is used to process the situation where the quality assessment result is within the range of a feasible solution, i.e., when the result does not meet the criteria for an exact solution, an intermediate result is calculated. The quality assessment result represents the gap between the solution result and the ideal result in this training process.

[0107] In the process of calculating the intermediate results, if the quality assessment result is less than or equal to the preset first quality assessment value, it means that the solution result reaches the standard of the exact solution, that is, it is very close to the preferred solution result, then the electronic device determines that the intermediate result is 0 based on the first sub-function; the purpose of this processing is that when the result is already very accurate, no positive or negative reward will be given to it, and a neutral state will be maintained.

[0108] If the quality assessment result is greater than the first quality assessment value, it means that the solution result is not accurate enough and may be within the range of feasible solutions. At this time, it is necessary to calculate the intermediate result based on the second sub-function. The electronic device processes the quality assessment result based on the second sub-function to obtain the intermediate result; wherein the second sub-function is used to instruct the electronic device to calculate the difference between the logarithmic value of the quality assessment result and the logarithmic value of the first quality assessment value, and this difference reflects the distance between the solution result and the exact solution; and calculate the difference between the logarithmic value of the preset second quality assessment value and the logarithmic value of the first quality assessment value, and determine the intermediate result based on the ratio between the two differences; the ratio reflects the relative distance between the current solution result and the exact solution. If the solution result is far from the exact solution, the ratio is large, and a large negative reward is generated. The second quality assessment value is used to evaluate whether the solution result is a feasible solution, and the first quality assessment value is used to evaluate whether the solution result is an exact solution. The second sub-function is used to measure the relative gap of the current solution result, and smooth these gaps through a logarithmic function, so that the result with a larger gap receives a larger penalty, and the result with a smaller gap receives a smaller penalty.

[0109] For example, the reward sub-function can be expressed as If x≤∈, then g=0, otherwise, Among them, g(x) represents the reward sub-function, x represents the quality evaluation result, ∈ represents the first quality evaluation value, ∈ feas represents the second quality assessment value.

[0110] After obtaining the intermediate result, the electronic device selects an appropriate reward coefficient according to different situations, multiplies it by the processed intermediate result, and obtains the final reward value.

[0111] If the quality evaluation result meets the preset second quality condition, it means that the current hyperparameter selection is valid, and the electronic device determines the product between the first reward coefficient and the intermediate result as the reward corresponding to the reinforcement learning in this training process; this step encourages the agent to select appropriate hyperparameters so that the algorithm converges well. Among them, the second quality condition can be specifically set according to the actual application scenario, and this embodiment does not impose any restrictions on this. For example, the second quality condition includes at least one of the following: the relative residual is lower than the preset threshold and the relative target error is less than the preset error.

[0112] If the quality assessment result does not meet the second quality condition and the rollback strategy is not executed in this training process, the device determines the product of the second reward coefficient and the intermediate result as the reward corresponding to the reinforcement learning in this training process; this step slightly punishes the agent, informing it that the current hyperparameter selection is not good, but not serious enough to require a reset.

[0113] If the rollback strategy is implemented in this training process, the electronic device determines the product between the third reward coefficient and the intermediate result as the reward corresponding to the reinforcement learning in this training process. This step strongly indicates that the current hyperparameter selection of the agent is problematic and needs to be readjusted to avoid system divergence through a larger penalty.

[0114] The reward function of this embodiment distinguishes different degrees of error, so that the electronic device can adjust the severity of the penalty according to the actual situation, thereby improving the training efficiency. By combining positive incentives and negative penalties, the intelligent agent in reinforcement learning is helped to gradually find the optimal hyperparameter combination to accelerate the solution process, improve the convergence speed and prevent the system from diverging.

[0115] Exemplarily, the quality assessment result includes at least one of a relative residual and a relative target error. The above-mentioned reward calculation process can be performed on the relative residual to obtain a reward value corresponding to the relative residual, or the above-mentioned reward calculation process can be performed on the relative target error to obtain a reward value corresponding to the relative target error, or both can be performed to obtain two reward values. This implementation does not impose any restrictions on this.

[0116] In S306, the parameters of the policy function are updated based on the reward, and the state corresponding to the reinforcement learning in the current training process is input into the updated policy function, so that the updated policy function determines the hyperparameters for the next training process.

[0117] In this step, the electronic device updates the parameters of the strategy function according to the reward generated in step S305. The higher the reward, the more reasonable the hyperparameter selection is. Otherwise, the strategy function needs to be updated. Figure 4 States in i+1) inputs the updated policy function to determine the hyperparameters for the next round of training. This cyclic process allows the policy function to be gradually improved and eventually converges to the optimal hyperparameter selection strategy. Through the feedback mechanism in reinforcement learning, the policy function can be gradually optimized according to the performance of the solution results, thereby finding the optimal hyperparameter settings.

[0118] Exemplarily, the electronic device can update the parameters of the policy function based on the reward according to the pre-configured reinforcement learning algorithm. The specific type of the above reinforcement learning algorithm is not particularly limited in this specification; for example, it can include PPO (Proximal Policy Optimization) algorithm, DQN (Deep Q-Network) algorithm or Actor-Critic, etc.

[0119] In some embodiments, in order to reduce the frequent decision-making of the agent, the action reuse count can be set, which is defined as the number of times an action is reused, which determines the frequency of the agent making decisions. At the beginning of training, an action reuse count is set, such as 1, 2 or higher, depending on the complexity and requirements of the specific task. When the agent needs to select an action, it first checks the current action reuse count. If the current number does not reach the set value, the agent will continue to use the previously selected action without making a new decision. Once the reuse count is reached, the agent will make a new decision and select the next preferred action. After each action is executed, the reuse count will be reduced accordingly. When the reuse count is exhausted, the agent will re-evaluate the state of the environment and select a new action based on the current state. In this embodiment, by reducing the number of decisions, the agent can reduce the computational complexity in each iteration, thereby saving time and resources, and a lower decision frequency helps to stabilize the training process and reduce the noise and uncertainty caused by frequent decisions.

[0120] The various technical features in the above embodiments can be arbitrarily combined as long as there is no conflict or contradiction between the combinations of features. However, due to space limitations, they are not described one by one. Therefore, any combination of the various technical features in the above embodiments also falls within the scope of this specification.

[0121] In some embodiments, the embodiments of this specification also provide an electronic device, including: a processor; a memory for storing processor executable instructions; wherein the processor implements any of the above methods by running the executable instructions.

[0122] Figure 5 is a schematic structural diagram of a device provided by an exemplary embodiment. Figure 5At the hardware level, the device includes a processor 502, an internal bus 504, a network interface 506, a memory 508, and a non-volatile memory 510, and may also include hardware required for other functions. One or more embodiments of this specification may be implemented based on software, such as the processor 502 reading the corresponding computer program from the non-volatile memory 510 into the memory 508 and then running it. Of course, in addition to the software implementation, one or more embodiments of this specification do not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0123] The hyperparameter determination device of the distribution task solver can be applied to Figure 5 The device shown in the figure is used to implement the technical solution of this specification. The hyperparameter determination device of the distribution task solver may include:

[0124] A reinforcement learning setting module, used to use a policy function for determining hyperparameters of an assignment task solver as an agent in reinforcement learning, and the assignment task solver as an environment in reinforcement learning; wherein the assignment task solver is used to solve a plurality of assignment tasks in the first set to determine a target item to be assigned to a target object indicated by each assignment task from a plurality of candidate items;

[0125] A reinforcement learning training module, used to use the hyperparameters determined by the policy function as actions in reinforcement learning, so that the allocation task solver solves the multiple allocation tasks in the first set based on the hyperparameters to obtain solution results, and calculates the state and reward corresponding to the reinforcement learning based on the solution results, and performs reinforcement learning training on the policy function with maximizing the reward as the optimization goal;

[0126] The target hyperparameter determination module is used to determine the target hyperparameters of the allocation task solver based on the policy function after training.

[0127] In one implementation, the reinforcement learning training module is specifically used to loop the following training process to train the policy function with the optimization goal of maximizing the reward in reinforcement learning: obtaining the hyperparameters determined by the policy function and feeding them back to the assigned task solver; obtaining the solution results obtained by the assigned task solver to solve multiple assigned tasks in the first set according to the hyperparameters; performing quality assessment on the solution results to obtain quality assessment results; determining the state corresponding to the reinforcement learning in this training process based on the quality assessment results; processing the quality assessment results using a preset reward function to obtain the reward corresponding to the reinforcement learning in this training process; updating the parameters of the policy function based on the reward, and inputting the state corresponding to the reinforcement learning in this training process into the updated policy function, so that the updated policy function determines the hyperparameters for the next training process.

[0128] In one implementation, the reinforcement learning training module is specifically used to determine the state corresponding to the reinforcement learning in this training process based on the quality assessment result if the quality assessment result meets the preset first quality condition; if the quality assessment result does not meet the first quality condition, execute the rollback strategy, and the rollback strategy indicates that the state in the previous training process will be used as the state corresponding to the reinforcement learning in this training process.

[0129] In one implementation, the reinforcement learning training module is specifically used to execute the rollback strategy if the quality assessment result does not meet the preset quality conditions and the number of consecutive rollbacks does not exceed the preset number of times; if the quality assessment result does not meet the preset quality conditions and the number of consecutive rollbacks exceeds the preset number of times, determine the state corresponding to the reinforcement learning in this training process based on the quality assessment result.

[0130] In one implementation, the allocation task includes an objective function and a constraint function for decision variables; the optimization goal of the objective function is to maximize the benefits that the target object can bring after being allocated to the target project and / or minimize the risks that the target object needs to bear after being allocated to the target project; the constraint function is used to describe the constraints that the decision variables should satisfy; the solution result is the solution of the decision variables in the allocation task, which represents the allocation amount of the target object to each candidate project.

[0131] The reinforcement learning training module is specifically used to determine the relative residual of the assigned task solver in this solution process based on the constraint function and the solution results; based on the preset optimal solution and the solution results of the objective function, determine the relative target error of the assigned task solver in this solution process.

[0132] In one implementation, the relative residual of the assigned task solver in this solution process is the ratio between the residual of the assigned task solver in this solution process and the constraint value indicated by the constraint function. The residual of the assigned task solver in this solution process is the difference between the calculation result of the constraint function in this solution process and the constraint value indicated by the constraint function. The calculation result of the constraint function in this solution process is obtained by processing the solution result based on the constraint function.

[0133] The relative target error of the assigned task solver in this solution process is the ratio of the target error of the assigned task solver in this solution process to the preferred target value; wherein, the target error is the difference between the target value and the preferred target value of the assigned task solver in this solution process, and the target value of the assigned task solver in this solution process is obtained by processing the solution result based on the objective function, and the preferred target value is obtained by processing the preset preferred solution based on the objective function.

[0134] In one implementation, the reinforcement learning training module is specifically used to determine the evaluation information for the current solution process based on the hyperparameters, relative residuals and relative target errors in the current training process; the evaluation information for the current solution process and the evaluation information for at least one historical solution process are determined as the state in the current training process corresponding to the reinforcement learning.

[0135] In one implementation, the state corresponding to the reinforcement learning in this training process also includes the number of consecutive rollbacks corresponding to this training process.

[0136] In one implementation, the preset reward function is the product of a reward coefficient and a reward sub-function; the reward sub-function is used to describe a processing method for a quality assessment result;

[0137] The reward coefficient includes a first reward coefficient, a second reward coefficient and a third reward coefficient; wherein the first reward coefficient is a positive number; the second reward coefficient and the third reward coefficient are both negative numbers, and the second reward coefficient is greater than the third reward coefficient;

[0138] The reinforcement learning training module is specifically used to process the quality assessment results based on the reward sub-function to obtain the intermediate results; if the quality assessment results meet the preset second quality condition, the product between the first reward coefficient and the intermediate result is determined as the reward corresponding to the reinforcement learning in this training process; if the quality assessment results do not meet the second quality condition and the rollback strategy is not executed in this training process, the product between the second reward coefficient and the intermediate result is determined as the reward corresponding to the reinforcement learning in this training process; if the rollback strategy is executed in this training process, the product between the third reward coefficient and the intermediate result is determined as the reward corresponding to the reinforcement learning in this training process.

[0139] In one implementation, the reward sub-function is a piecewise function including a first sub-function and a second sub-function.

[0140] The reinforcement learning training module is specifically used to determine that the intermediate result is 0 based on the first sub-function if the quality assessment result is less than or equal to the preset first quality assessment value; otherwise, the quality assessment result is processed based on the second sub-function to obtain the intermediate result; wherein the second sub-function is used to indicate the difference between the logarithm of the calculated quality assessment result and the logarithm of the first quality assessment value, and to calculate the difference between the logarithm of the preset second quality assessment value and the logarithm of the first quality assessment value, and to determine the intermediate result based on the ratio between the two differences; the second quality assessment value is used to evaluate whether the solution result is a feasible solution, and the first quality assessment value is used to evaluate whether the solution result is an exact solution.

[0141] In one implementation, the assigned tasks in the first set are obtained by sampling the assigned tasks in the second set;

[0142] The target hyperparameter determination module is specifically used to scale the hyperparameters determined by the strategy function after training is completed based on the quantity ratio between the first set and the second set, so as to obtain the target hyperparameters of the assigned task solver for solving the second set.

[0143] In one implementation, the allocation task solver includes a PJ-ADMM solver; and the hyperparameters of the PJ-ADMM solver include a penalty parameter and a proximal term parameter.

[0144] The target hyperparameter determination module is specifically used to determine the target penalty parameter of the allocation solver for solving the second set based on the product of the penalty parameter determined by the strategy function after the training is completed and the scaling ratio; and determine the proximal term parameter determined by the strategy function after the training is completed as the target proximal term parameter of the allocation solver for solving the second set.

[0145] The implementation process of the functions and effects of each module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, which will not be repeated here.

[0146] Based on the same concept as the above method, this specification also provides a computer-readable storage medium on which computer instructions are stored. When the instructions are executed by a processor, the steps of the method described in any of the above embodiments are implemented.

[0147] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0148] Based on the same concept as the above method, this specification also provides a computer program product, including a computer program / instruction, which implements the steps of the method described in any of the above embodiments when executed by a processor.

[0149] The above description is merely a preferred embodiment of one or more embodiments of the present specification and is not intended to limit one or more embodiments of the present specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present specification shall be included in the scope of protection of one or more embodiments of the present specification.

Claims

1. A method for determining hyperparameters of a distribution task solver, comprising: Using a policy function for determining hyperparameters of an assignment task solver as an agent in reinforcement learning, and using the assignment task solver as an environment in reinforcement learning; wherein the assignment task solver is used to solve a plurality of assignment tasks in a first set to determine a target item to be assigned to a target object indicated by each of the assignment tasks from a plurality of candidate items; Using the hyperparameters determined by the policy function as actions in the reinforcement learning, so that the assigned task solver solves the multiple assigned tasks in the first set based on the hyperparameters to obtain solution results, and calculates the state and reward corresponding to the reinforcement learning based on the solution results, and performs reinforcement learning training on the policy function with maximizing the reward as the optimization goal; Based on the policy function after training, target hyperparameters of the assigned task solver are determined.

2. The method according to claim 1, wherein the hyperparameters determined by the policy function are used as actions in the reinforcement learning, so that the assigned task solver solves the multiple assigned tasks in the first set based on the hyperparameters to obtain solution results, and calculates the state and reward corresponding to the reinforcement learning based on the solution results, and performs reinforcement learning training on the policy function with maximizing the reward as the optimization goal, comprising: The following training process is repeated to train the policy function with the optimization goal of maximizing the reward in the reinforcement learning: Obtaining hyperparameters determined by the strategy function and feeding them back to the assigned task solver; Obtaining a solution result obtained by the assigned task solver by solving a plurality of assigned tasks in the first set according to the hyperparameters; Performing quality assessment on the solution result to obtain a quality assessment result; Determine a state in this training process corresponding to the reinforcement learning based on the quality assessment result; Processing the quality assessment result using a preset reward function to obtain a reward corresponding to the reinforcement learning in this training process; The parameters of the policy function are updated based on the reward, and the state corresponding to the reinforcement learning in this training process is input into the updated policy function, so that the updated policy function determines the hyperparameters for the next training process.

3. The method according to claim 2, wherein determining the state corresponding to the reinforcement learning in the current training process based on the quality assessment result comprises: If the quality assessment result satisfies a preset first quality condition, determining a state in the current training process corresponding to the reinforcement learning based on the quality assessment result; If the quality assessment result does not meet the first quality condition, a rollback strategy is executed, wherein the rollback strategy indicates that the state in the previous training process is used as the state in the current training process corresponding to the reinforcement learning.

4. The method according to claim 3, wherein executing the rollback strategy comprises: If the quality assessment result does not meet the preset quality condition and the number of consecutive rollbacks does not exceed the preset number, executing the rollback strategy; The method further comprises: If the quality assessment result does not meet the preset quality condition and the number of consecutive rollbacks exceeds a preset number, a state in this training process corresponding to the reinforcement learning is determined based on the quality assessment result.

5. According to the method described in any one of claims 2 to 4, the assignment task includes an objective function and a constraint function for the decision variables; the optimization goal of the objective function is to maximize the benefits that the target object can bring after being assigned to the target project and / or minimize the risks that the target object needs to bear after being assigned to the target project, and the constraint function is used to describe the constraint conditions that the decision variables should satisfy; The solution result is the solution of the decision variables in the allocation task, which represents the allocation amount of the target object to each candidate project; The performing quality assessment on the solution result to obtain a quality assessment result includes: Based on the constraint function and the solution result, determining the relative residual of the assigned task solver in this solution process; Based on the preset optimal solution of the objective function and the solution result, the relative target error of the assigned task solver in this solution process is determined.

6. According to the method of claim 5, the relative residual of the assigned task solver in the current solution process is the ratio between the residual of the assigned task solver in the current solution process and the constraint value indicated by the constraint function, and the residual of the assigned task solver in the current solution process is obtained by processing the solution result based on the constraint function; and / or The relative target error of the assigned task solver in this solution process is the ratio between the target error of the assigned task solver in this solution process and the preferred target value; wherein, The target error is the difference between the target value of the assigned task solver in this solution process and the preferred target value. The target value of the assigned task solver in this solution process is obtained by processing the solution result based on the objective function, and the preferred target value is obtained by processing the preset preferred solution based on the objective function.

7. The method according to claim 5, wherein determining the state corresponding to the reinforcement learning in the current training process based on the quality assessment result comprises: Determine evaluation information for this solution process based on the hyperparameters, the relative residual and the relative target error in this training process; The evaluation information for the current solution process and the evaluation information for at least one historical solution process are determined as the state in the current training process corresponding to the reinforcement learning.

8. According to the method of claim 7, the state in the current training process corresponding to the reinforcement learning also includes the number of consecutive rollbacks corresponding to the current training process.

9. The method according to claim 2, wherein the preset reward function is the product of a reward coefficient and a reward sub-function; the reward sub-function is used to describe a processing method for the quality assessment result; The reward coefficient includes a first reward coefficient, a second reward coefficient and a third reward coefficient; wherein, The first reward coefficient is a positive number; the second reward coefficient and the third reward coefficient are both negative numbers, and the second reward coefficient is greater than the third reward coefficient; The using of a preset reward function to process the quality assessment result to obtain a reward corresponding to the reinforcement learning in this training process includes: Processing the quality assessment result based on the reward sub-function to obtain an intermediate result; If the quality evaluation result satisfies a preset second quality condition, determining the product of the first reward coefficient and the intermediate result as the reward corresponding to the reinforcement learning in the current training process; If the quality assessment result does not meet the second quality condition and the rollback strategy is not executed in this training process, the product of the second reward coefficient and the intermediate result is determined as the reward corresponding to the reinforcement learning in this training process; If the rollback strategy is implemented in this training process, the product of the third reward coefficient and the intermediate result is determined as the reward in this training process corresponding to the reinforcement learning.

10. The method according to claim 9, wherein the reward sub-function is a piecewise function including a first sub-function and a second sub-function; The step of processing the quality assessment result based on the reward sub-function to obtain an intermediate result includes: If the quality assessment result is less than or equal to a preset first quality assessment value, determining the intermediate result to be 0 based on the first sub-function; Otherwise, the quality assessment result is processed based on the second sub-function to obtain the intermediate result; wherein the second sub-function is used to indicate the calculation of the difference between the logarithm of the quality assessment result and the logarithm of the first quality assessment value, and the calculation of the difference between the logarithm of a preset second quality assessment value and the logarithm of the first quality assessment value, and the intermediate result is determined based on the ratio between the two differences; the second quality assessment value is used to evaluate whether the solution result is a feasible solution, and the first quality assessment value is used to evaluate whether the solution result is an exact solution.

11. The method according to claim 1, wherein the assigned tasks in the first set are obtained by sampling the assigned tasks in the second set; The step of determining target hyperparameters of the task allocation solver based on the strategy function after training includes: Based on the quantity ratio between the first set and the second set, the hyperparameters determined by the strategy function after the training are scaled to obtain the target hyperparameters of the assigned task solver for solving the second set.

12. The method of claim 11, wherein the assigned task solver comprises a PJ-ADMM solver; The hyperparameters of the PJ-ADMM solver include penalty parameters and proximal term parameters; The scaling process of the hyperparameters determined by the strategy function after the training is completed based on the quantity ratio between the first set and the second set includes: Determining a target penalty parameter of the allocation solver for solving the second set based on the product of the penalty parameter determined by the policy function after training and the scaling ratio; as well as The proximal term parameters determined by the policy function after training are determined as target proximal term parameters of the allocation solver for solving the second set.

13. An electronic device, comprising: processor; A memory for storing processor-executable instructions; wherein the processor implements the steps of the method according to any one of claims 1 to 12 by executing the executable instructions.

14. A computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 12.

15. A computer program product comprising a computer program / instructions, which when executed by a processor implement the steps of the method according to any one of claims 1 to 12.

Images