A task scheduling method and device thereof

Through the two-layer reinforcement learning model and hierarchical progressive method, the problem of solving accuracy and efficiency in scheduling problems in the prior art is solved, and efficient and accurate task scheduling and multi-level resource management are achieved.

CN111738488BActive Publication Date: 2024-11-19HUAWEI TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202010407594.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-05-14
Publication Date
2024-11-19
Estimated Expiration
2040-05-14

AI Technical Summary

Technical Problem

The prior art is difficult to take into account both solution accuracy and efficiency in scheduling problems, especially in large tasks, and the generalization ability is weak, so it is impossible to schedule tasks efficiently and accurately.

Method used

The two-layer reinforcement learning model is adopted. The first layer model handles the allocation relationship between tasks and resource groups. The second layer model calculates the correspondence between tasks and minimum granular resources through hierarchical progressive mode, uses sliding window sampling to improve efficiency, and considers the multi-level structure. characteristic.

Benefits of technology

It realizes efficient and accurate task scheduling, improves solution efficiency, is suitable for large tasks and multi-level resource structures, and enhances generalization capabilities.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN111738488B_ABST
    Figure CN111738488B_ABST
Patent Text Reader

Abstract

The present application discloses a task scheduling method and device thereof, which are applied to the field of artificial intelligence, including: processing a plurality of tasks to be scheduled and a plurality of schedulable resource groups through a first reinforcement learning model to obtain a first allocation relationship, wherein the first allocation relationship represents the correspondence between the plurality of tasks to be scheduled and the plurality of schedulable resource groups; processing the plurality of tasks to be scheduled and the plurality of schedulable resources through a second reinforcement learning model at least based on the first allocation relationship to obtain a second allocation relationship. The present application can sample the first allocation relationship, and respectively schedule and allocate each sampled portion of the tasks to be scheduled through a second reinforcement learning model, thereby obtaining the correspondence between each portion of the tasks to be scheduled and the plurality of schedulable resources, and the second reinforcement learning model does not need to allocate resources to the global tasks to be scheduled, thereby improving the efficiency of solving the problem.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to a task scheduling method and device thereof. Background Art

[0002] Scheduling is one of the most common problems in large-scale manufacturing, logistics, production and other links. Scheduling always has different meanings in different scenarios. For example, logistics scheduling mainly refers to the reasonable arrangement and scheduling of the vehicles and personnel of logistics companies according to the weight, destination, specifications, and urgency of the goods to be shipped during the logistics process; scheduling in the production environment is to sort the tasks and match the tasks and machines (schedulable resources) in several tasks (jobs) according to the production capacity and production needs of different machines in different production lines; timetabling of workers / flight attendants in large manufacturing plants / airports is also a type of scheduling problem. This is because the goal of this type of problem is also to achieve the optimal match in different time periods according to the work characteristics of workers / flight attendants and the needs of the scene. Therefore, the core is sorting and optimal allocation, regardless of whether the task is people or goods. Generally speaking, the goal of the scheduling problem is to obtain the sorting corresponding to the minimum total working time (makespan) under the premise of a given number of tasks.

[0003] In the prior art, methods such as the greedy method or mixed integer programming (Mixed Integer Programming, MIP) or constraint programming (Constraint Programming, CP) are often used to solve scheduling problems. However, these methods often have difficulty in balancing solution accuracy and solution efficiency (especially when the task is large), and the solutions of the prior art can often only solve a single type of task and have weak generalization capabilities.

[0004] Therefore, a solution is needed that can schedule tasks efficiently and accurately, especially for large tasks; in addition, this solution is also required to have high generalization ability. Summary of the invention

[0005] In a first aspect, the present application provides a task scheduling method, the method comprising:

[0006] Acquire multiple tasks to be scheduled and multiple schedulable resource groups; wherein each of the multiple schedulable resource groups includes multiple schedulable resources; for example, in a product production scenario, the tasks to be scheduled may be products to be produced, and in a personnel scheduling scenario, the tasks to be scheduled may be personnel to be produced, and so on; in a product production scenario, each of the multiple schedulable resource groups may be a production line, for example, in a mobile phone production scenario, each of the multiple schedulable resource groups may be a production line for a mobile phone component, such as a battery production line, a shell production line, a chip production line, and so on. Accordingly, each schedulable resource group may include multiple schedulable resources, multiple schedulable Each schedulable resource in the scheduling resource is a production equipment in the production line, for example, a battery production line may include multiple battery production equipment, and a shell production line may include multiple shell production equipment; in the scenario of personnel scheduling, each schedulable resource group in the multiple schedulable resource groups may be a time period, for example, in the scenario of personnel scheduling, each schedulable resource group in the multiple schedulable resource groups may be a day, for example, it may be Monday, Tuesday, Wednesday, or a certain day of some months, etc., accordingly, each schedulable resource group may include multiple schedulable resources, and each schedulable resource in the multiple schedulable resources is a sub-time period in the time period, for example, a certain day may include multiple hours, multiple minutes, or other multiple sub-time periods;

[0007] It should be noted that in addition to obtaining multiple tasks to be scheduled and multiple schedulable resource groups, other scheduling information can also be obtained, such as the time for each schedulable resource group to perform and complete each task, and some constraints, for example, each schedulable resource can be occupied by at most one task at the same time, each task can occupy at most one schedulable resource at the same time, each schedulable resource cannot execute the next task before completing a specific task, the waiting time should not exceed the execution time of the previous task, etc.;

[0008] The plurality of tasks to be scheduled and the plurality of schedulable resource groups are processed by a first reinforcement learning model to obtain a first allocation relationship, wherein the first allocation relationship includes a correspondence relationship between the plurality of tasks to be scheduled and the plurality of schedulable resource groups; the first reinforcement learning model has a function of implementing scheduling allocation for input scheduling information. In one implementation, the first reinforcement learning model can perform iterative processing on the plurality of tasks to be scheduled, the plurality of schedulable resource groups and other acquired scheduling information, and a first allocation relationship can be obtained each time the processing is performed; the first allocation relationship in this embodiment can represent a correspondence relationship between the plurality of tasks to be scheduled and the plurality of schedulable resource groups, wherein, in the first allocation relationship, for any task, its correspondence relationship with the plurality of schedulable resource groups can be represented as a sorting of the plurality of schedulable resource groups;

[0009] At least based on the first allocation relationship, the multiple tasks to be scheduled and the multiple schedulable resources are processed by a second reinforcement learning model to obtain a second allocation relationship, and the second allocation relationship includes a corresponding relationship between the multiple tasks to be scheduled and the multiple schedulable resources; wherein, in the second allocation relationship, each of the multiple tasks to be scheduled and the schedulable resource group to which the corresponding schedulable resource belongs satisfies the first allocation relationship. For task scheduling, the scheduling resources that need to be allocated can usually be divided into different granularities. For example, in the scenario of product production, the scheduling resources can be divided into production lines, production stages, and production equipment, etc. For the above scenarios, the results of task scheduling should be expressed as the correspondence between tasks and the scheduling resources of the smallest granularity. However, if the correspondence between each task and the scheduling resources of the smallest granularity is directly calculated, due to the large solution space, the corresponding calculation overhead is also large. As the number of tasks and schedulable resources increase, the increase in calculation overhead is unacceptable. Therefore, in this embodiment, the correspondence between each task and the scheduling resources of the smallest granularity is not directly calculated. Instead, the correspondence between each task and the scheduling resources of the larger granularity is calculated in a hierarchical progressive manner, and then the correspondence between each task and the scheduling resources of the smaller granularity is further calculated with the above-mentioned correspondence as a constraint. Specifically, the first reinforcement learning model can calculate the correspondence between tasks and scheduling resources of larger granularity (first allocation relationship), which specifies the correspondence between each task and scheduling resources of larger granularity. Then, using the first allocation relationship as a constraint, the correspondence between each task and scheduling resources of larger granularity is fixed, and the correspondence between each task and scheduling resources of smaller granularity is calculated through the second reinforcement learning model (second allocation relationship).

[0010] This embodiment can sample the first allocation relationship, and use the second reinforcement learning model to schedule and allocate each sampled part of the tasks to be scheduled, thereby obtaining the corresponding relationship between each part of the tasks to be scheduled and multiple schedulable resources. Through the above method, on the one hand, the second reinforcement learning model does not need to allocate resources for the global tasks to be scheduled, which improves the solution efficiency. On the other hand, the schedulable resource group in the scheduling problem is often a multi-level structure, which can be divided into multiple granularity levels. In the prior art, the substantial structure of the scheduling problem is not considered, while this case takes into account the structure of the scheduling problem itself (hierarchical characteristics) and designs a multi-level task scheduling model, thereby achieving efficient and accurate solutions.

[0011] In an optional implementation, each of the multiple schedulable resource groups is a production line, and each of the multiple schedulable resources is a production equipment in the production line; or,

[0012] Each of the multiple schedulable resource groups is a time period, and each of the multiple schedulable resources is a sub-time period in the time period.

[0013] In an optional implementation, processing the plurality of tasks to be scheduled and the plurality of schedulable resources by using a second reinforcement learning model based on the first allocation relationship includes:

[0014] At least based on the first allocation relationship, part of the multiple tasks to be scheduled and the multiple schedulable resources are processed through a second reinforcement learning model to obtain a sub-allocation relationship, and the sub-allocation relationship represents the corresponding relationship between the part of the tasks to be scheduled and the multiple schedulable resources.

[0015] In an embodiment of the present application, the second reinforcement learning model can sample the first allocation relationship through partial information sharing of sliding-window sampling, obtain the corresponding relationship between some tasks to be scheduled and multiple schedulable resources, and then calculate the sub-allocation relationship representing the corresponding relationship between these tasks to be scheduled and the multiple schedulable resources. That is, in this embodiment, the first allocation relationship can be sampled, and the second reinforcement learning model can be used to schedule and allocate each sampled partial task to be scheduled, thereby obtaining the corresponding relationship between each partial task to be scheduled and multiple schedulable resources. In the above manner, the second reinforcement learning model does not need to allocate resources for the global tasks to be scheduled, thereby improving the solution efficiency.

[0016] In one implementation, the first reinforcement learning model and the second reinforcement learning model can be trained independently, so only a smaller training set is required, and then generalization is used to guide scheduling on a larger data set.

[0017] In an optional implementation, the multiple tasks to be scheduled include multiple target tasks to be scheduled, the multiple schedulable resource groups include a target schedulable resource group, the target schedulable resource group includes multiple target schedulable resources, and in the first allocation relationship, the multiple target tasks to be scheduled correspond to the target schedulable resource group; the method also includes: when less than a preset number of target schedulable resources among the multiple target schedulable resources can be allocated to the at least one target task to be scheduled, determining a target constraint, the target constraint indicating that the at least one target task to be scheduled does not correspond to the target schedulable resource group.

[0018] In one case, when the second reinforcement learning model performs resource allocation calculations, there are not enough multiple target schedulable resources to be allocated to multiple target tasks to be scheduled (corresponding to the target schedulable resource group in the first allocation relationship), that is, the target schedulable resources are in a busy state at this time, then the target constraint is determined, and the target constraint indicates that at least one target task to be scheduled does not correspond to the target schedulable resource group, that is, the second reinforcement learning model determines that the first allocation relationship calculated by the first reinforcement learning model is not the optimal allocation relationship. At this time, the second allocation relationship only includes the correspondence between some tasks to be scheduled and multiple schedulable resources.

[0019] In an optional implementation, the method further includes:

[0020] At least based on the target constraint and the second allocation relationship, the multiple tasks to be scheduled and the multiple schedulable resource groups are processed by the first reinforcement learning model to obtain an updated first allocation relationship, and the updated first allocation relationship represents the corresponding relationship between the multiple tasks to be scheduled and the multiple schedulable resource groups; wherein, in the updated first allocation relationship, the at least one target task to be scheduled corresponds to a schedulable resource group in the multiple schedulable resource groups except the target schedulable resource group; at least based on the updated first allocation relationship, the multiple tasks to be scheduled and the multiple schedulable resources are processed by the second reinforcement learning model to obtain an updated second allocation relationship, and the updated second allocation relationship represents the corresponding relationship between the multiple tasks to be scheduled and the multiple schedulable resources; wherein, in the updated second allocation relationship, each of the multiple tasks to be scheduled and the schedulable resource group to which the corresponding schedulable resource belongs satisfies the updated first allocation relationship.

[0021] That is, the second reinforcement learning model can synchronize the determined target constraints to the first reinforcement learning model, and the first reinforcement learning model can reprocess the multiple tasks to be scheduled and the multiple schedulable resource groups based on the target constraints to obtain an updated first allocation relationship, wherein, in the updated first allocation relationship, the at least one target task to be scheduled corresponds to a schedulable resource group in the multiple schedulable resource groups except the target schedulable resource group. In one implementation, if the second reinforcement learning model determines that under the constraints of the first allocation relationship, all tasks to be scheduled have schedulable resources that can be allocated, the second allocation relationship obtained by the second reinforcement learning model can be directly output.

[0022] In an optional implementation, the first reinforcement learning model and the second reinforcement learning model can be trained separately. In this embodiment, the first reinforcement learning model and the second reinforcement learning model are connected through a sliding window sampling, so that the first reinforcement learning model and the second reinforcement learning model can be obtained through non-joint training, and the computational overhead of the training process can be reduced through the expanded training method. The first reinforcement learning model and the second reinforcement learning model can be trained independently, so only a smaller training set is required, and then generalization is used to guide scheduling on a larger data set.

[0023] In an optional implementation, the first reinforcement learning model includes a first reward function; the second reinforcement learning model includes a second reward function. This embodiment is different from the reinforcement learning model that relies on a common reward function. In order to be compatible with multi-level reinforcement learning models, different reward functions are set for different reinforcement learning models in this embodiment, so that it is more compatible with the multi-level mathematical model and the corresponding model training method is more flexible.

[0024] In an optional implementation, the method also includes: generating a first shared vector, the first shared vector including multiple state variables; each state variable can represent the allocation state of a task to be scheduled or the occupied state of a schedulable resource group; at least based on the first allocation relationship, processing the multiple tasks to be scheduled and the multiple schedulable resources through a second reinforcement learning model, including: processing the multiple tasks to be scheduled and the multiple schedulable resources through a second reinforcement learning model based on at least the first shared vector and the first allocation relationship.

[0025] In an optional implementation, the method includes: generating a second shared vector, the second shared vector including multiple state variables; each state variable can represent the allocation state of a task to be scheduled, the occupied state of a schedulable resource group or the occupied state of a schedulable resource; at least based on the target constraint and the second allocation relationship, processing the multiple tasks to be scheduled and the multiple schedulable resource groups through a first reinforcement learning model, including: at least based on the second shared vector, the target constraint and the second allocation relationship, processing the multiple tasks to be scheduled and the multiple schedulable resource groups through a first reinforcement learning model.

[0026] In a second aspect, the present application provides a system, comprising: a first reinforcement learning model and a second reinforcement learning model implemented by one or more computers;

[0027] The first reinforcement learning model is used to obtain a plurality of tasks to be scheduled and a plurality of schedulable resource groups, and process the plurality of tasks to be scheduled and the plurality of schedulable resource groups to obtain a first allocation relationship; wherein each of the plurality of schedulable resource groups includes a plurality of schedulable resources, and the first allocation relationship includes a correspondence relationship between the plurality of tasks to be scheduled and the plurality of schedulable resource groups;

[0028] The second reinforcement learning model is used to process the multiple tasks to be scheduled and the multiple schedulable resources through the second reinforcement learning model at least based on the first allocation relationship to obtain a second allocation relationship, and the second allocation relationship includes the corresponding relationship between the multiple tasks to be scheduled and the multiple schedulable resources; wherein, in the second allocation relationship, each of the multiple tasks to be scheduled and the schedulable resource group to which the corresponding schedulable resource belongs satisfies the first allocation relationship.

[0029] In an optional implementation, each of the multiple schedulable resource groups is a production line, and each of the multiple schedulable resources is a production equipment in the production line; or,

[0030] Each of the multiple schedulable resource groups is a time period, and each of the multiple schedulable resources is a sub-time period in the time period.

[0031] In an optional implementation, the second reinforcement learning model is used to process part of the multiple tasks to be scheduled and the multiple schedulable resources through the second reinforcement learning model based at least on the first allocation relationship to obtain a sub-allocation relationship, wherein the sub-allocation relationship represents the correspondence between the part of the tasks to be scheduled and the multiple schedulable resources.

[0032] In an optional implementation, the multiple tasks to be scheduled include multiple target tasks to be scheduled, the multiple schedulable resource groups include a target schedulable resource group, the target schedulable resource group includes multiple target schedulable resources, and in the first allocation relationship, the multiple target tasks to be scheduled correspond to the target schedulable resource group;

[0033] The second reinforcement learning model is also used to determine a target constraint when less than a preset number of target schedulable resources among the multiple target schedulable resources can be allocated to the at least one target task to be scheduled, and the target constraint indicates that the at least one target task to be scheduled does not correspond to the target schedulable resource group.

[0034] In an optional implementation, the first reinforcement learning model is further used to:

[0035] Based at least on the target constraint and the second allocation relationship, the plurality of tasks to be scheduled and the plurality of schedulable resource groups are processed by a first reinforcement learning model to obtain an updated first allocation relationship, wherein the updated first allocation relationship represents a corresponding relationship between the plurality of tasks to be scheduled and the plurality of schedulable resource groups; wherein, in the updated first allocation relationship, the at least one target task to be scheduled corresponds to a schedulable resource group among the plurality of schedulable resource groups except the target schedulable resource group;

[0036] The second reinforcement learning model is also used to:

[0037] At least based on the updated first allocation relationship, the multiple tasks to be scheduled and the multiple schedulable resources are processed by the second reinforcement learning model to obtain an updated second allocation relationship, and the updated second allocation relationship represents the corresponding relationship between the multiple tasks to be scheduled and the multiple schedulable resources; wherein, in the updated second allocation relationship, each of the multiple tasks to be scheduled and the schedulable resource group to which the corresponding schedulable resource belongs satisfies the updated first allocation relationship.

[0038] In an optional implementation, the first reinforcement learning model and the second reinforcement learning model are trained separately.

[0039] In an optional implementation, the first reinforcement learning model includes a first reward function; the second reinforcement learning model includes a second reward function.

[0040] In an optional implementation, the first reinforcement learning model is further used to: generate a first shared vector, the first shared vector comprising a plurality of state variables; wherein each state variable may represent an allocation state of a task to be scheduled or an occupied state of a schedulable resource group;

[0041] The second reinforcement learning model is used to: process the multiple tasks to be scheduled and the multiple schedulable resources through the second reinforcement learning model based at least on the first sharing vector and the first allocation relationship.

[0042] In an optional implementation, the first reinforcement learning model is further used to: generate a second shared vector, the second shared vector comprising a plurality of state variables; wherein each state variable may represent an allocation state of a task to be scheduled, an occupied state of a schedulable resource group, or an occupied state of a schedulable resource;

[0043] The second reinforcement learning model is used to: process the multiple tasks to be scheduled and the multiple schedulable resource groups through the first reinforcement learning model based at least on the second sharing vector, the target constraint and the second allocation relationship.

[0044] The first reinforcement learning model includes DDQN, and the second reinforcement learning model includes a pointer network PN.

[0045] In an optional implementation, the first reinforcement learning model and the second reinforcement learning model include recurrent neural networks.

[0046] In an optional implementation, the first reinforcement learning model includes two long short-term memory networks LSTM based on an attention mechanism, and the second reinforcement learning model includes a first recurrent neural network RNN ​​for encoding and a second RNN for decoding.

[0047] In a third aspect, the present application provides a task scheduling device, the device comprising:

[0048] An acquisition module, used to acquire a plurality of tasks to be scheduled and a plurality of schedulable resource groups; wherein each of the plurality of schedulable resource groups includes a plurality of schedulable resources;

[0049] A first reinforcement learning module, configured to process the plurality of tasks to be scheduled and the plurality of schedulable resource groups through a first reinforcement learning model to obtain a first allocation relationship, wherein the first allocation relationship includes a correspondence relationship between the plurality of tasks to be scheduled and the plurality of schedulable resource groups;

[0050] A second reinforcement learning module is used to process the multiple tasks to be scheduled and the multiple schedulable resources through a second reinforcement learning model at least based on the first allocation relationship to obtain a second allocation relationship, wherein the second allocation relationship includes a corresponding relationship between the multiple tasks to be scheduled and the multiple schedulable resources; wherein, in the second allocation relationship, each of the multiple tasks to be scheduled and the schedulable resource group to which the corresponding schedulable resource belongs satisfies the first allocation relationship.

[0051] In an optional implementation, each of the multiple schedulable resource groups is a production line, and each of the multiple schedulable resources is a production equipment in the production line; or,

[0052] Each of the multiple schedulable resource groups is a time period, and each of the multiple schedulable resources is a sub-time period in the time period.

[0053] In an optional implementation, the second reinforcement learning module is specifically used to:

[0054] At least based on the first allocation relationship, part of the multiple tasks to be scheduled and the multiple schedulable resources are processed through a second reinforcement learning model to obtain a sub-allocation relationship, and the sub-allocation relationship represents the corresponding relationship between the part of the tasks to be scheduled and the multiple schedulable resources.

[0055] In an optional implementation, the multiple tasks to be scheduled include multiple target tasks to be scheduled, the multiple schedulable resource groups include a target schedulable resource group, the target schedulable resource group includes multiple target schedulable resources, and in the first allocation relationship, the multiple target tasks to be scheduled correspond to the target schedulable resource group;

[0056] The second reinforcement learning module is specifically used for:

[0057] In the case that less than a preset number of target schedulable resources among the multiple target schedulable resources can be allocated to the at least one target task to be scheduled, a target constraint is determined, wherein the target constraint indicates that the at least one target task to be scheduled does not correspond to the target schedulable resource group.

[0058] In an optional implementation, the first reinforcement learning module is further used to process the multiple tasks to be scheduled and the multiple schedulable resource groups through a first reinforcement learning model based at least on the target constraint and the second allocation relationship to obtain an updated first allocation relationship, wherein the updated first allocation relationship represents a corresponding relationship between the multiple tasks to be scheduled and the multiple schedulable resource groups; wherein, in the updated first allocation relationship, the at least one target task to be scheduled corresponds to a schedulable resource group in the multiple schedulable resource groups except the target schedulable resource group;

[0059] The second reinforcement learning module is also used to process the multiple tasks to be scheduled and the multiple schedulable resources through the second reinforcement learning model at least based on the updated first allocation relationship to obtain an updated second allocation relationship, and the updated second allocation relationship represents the corresponding relationship between the multiple tasks to be scheduled and the multiple schedulable resources; wherein, in the updated second allocation relationship, each of the multiple tasks to be scheduled and the schedulable resource group to which the corresponding schedulable resource belongs satisfies the updated first allocation relationship.

[0060] In an optional implementation, the first reinforcement learning model and the second reinforcement learning model are trained separately.

[0061] In an optional implementation, the first reinforcement learning model includes a first reward function; the second reinforcement learning model includes a second reward function.

[0062] In an optional implementation, the first reinforcement learning module is further used to generate a first shared vector, wherein the first shared vector includes a plurality of state variables; each state variable may represent an allocation state of a task to be scheduled or an occupied state of a schedulable resource group;

[0063] The second reinforcement learning module is further used to process the multiple tasks to be scheduled and the multiple schedulable resources through a second reinforcement learning model based at least on the first shared vector and the first allocation relationship.

[0064] In an optional implementation, the second reinforcement learning module is further used to generate a second shared vector, wherein the second shared vector includes a plurality of state variables; wherein each state variable may represent an allocation state of a task to be scheduled, an occupied state of a schedulable resource group, or an occupied state of a schedulable resource;

[0065] The first reinforcement learning module is further used to process the multiple tasks to be scheduled and the multiple schedulable resource groups through a first reinforcement learning model based at least on the second shared vector, the target constraint and the second allocation relationship.

[0066] In a fourth aspect, an embodiment of the present application provides a device, including a memory, a processor, and a bus system, wherein the memory is used to store programs, and the processor is used to execute the programs in the memory to execute the first aspect and any optional method of the first aspect mentioned above.

[0067] In a fifth aspect, an embodiment of the present invention further provides a system, the system comprising at least one processor, at least one memory and at least one communication interface; the processor, the memory and the communication interface are connected via a communication bus and communicate with each other;

[0068] The memory is used to store the application code for executing the above solution, and the execution is controlled by the processor. The processor is used to execute the application code stored in the memory to obtain the task scheduling result; wherein the code stored in the memory can execute a task scheduling method provided above.

[0069] The communication interface is used to communicate with other devices or communication networks to send the task scheduling results to the devices or communication networks.

[0070] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer executes the above-mentioned first aspect and any optional method thereof.

[0071] In the seventh aspect, an embodiment of the present application provides a computer-readable storage medium, which stores one or more instructions, and when the instructions are executed by one or more computers, the one or more computers implement the above-mentioned second aspect and any optional system thereof.

[0072] In an eighth aspect, an embodiment of the present application provides a computer program which, when executed on a computer, enables the computer to execute the above-mentioned first aspect and any optional method thereof.

[0073] In a ninth aspect, the present application provides a chip system, which includes a processor for supporting an execution device or a training device to implement the functions involved in the above aspects, such as sending or processing the data involved in the above methods; or, information. In one possible design, the chip system also includes a memory, which is used to store program instructions and data necessary for the execution device or the training device. The chip system can be composed of chips, or it can include chips and other discrete devices.

[0074] The embodiment of the present application provides a task scheduling method, including: obtaining multiple tasks to be scheduled and multiple schedulable resource groups; wherein each of the multiple schedulable resource groups includes multiple schedulable resources; processing the multiple tasks to be scheduled and the multiple schedulable resource groups through a first reinforcement learning model to obtain a first allocation relationship, the first allocation relationship represents the corresponding relationship between the multiple tasks to be scheduled and the multiple schedulable resource groups; at least based on the first allocation relationship, processing the multiple tasks to be scheduled and the multiple schedulable resources through a second reinforcement learning model to obtain a second allocation relationship, the second allocation relationship represents the corresponding relationship between the multiple tasks to be scheduled and the multiple schedulable resources; wherein, in the second allocation relationship, each of the multiple tasks to be scheduled and the schedulable resource group to which the corresponding schedulable resource belongs satisfies the first allocation relationship. In this embodiment, the first allocation relationship can be sampled, and the second reinforcement learning model can be used to schedule and allocate the sampled partial tasks to be scheduled, thereby obtaining the corresponding relationship between each partial task to be scheduled and the multiple schedulable resources. In the above manner, the second reinforcement learning model does not need to allocate resources to the global tasks to be scheduled, thereby improving the solution efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] Figure 1 A structural diagram of the main framework of artificial intelligence;

[0076] Figure 2 A schematic diagram of a system architecture provided for an embodiment of the present application;

[0077] Figure 3 A hardware structure of a chip provided in an embodiment of the present application;

[0078] Figure 4 A flowchart of a task scheduling method provided in an embodiment of the present application;

[0079] Figure 5 A schematic diagram of the structure of a system 500 provided in an embodiment of the present application;

[0080] Figure 6 A schematic diagram of the structure of a scheduler in an embodiment of the present application;

[0081] Figure 7 It is a schematic diagram of the computing performance of the embodiment of the present application;

[0082] Figure 8 and Figure 9a It is a comparison of the computing performance of the embodiment of the present application and other existing technologies;

[0083] Figure 9bA flowchart of a task scheduling method provided in an embodiment of the present application;

[0084] Fig.10 A schematic diagram of the structure of a task scheduling device provided in this embodiment;

[0085] Fig.11 A schematic diagram of the structure of an execution device provided in an embodiment of the present application;

[0086] Fig.12 A schematic diagram of the structure of a training device provided in an embodiment of the present application;

[0087] Fig.13 A schematic diagram of the structure of a chip provided in an embodiment of the present application. DETAILED DESCRIPTION

[0088] The embodiments of the present invention are described below in conjunction with the accompanying drawings in the embodiments of the present invention. The terms used in the implementation mode of the present invention are only used to explain the specific embodiments of the present invention, and are not intended to limit the present invention.

[0089] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0090] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0091] First, the overall workflow of the artificial intelligence system is described. Figure 1 , Figure 1The figure shows a structural diagram of the main framework of artificial intelligence. The following is an explanation of the above artificial intelligence theme framework from the two dimensions of "intelligent information chain" (horizontal axis) and "IT value chain" (vertical axis). Among them, the "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it can be a general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, intelligent execution and output. In this process, the data has undergone a condensation process of "data-information-knowledge-wisdom". The "IT value chain" reflects the value that artificial intelligence brings to the information technology industry from the underlying infrastructure of human intelligence, information (providing and processing technology implementation) to the industrial ecology process of the system.

[0092] (1) Infrastructure

[0093] The infrastructure provides computing power support for the AI ​​system, enables communication with the outside world, and supports it through the basic platform. It communicates with the outside world through sensors; computing power is provided by smart chips (CPU, NPU, GPU, ASIC, FPGA and other hardware acceleration chips); the basic platform includes distributed computing frameworks and networks and other related platform guarantees and support, which can include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to obtain data, and these data are provided to the smart chips in the distributed computing system provided by the basic platform for calculation.

[0094] (2) Data

[0095] The data on the upper layer of the infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data of traditional devices, including business data of existing systems and perception data such as force, displacement, liquid level, temperature, and humidity.

[0096] (3) Data processing

[0097] Data processing usually includes data training, machine learning, deep learning, search, reasoning, decision-making and other methods.

[0098] Among them, machine learning and deep learning can symbolize and formalize data for intelligent information modeling, extraction, preprocessing, and training.

[0099] Reasoning refers to the process of simulating human intelligent reasoning in computers or intelligent systems, using formalized information to perform machine thinking and solve problems based on reasoning control strategies. Typical functions are search and matching.

[0100] Decision-making refers to the process of making decisions after intelligent information is reasoned, usually providing functions such as classification, sorting, and prediction.

[0101] (4) General ability

[0102] After the data has undergone the data processing mentioned above, some general capabilities can be further formed based on the results of the data processing, such as an algorithm or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0103] (5) Smart products and industry applications

[0104] Smart products and industry applications refer to the products and applications of artificial intelligence systems in various fields. They are the encapsulation of the overall artificial intelligence solution, which productizes intelligent information decision-making and realizes practical applications. Its application areas mainly include: smart terminals, smart transportation, smart medical care, autonomous driving, safe cities, etc.

[0105] Scheduling is one of the most common problems in large-scale manufacturing / logistics / production, and scheduling always has different meanings in different scenarios. For example, logistics scheduling mainly refers to the logistics company's reasonable arrangement and scheduling of its vehicles and personnel according to the weight, destination, specifications, and urgency of the goods to be shipped during the logistics process.

[0106] Scheduling in a production environment is to sort tasks and match tasks with production equipment in a number of jobs according to the capacity and production needs of different machines in different production lines. That is, multiple tasks are assigned to production equipment in each production line.

[0107] For example, in the job-shop scheduling scenario, n workpieces are processed on m machines, each workpiece has a specific processing technology, the processing order of each workpiece and the time spent on each process are given, and the processing order of the workpieces on each machine is arranged to optimize a certain indicator. It is not required that each workpiece be executed on each machine.

[0108] For example, in the scenario of flow-shop scheduling, this type of scheduling problem requires that each task must be executed in each stage in sequence. It does not involve the matching of tasks and stages, but mainly determines the execution order of tasks to prevent the overall completion time from being prolonged due to excessive waiting time in the middle.

[0109] Slightly different from general cargo scheduling, the scheduling of workers / stewardess at airports / large manufacturing plants is also a type of scheduling problem. This is because the goal of this type of problem is also to achieve the best match in different time periods according to the work characteristics and scenarios of the workers / stewardess. Therefore, the core is sorting and the most efficient allocation, regardless of whether the "task" is people or goods. Generally speaking, the goal of the scheduling problem is to get the sorting corresponding to the minimum total working time (makespan) under the premise of a given number of tasks.

[0110] Meanwhile, the scheduling problem in computers is the method of allocating resources needed for work. Resources can refer to virtual computing resources, such as threads, processes, or data streams; they can also refer to hardware resources, such as processors, network connections, or expansion cards. The program that performs scheduling work is called a scheduler. The scheduler is usually implemented so that all computing resources are busy (in load balancing), allowing multiple users to effectively share system resources at the same time, or to achieve a specified quality of service.

[0111] The following describes the method provided by this application from the perspectives of model training and model application:

[0112] Figure 2 is a schematic diagram of a system 100 architecture provided in an embodiment of the present application. Figure 2 In the embodiment of the present application, the execution device 110 is configured with an input / output (I / O) interface 112 for data interaction with an external device. A user may input data to the I / O interface 112 through a client device 140. The input data may include: various tasks to be scheduled, callable resources, and other parameters.

[0113] When the execution device 110 preprocesses the input data, or when the computing module 111 of the execution device 110 performs calculation and other related processing (such as implementing the function of the neural network in the present application), the execution device 110 can call the data, code, etc. in the data storage system 150 for the corresponding processing, and can also store the data, instructions, etc. obtained by the corresponding processing in the data storage system 150.

[0114] Finally, the I / O interface 112 returns the processing result to the client device 140 so as to provide it to the user.

[0115] It is worth noting that the training device 120 can generate corresponding target models / rules based on different training data for different goals or different tasks, and the corresponding target models / rules can be used to achieve the above goals or complete the above tasks, thereby providing users with the desired results.

[0116] exist Figure 2In the case shown in the figure, the user can manually give input data, and the manual giving can be operated through the interface provided by the I / O interface 112. In another case, the client device 140 can automatically send input data to the I / O interface 112. If the client device 140 is required to automatically send input data and needs to obtain the user's authorization, the user can set the corresponding authority in the client device 140. The user can view the results output by the execution device 110 on the client device 140, and the specific presentation form can be a specific method such as display, sound, action, etc. The client device 140 can also be used as a data acquisition terminal to collect the input data of the input I / O interface 112 and the output results of the output I / O interface 112 as shown in the figure as new sample data, and store them in the database 130. Of course, it is also possible not to collect through the client device 140, but the I / O interface 112 directly stores the input data of the input I / O interface 112 and the output results of the output I / O interface 112 as new sample data in the database 130.

[0117] It is worth noting that Figure 2 This is only a schematic diagram of a system architecture provided by an embodiment of the present application. The positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. For example, in Figure 2 In the embodiment, the data storage system 150 is an external memory relative to the execution device 110. In other cases, the data storage system 150 may also be placed in the execution device 110. Figure 2 As shown, a reinforcement learning model can be obtained by training according to the training device 120.

[0118] Figure 3 The hardware structure of a chip provided in the embodiment of the present application includes a neural network processor NPU50. The chip can be set as follows Figure 2 The execution device 110 shown in FIG. 1 is used to complete the calculation work of the calculation module 111. The chip can also be set in Figure 2 The training device 120 shown is used to complete the training work of the training device 120 and output the target model / rule.

[0119] The neural network processor NPU 30 is mounted on the host central processing unit (CPU) as a coprocessor, and the host CPU assigns tasks. The core part of the NPU is the operation circuit 303, and the controller 304 controls the operation circuit 303 to extract data from the memory (weight memory or input memory) and perform operations.

[0120] In some implementations, the operation circuit 303 includes multiple processing units (process engines, PEs) inside. In some implementations, the operation circuit 303 is a two-dimensional systolic array. The operation circuit 303 can also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the operation circuit 303 is a general-purpose matrix processor.

[0121] For example, assume there is an input matrix A, a weight matrix B, and an output matrix C. The operation circuit takes the corresponding data of matrix B from the weight memory 302 and caches it on each PE in the operation circuit. The operation circuit takes the matrix A data from the input memory 301 and performs matrix operation with matrix B, and the partial result or final result of the matrix is ​​stored in the accumulator 308.

[0122] The vector calculation unit 307 can further process the output of the operation circuit, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. For example, the vector calculation unit 307 can be used for network calculations of non-convolutional / non-FC layers in a neural network, such as pooling, batch normalization, local response normalization, etc.

[0123] In some implementations, the vector calculation unit 307 can store the processed output vector to the unified buffer 306. For example, the vector calculation unit 307 can apply a nonlinear function to the output of the operation circuit 303, such as a vector of accumulated values, to generate an activation value. In some implementations, the vector calculation unit 307 generates a normalized value, a merged value, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 303, for example, for use in a subsequent layer in a neural network.

[0124] The unified memory 306 is used to store input data and output data.

[0125] The weight data is directly transferred from the external memory to the input memory 301 and / or the unified memory 306 through the direct memory access controller 305 (DMAC), the weight data in the external memory is stored in the weight memory 302, and the data in the unified memory 306 is stored in the external memory.

[0126] The bus interface unit (BIU) 310 is used to implement the interaction between the main CPU, DMAC and instruction fetch memory 309 through the bus.

[0127] An instruction fetch buffer 309 connected to the controller 304 and used to store instructions used by the controller 304;

[0128] The controller 304 is used to call the instructions cached in the instruction fetch memory 309 to control the working process of the computing accelerator.

[0129] Generally, the unified memory 306, the input memory 301, the weight memory 302 and the instruction fetch memory 309 are all on-chip memories, and the external memory is a memory outside the NPU, which can be a double data rate synchronous dynamic random access memory (DDR SDRAM), a high bandwidth memory (HBM) or other readable and writable memory.

[0130] See also Figure 4 , Figure 4 A flowchart of a task scheduling method provided in an embodiment of the present application is shown below. Figure 4 As shown, the task scheduling method provided in the embodiment of the present application includes:

[0131] 401. Acquire multiple tasks to be scheduled and multiple schedulable resource groups; wherein each of the multiple schedulable resource groups includes multiple schedulable resources.

[0132] In an embodiment of the present application, when scheduling tasks, multiple tasks to be scheduled and multiple schedulable resource groups may be obtained; wherein each of the multiple schedulable resource groups includes multiple schedulable resources.

[0133] For example, in a product production scenario, the task to be scheduled may be a product to be produced; in a personnel scheduling scenario, the task to be scheduled may be a person to be produced, and so on, but the embodiments of the present application are not limited thereto.

[0134] In the product production scenario, each of the multiple schedulable resource groups can be a production line. For example, in the mobile phone production scenario, each of the multiple schedulable resource groups can be a production line for mobile phone components, such as a battery production line, a shell production line, a chip production line, etc. Correspondingly, each schedulable resource group can include multiple schedulable resources, and each of the multiple schedulable resources is a production equipment in the production line. For example, a battery production line can include multiple battery production equipment, and a shell production line can include multiple shell production equipment. This is not limited here.

[0135] In the scenario of personnel scheduling, each of the multiple schedulable resource groups can be a time period. For example, in the scenario of personnel scheduling, each of the multiple schedulable resource groups can be a day, such as Monday, Tuesday, Wednesday, or a certain day of some months, etc. Correspondingly, each schedulable resource group can include multiple schedulable resources, and each of the multiple schedulable resources is a sub-time period in the time period. For example, a day can include multiple hours, multiple minutes, or other multiple sub-time periods, which is not limited here.

[0136] It should be noted that the schedulable resource group in this embodiment may be a resource collection with a larger granularity than the schedulable resource, and the schedulable resource group may include multiple schedulable resources.

[0137] It should be noted that in addition to obtaining multiple tasks to be scheduled and multiple schedulable resource groups, other scheduling information can also be obtained, such as the time for each schedulable resource group to perform and complete each task, and some constraints, such as each schedulable resource is occupied by at most one task at the same time, each task occupies at most one schedulable resource at the same time, each schedulable resource cannot execute the next task before completing a specific task, the waiting time should not exceed the execution time of the previous task, etc. It should be noted that other scheduling information can also be obtained based on different scenarios to be scheduled, and this embodiment is not limited.

[0138] 402. Process the plurality of tasks to be scheduled and the plurality of schedulable resource groups by using a first reinforcement learning model to obtain a first allocation relationship, where the first allocation relationship represents a correspondence between the plurality of tasks to be scheduled and the plurality of schedulable resource groups.

[0139] In this embodiment, the corresponding relationship represented by the first allocation relationship can be understood as an allocation relationship between tasks and schedulable resources. The allocation relationship can represent a schedulable resource group to which each of the multiple tasks to be scheduled is allocated.

[0140] In an embodiment of the present application, after a plurality of tasks to be scheduled and a plurality of schedulable resource groups are acquired, the plurality of tasks to be scheduled and the plurality of schedulable resource groups may be processed by a first reinforcement learning model.

[0141] In an embodiment of the present application, the first reinforcement learning model has the function of implementing scheduling allocation for the input scheduling information. In one implementation, the first reinforcement learning model can perform iterative processing on multiple tasks to be scheduled, the multiple schedulable resource groups and other acquired scheduling information, and each processing can obtain a first allocation relationship. During the above iterative processing, the first reinforcement learning model can proceed in the direction of increasing its own reward function, that is, as the iteration proceeds, the first reinforcement learning model will obtain a better allocation relationship, wherein the "better" allocation relationship can be understood as, under this allocation relationship, the overall overhead of implementing each task is smaller (for example, less time overhead, etc., which is specifically related to the optimization function specified in the first reinforcement learning model and is not limited here).

[0142] In one implementation, at the beginning of scheduling, the first reinforcement learning model can cluster all tasks according to the number of schedulable resource groups, and try to ensure that the number of tasks allocated to each schedulable resource group is close to an average.

[0143] In one implementation, the first reinforcement learning model may be a double deep q-network (DDQN).

[0144] It should be noted that the first allocation relationship in this embodiment can represent the correspondence between the multiple tasks to be scheduled and the multiple schedulable resource groups, wherein, in the first allocation relationship, for any task, the correspondence between it and the multiple schedulable resource groups can be expressed as a sorting of the multiple schedulable resource groups; taking the product production scenario as an example, the first allocation relationship can represent the correspondence between each product and multiple production lines, specifically, for any product, the correspondence between it and the multiple production lines can be expressed as a sorting of the multiple production lines; for example, if there are 5 production lines (production line 1, production line 2, production line 3, production line 4, production line 5), in the first allocation relationship, for product 1, the correspondence between the other 5 production lines is: production line 4—production line 2—production line 1—production line 5—production line 3, that is, in the first allocation relationship, product 1 needs to pass through production line 4, production line 2, production line 1, production line 5 and production line 3 in sequence.

[0145] 403. Based at least on the first allocation relationship, the multiple tasks to be scheduled and the multiple schedulable resources are processed by a second reinforcement learning model to obtain a second allocation relationship, wherein the second allocation relationship represents the corresponding relationship between the multiple tasks to be scheduled and the multiple schedulable resources; wherein, in the second allocation relationship, each of the multiple tasks to be scheduled and the schedulable resource group to which the corresponding schedulable resource belongs satisfies the first allocation relationship.

[0146] In an embodiment of the present application, the first reinforcement learning model can perform iterative processing on multiple tasks to be scheduled, the multiple schedulable resource groups, and other acquired scheduling information, and each processing can obtain a first allocation relationship, and each obtained first allocation relationship can be used as the input of the second reinforcement learning model. Furthermore, the second reinforcement learning model can process the multiple tasks to be scheduled and the multiple schedulable resources based on the first allocation relationship to obtain a second allocation relationship, and the second allocation relationship represents the corresponding relationship between the multiple tasks to be scheduled and the multiple schedulable resources; wherein, in the second allocation relationship, each of the multiple tasks to be scheduled and the schedulable resource group to which the corresponding schedulable resource belongs satisfies the first allocation relationship.

[0147] For task scheduling, the scheduling resources that need to be allocated can usually be divided into different granularities. For example, in the scenario of product production, the scheduling resources can be divided into production lines, production stages, and production equipment, etc. For the above scenarios, the results of task scheduling should be expressed as the correspondence between tasks and the scheduling resources of the smallest granularity. However, if the correspondence between each task and the scheduling resources of the smallest granularity is directly calculated, due to the large solution space, the corresponding calculation overhead is also large. As the number of tasks and schedulable resources increase, the increase in calculation overhead is unacceptable. Therefore, in this embodiment, the correspondence between each task and the scheduling resources of the smallest granularity is not directly calculated. Instead, the correspondence between each task and the scheduling resources of the larger granularity is calculated in a hierarchical progressive manner, and then the correspondence between each task and the scheduling resources of the smaller granularity is further calculated with the above-mentioned correspondence as a constraint. Specifically, the first reinforcement learning model can calculate the correspondence between tasks and scheduling resources of larger granularity (first allocation relationship), which specifies the correspondence between each task and scheduling resources of larger granularity. Then, using the first allocation relationship as a constraint, the correspondence between each task and scheduling resources of larger granularity is fixed, and the correspondence between each task and scheduling resources of smaller granularity is calculated through the second reinforcement learning model (second allocation relationship).

[0148] In one implementation, the second reinforcement learning model may be a pointer network (PN).

[0149] It should be noted that the second allocation relationship in this embodiment can represent the correspondence between the multiple tasks to be scheduled and the multiple schedulable resources, wherein, in the second allocation relationship, for any task, the correspondence between it and the multiple schedulable resources can be expressed as a sorting of the multiple schedulable resources; taking the product production scenario as an example, the second allocation relationship can represent the correspondence between each product and a plurality of production equipment, specifically, for any product, the correspondence between it and the multiple production equipment can be expressed as a sorting of the production equipment; for example, if there are 5 production lines (production line 1, production line 2, production line 3, production line 4, production line 5), production line 1 includes production equipment 1-10 (where each stage of the production line corresponds to a certain number of production equipment, which are no longer distinguished here), production line 2 includes production equipment 1-10, production line 3 includes production equipment 1-10, production line 4 includes production equipment 1-10, production line 5 includes production equipment 1-10, production line 5 includes production equipment 1-10, production line 1 includes production equipment 1-10, production line 2 includes production equipment 1-10, production line 3 includes production equipment 1-10, production line 4 includes production equipment 1-10, production line 5 includes production equipment 1-10, production line 1 ... Production line 3 includes production equipment 1-10, production line 4 includes production equipment 1-10, and production line 5 includes production equipment 1-10. In the first allocation relationship, for product 1, its corresponding relationship with the 5 production lines is: production line 4-production line 2-production line 1-production line 5-production line 3, that is, in the first allocation relationship, product 1 needs to pass through production line 4, production line 2, production line 1, production line 5 and production line 3 in sequence. In the second allocation relationship, for product 1, its corresponding relationship with the production equipment is: equipment 2 in production line 4-equipment 1 in production line 2-equipment 8 in production line 1-equipment 5 in production line 5-equipment 9 in production line 3, that is, in the second allocation relationship, product 1 needs to pass through equipment 2 in production line 4, equipment 1 in production line 2, equipment 8 in production line 1, equipment 5 in production line 5 and equipment 9 in production line 3 in sequence.

[0150] In one implementation, based on the first allocation relationship, a part of the multiple tasks to be scheduled and the multiple schedulable resources can be processed by a second reinforcement learning model to obtain a sub-allocation relationship, wherein the sub-allocation relationship represents the correspondence between the part of the tasks to be scheduled and the multiple schedulable resources.

[0151] In an embodiment of the present application, the second reinforcement learning model can sample the first allocation relationship through partial information sharing of sliding-window sampling, obtain the corresponding relationship between some tasks to be scheduled and multiple schedulable resources, and then calculate the sub-allocation relationship representing the corresponding relationship between these tasks to be scheduled and the multiple schedulable resources. That is, in this embodiment, the first allocation relationship can be sampled, and the second reinforcement learning model can be used to schedule and allocate each sampled partial task to be scheduled, thereby obtaining the corresponding relationship between each partial task to be scheduled and multiple schedulable resources. In the above manner, the second reinforcement learning model does not need to allocate resources for the global tasks to be scheduled, thereby improving the solution efficiency.

[0152] In the embodiment of the present application, the upper and lower reinforcement learning models can respectively include corresponding reward functions (the first reinforcement learning model includes the first reward function, and the second reinforcement learning model includes the second reward function), but for the overall task scheduling model, it is still necessary to consider the global optimum, so the two reward functions of the upper and lower models should be combined together. In the existing implementation of the multi-layer model, the reward functions of the multi-layer models are directly added together, while in the present embodiment, a sliding window sampling method is adopted. For the task scheduling model as a whole, it is equivalent to realizing the local addition of the reward function, that is, the present embodiment connects the reward functions of the upper and lower reinforcement learning models by sliding window sampling, avoiding the loss of solution performance caused by excessive solution space if they are directly added, thereby effectively improving the solution quality.

[0153] In the embodiment of the present application, in the second allocation relationship, each to-be-scheduled task of some or all of the multiple to-be-scheduled tasks and the schedulable resource group to which the corresponding schedulable resource belongs satisfy the first allocation relationship.

[0154] Specifically, the multiple tasks to be scheduled include multiple target tasks to be scheduled, the multiple schedulable resource groups include a target schedulable resource group, the target schedulable resource group includes multiple target schedulable resources, and in the first allocation relationship, the multiple target tasks to be scheduled correspond to the target schedulable resource group; in the case where less than a preset number of target schedulable resources in the multiple target schedulable resources can be allocated to the at least one target task to be scheduled, a target constraint is determined, and the target constraint indicates that the at least one target task to be scheduled does not correspond to the target schedulable resource group. That is, in one case, since the second reinforcement learning model does not have sufficient multiple target schedulable resources to be allocated to the multiple target tasks to be scheduled (corresponding to the target schedulable resource group in the first allocation relationship) when performing resource allocation calculation, that is, the target schedulable resources are in a busy state at this time, then a target constraint is determined, and the target constraint indicates that the at least one target task to be scheduled does not correspond to the target schedulable resource group, that is, the second reinforcement learning model determines that the first allocation relationship calculated by the first reinforcement learning model is not the optimal allocation relationship. At this time, the second allocation relationship only includes the correspondence between some tasks to be scheduled and multiple schedulable resources.

[0155] In this embodiment, the first reinforcement learning model can process the multiple tasks to be scheduled and the multiple schedulable resource groups based on the target constraint and the second allocation relationship to obtain an updated first allocation relationship, and the updated first allocation relationship represents the correspondence between the multiple tasks to be scheduled and the multiple schedulable resource groups; wherein, in the updated first allocation relationship, the at least one target task to be scheduled corresponds to a schedulable resource group among the multiple schedulable resource groups except the target schedulable resource group.

[0156] That is, the second reinforcement learning model can synchronize the determined target constraints to the first reinforcement learning model, and the first reinforcement learning model can reprocess the multiple tasks to be scheduled and the multiple schedulable resource groups based on the target constraints to obtain an updated first allocation relationship, wherein, in the updated first allocation relationship, the at least one target task to be scheduled corresponds to a schedulable resource group in the multiple schedulable resource groups except the target schedulable resource group. In one implementation, if the second reinforcement learning model determines that under the constraints of the first allocation relationship, all tasks to be scheduled have schedulable resources that can be allocated, the second allocation relationship obtained by the second reinforcement learning model can be directly output.

[0157] Furthermore, the second reinforcement learning model can process the multiple tasks to be scheduled and the multiple schedulable resources based on the updated first allocation relationship to obtain an updated second allocation relationship, and the updated second allocation relationship represents the corresponding relationship between the multiple tasks to be scheduled and the multiple schedulable resources; wherein, in the updated second allocation relationship, each of the multiple tasks to be scheduled and the schedulable resource group to which the corresponding schedulable resource belongs satisfies the updated first allocation relationship.

[0158] In the embodiment of the present application, taking the scenario of product production as an example, the first reinforcement learning model of the upper layer determines the allocation and order of tasks at different stages, and the second reinforcement learning model of the lower layer determines the order and allocation of tasks of different production equipment at each specific stage. Then, the upper and lower layers can enter a back-and-forth iterative cycle (when the output result of the lower model is inconsistent with the upper result), and the entire scheduling is completed after convergence and stability (when the output result of the lower model is not inconsistent with the upper result).

[0159] Optionally, the first reinforcement learning model and the second reinforcement learning model may be trained separately. In this embodiment, the first reinforcement learning model and the second reinforcement learning model are connected through a sliding window sampling, so that the first reinforcement learning model and the second reinforcement learning model can be obtained through non-joint training, and the computational overhead of the training process can be reduced through the expanded training method.

[0160] Optionally, the reward function of the first reinforcement learning model is different from the reward function of the second reinforcement learning model. Unlike the existing implementation, which relies on a two-layer model with a common reward function, in this embodiment, in order to be compatible with multi-level reinforcement learning models, different reward functions are set for different reinforcement learning models, so that they are more compatible with the multi-level mathematical model and the corresponding model training method is more flexible.

[0161] Optionally, the first reinforcement learning model can also generate a first shared vector, which includes multiple state variables; each state variable can represent the allocation state of a task to be scheduled or the occupied state of a schedulable resource group; and then, the second reinforcement learning model can process the multiple tasks to be scheduled and the multiple schedulable resources through the second reinforcement learning model based at least on the first shared vector and the first allocation relationship.

[0162] Optionally, the second reinforcement learning model can also generate a second shared vector, which includes multiple state variables; each state variable can represent the allocation state of a task to be scheduled, the occupied state of a schedulable resource group, or the occupied state of a schedulable resource; further, the first reinforcement learning model can process the multiple tasks to be scheduled and the multiple schedulable resource groups through the first reinforcement learning model based at least on the second shared vector, the target constraint, and the second allocation relationship. In this embodiment, the entire task scheduling process is equivalent to forming a constrained Markov decision process, in which the state between the upper and lower layers needs to be shared; wherein the state of the entire process can be represented based on a shared vector, for example, the scheduling process of N tasks constitutes an N-dimensional vector. Each element of the vector can be a state variable of the task to be scheduled, and its initial value can be 0. The action in the decision-making process is to select some elements in the state from 0 to 1. If the state variable changes from 0 to 1, it means that the task is to be executed next.

[0163] The embodiment of the present application provides a task scheduling method, including: obtaining multiple tasks to be scheduled and multiple schedulable resource groups; wherein each of the multiple schedulable resource groups includes multiple schedulable resources; processing the multiple tasks to be scheduled and the multiple schedulable resource groups through a first reinforcement learning model to obtain a first allocation relationship, the first allocation relationship represents the corresponding relationship between the multiple tasks to be scheduled and the multiple schedulable resource groups; based at least on the first allocation relationship, processing the multiple tasks to be scheduled and the multiple schedulable resources through a second reinforcement learning model to obtain a second allocation relationship, the second allocation relationship represents the corresponding relationship between the multiple tasks to be scheduled and the multiple schedulable resources; wherein, in the second allocation relationship, each of the multiple tasks to be scheduled and the schedulable resource group to which the corresponding schedulable resource belongs satisfies the first allocation relationship. In this embodiment, the first allocation relationship can be sampled, and the partial tasks to be scheduled obtained by the sampling can be scheduled and allocated through the second reinforcement learning model, thereby obtaining the corresponding relationship between each partial task to be scheduled and the multiple schedulable resources. The schedulable resource groups in the scheduling problem are often multi-level structures, which can be divided into multiple granularity levels. In the existing task scheduling implementation, there is no good match between the task scheduling method and the actual structure of the scheduling problem. This case takes into account the structure of the scheduling problem itself (hierarchical characteristics) and designs a multi-level task scheduling model, thereby achieving efficient and accurate solutions. In this way, the second reinforcement learning model does not need to allocate resources for the global tasks to be scheduled, which improves the solution efficiency.

[0164] Reference Figure 5 , Figure 5 A schematic diagram of the structure of a system 500 provided in an embodiment of the present application is shown in FIG. Figure 5 As shown, the system 500 includes: a first reinforcement learning model 501 and a second reinforcement learning model 502 implemented by one or more computers, wherein the first reinforcement learning model and the second reinforcement learning model are connected;

[0165] The first reinforcement learning model 501 is used to obtain a plurality of tasks to be scheduled 503 and a plurality of schedulable resource groups 504, and process the plurality of tasks to be scheduled 503 and the plurality of schedulable resource groups 504 to output a first allocation relationship 505; wherein each of the plurality of schedulable resource groups includes a plurality of schedulable resources, and the first allocation relationship represents a corresponding relationship between the plurality of tasks to be scheduled and the plurality of schedulable resource groups;

[0166] The second reinforcement learning model 502 is used to process the multiple tasks to be scheduled and the multiple schedulable resources based on at least the first allocation relationship 505 to output a second allocation relationship 506, wherein the second allocation relationship 506 represents the corresponding relationship between the multiple tasks to be scheduled and the multiple schedulable resources; wherein, in the second allocation relationship 506, each of the multiple tasks to be scheduled and the schedulable resource group to which the corresponding schedulable resource belongs satisfies the first allocation relationship.

[0167] Optionally, each of the multiple schedulable resource groups is a production line, and each of the multiple schedulable resources is a production equipment in the production line; or,

[0168] Each of the multiple schedulable resource groups is a time period, and each of the multiple schedulable resources is a sub-time period in the time period.

[0169] Optionally, the second reinforcement learning model 502 is specifically used to process part of the multiple tasks to be scheduled and the multiple schedulable resources through a second reinforcement learning model based at least on the first allocation relationship to obtain a sub-allocation relationship, wherein the sub-allocation relationship represents the correspondence between the part of the tasks to be scheduled and the multiple schedulable resources.

[0170] Optionally, the multiple tasks to be scheduled include multiple first tasks to be scheduled and at least one second task to be scheduled, the multiple schedulable resource groups include a target schedulable resource group, the target schedulable resource group includes multiple target schedulable resources, and in the first allocation relationship, the multiple first tasks to be scheduled and at least one second task to be scheduled correspond to the target schedulable resource group;

[0171] The second reinforcement learning model is also used to determine a target constraint when less than a preset number of target schedulable resources among the multiple target schedulable resources can be allocated to the at least one target task to be scheduled, and the target constraint indicates that the at least one target task to be scheduled does not correspond to the target schedulable resource group.

[0172] Optionally, the first reinforcement learning model 501 is further used for:

[0173] At least based on the target constraint, the plurality of tasks to be scheduled and the plurality of schedulable resource groups are processed to obtain an updated first allocation relationship 507, wherein the updated first allocation relationship 507 represents a corresponding relationship between the plurality of tasks to be scheduled and the plurality of schedulable resource groups; wherein in the updated first allocation relationship 507, the at least one target task to be scheduled corresponds to a schedulable resource group in the plurality of schedulable resource groups except the target schedulable resource group;

[0174] Optionally, the second reinforcement learning model 502 is further used for:

[0175] At least based on the updated first allocation relationship 507, the multiple tasks to be scheduled and the multiple schedulable resources are processed to obtain an updated second allocation relationship 508, and the updated second allocation relationship 508 represents the corresponding relationship between the multiple tasks to be scheduled and the multiple schedulable resources; wherein, in the updated second allocation relationship 508, each of the multiple tasks to be scheduled and the schedulable resource group to which the corresponding schedulable resource belongs satisfies the updated first allocation relationship 507.

[0176] Optionally, the first reinforcement learning model and the second reinforcement learning model are trained separately.

[0177] Optionally, the first reinforcement learning model includes a first reward function; and the second reinforcement learning model includes a second reward function.

[0178] The first reinforcement learning model is further used to: generate a first shared vector, the first shared vector comprising a plurality of state variables; each state variable may represent an allocation state of a task to be scheduled or an occupied state of a schedulable resource group;

[0179] The second reinforcement learning model is used to: process the multiple tasks to be scheduled and the multiple schedulable resources through the second reinforcement learning model based at least on the first sharing vector and the first allocation relationship.

[0180] Optionally, the first reinforcement learning model is further used to: generate a second shared vector, the second shared vector comprising a plurality of state variables; wherein each state variable may represent an allocation state of a task to be scheduled, an occupied state of a schedulable resource group, or an occupied state of a schedulable resource;

[0181] The second reinforcement learning model is used to: process the multiple tasks to be scheduled and the multiple schedulable resource groups through the first reinforcement learning model based at least on the second sharing vector, the target constraint and the second allocation relationship.

[0182] Optionally, the first reinforcement learning model includes DDQN, and the second reinforcement learning model includes a pointer network PN.

[0183] Optionally, the first reinforcement learning model and the second reinforcement learning model are recurrent neural networks.

[0184] Optionally, the first reinforcement learning model includes a long short-term memory network LSTM based on an attention mechanism, and the second reinforcement learning model includes a first recurrent neural network RNN ​​for encoding and a second RNN for decoding.

[0185] The embodiment of the present application provides a system, including: a first reinforcement learning model and a second reinforcement learning model implemented by one or more computers, the first reinforcement learning model and the second reinforcement learning model are connected; the first reinforcement learning model is used to obtain multiple tasks to be scheduled and multiple schedulable resource groups, and process the multiple tasks to be scheduled and the multiple schedulable resource groups to output a first allocation relationship; wherein each of the multiple schedulable resource groups includes multiple schedulable resources, and the first allocation relationship represents the corresponding relationship between the multiple tasks to be scheduled and the multiple schedulable resource groups; the second reinforcement learning model is used to process the multiple tasks to be scheduled and the multiple schedulable resources based on the first allocation relationship to output a second allocation relationship, and the second allocation relationship represents the corresponding relationship between the multiple tasks to be scheduled and the multiple schedulable resources; wherein in the second allocation relationship, each of the multiple tasks to be scheduled and the schedulable resource group to which the corresponding schedulable resource belongs satisfies the first allocation relationship. In this embodiment, the first allocation relationship can be sampled, and the second reinforcement learning model can be used to schedule and allocate the sampled partial tasks to be scheduled, thereby obtaining the corresponding relationship between each partial task to be scheduled and the multiple schedulable resources. Through the above method, the second reinforcement learning model does not need to allocate resources for the global tasks to be scheduled, thereby improving the solution efficiency.

[0186] In one embodiment, the first reinforcement learning model is a dual Q network DDQN, and the second reinforcement learning model is a pointer network PN. Figure 6 , Figure 6 This is a schematic diagram of the structure of a scheduler in an embodiment of the present application, from Figure 6 It can be seen that the scheduler provided in this embodiment is a two-level model, the upper layer is a DDQN, the lower layer is a PN, and the two parts are connected through a sliding window sampling. The BDP scheduler in this embodiment can also have a wider range of application scenarios, and is not limited to the two-level structure itself. For example, when we deal with more complex hierarchical structures, we can rely on the structure of the BDP scheduler to form a multi-level training method, and different layers can be expanded based on the tree structure. Therefore, the BDP scheduler in this embodiment has a high generalization ability.

[0187] For example, the reinforcement learning model can be constructed in the following form:

[0188]

[0189] Among them, Ω u It can be concreted as:

[0190]

[0191] Ω l It can be concreted as:

[0192]

[0193] Where i can represent the stage number (i=1,…,I); j can represent the task number (j=1,…,N); k can represent the production equipment number (k=1,…,M); π u * It can represent the best learning strategy of the upper model; S u * Can represent the overall state of the upper model; π l * It can represent the best learning strategy of the underlying model; S l * Can represent the overall state of the underlying model; Ω u ,Ω l Can represent a set of constraint functions; γ t Can represent the discount factor in reinforcement learning; R t,u Can represent the upper model reward function; R t,,l Can represent the reward function of the lower model; s ij (u) It can represent the status of each task at different stages in the upper model, 1 means being executed, 0 means idle; ijk (l) It can represent the status of each task at different stages in the lower-level model, 1 means being executed, and 0 means idle; v j,k It can represent the start time of task j at production equipment k; i,j,k It can represent the processing time of task j on production equipment k at stage i; u i,j It can represent the start time of task j in stage i; ξ i It can be expressed as a positive constant controlling the idle time of different tasks in a stage.

[0194] The reinforcement learning model example provided in this embodiment is a two-level planning structure, where the upper and lower layers complete their respective goals. Related constraints (2) to (7) are constraints that need to be considered in the scheduling problem: for example, constraints (2) and (4) respectively indicate that each production device is occupied by at most one task at the same time, and each task is executed at most on one production device and one stage at the same time. Constraints (5) to (7) indicate that each production device cannot execute the next task before completing a specific task, and the waiting time should not exceed the execution time of the previous task, etc.

[0195] In the above model example, there is no need to rely on a given goal to directly complete the optimization, based on timely reward feedback: the optimization process at each moment mainly depends on the optimization result of the previous moment, but a discount parameter is used as a whole to consider the global goal. In actual optimization, this staged goal definition can guide optimization and solution more accurately than a unified goal. Exemplarily, the upper-layer model double deep q-network (DDQN) can be trained online using a deep reinforcement learning algorithm, in which an experience replay pool can be used. Each time training is performed, only a batch is sampled instead of all the data being read in, which can greatly improve the training efficiency; and the lower-layer pointer network (PN) mainly includes two recurrent neural network (RNN) models, which are used for encoding and decoding respectively. The encoding operation is based on the long short-term memory (LSTM) algorithm, which maps the read-in processing time information to a higher-dimensional space for more precise training, and then uses an attention-based algorithm to perform decoding operations, and finally outputs a quadratically optimized local optimal sequence.

[0196] The entire BDP scheduling process in the above example forms a constrained Markov decision process, in which the state between the upper and lower layers is shared. The state of the entire process is a 0-1 vector, and the scheduling process of N tasks constitutes an N-dimensional vector. The initial value of each element of the vector is 0. The action in the decision process is to change some elements in the selected state from 0 to 1, indicating that the next task is to be executed.

[0197] Exemplarily, this embodiment can complete the test on two sets of data sets. Each set of data sets has 9 subsets, each subset has 10,000 samples, including 8,000 training samples, 1,000 test samples, and 1,000 cross-validation samples. Each sample is a number pair consisting of the execution time of different tasks on different machines. The difference between different data sets is the distribution of the execution time of tasks on different machines, one is uniform distribution and the other is chi-square distribution. The difference between different subsets of the same data set is that the number of tasks involved in the training set is different, which are 25, 30, 50, 100, 250, 500, 1000, 2000 and 5000 respectively.

[0198] There are mainly two types of comparison algorithms in the following tests. One type is heuristic algorithms, which mainly involve two greedy-based heuristic algorithms 1 and NEH-based heuristic algorithms 2 that are currently used in scheduling scenarios. The other type is two deep learning algorithms, one based on DDQN and the other based on PN.

[0199] First, a comparison of the solution quality in different data sets is given. In Table 1 and Table 2, each row represents the size of the reward function makespan in different sub-data sets. The smaller the size, the better the corresponding algorithm. As shown in Table 1 and Table 2, compared with the greedy-based heuristic algorithm 1 (Heuristic 1 in Table 1 and Table 2), the NEH-based heuristic algorithm 2 (Heuristic 2 in Table 1 and Table 2), DDQN and PN, the technical solution of this application (BDP) has a smaller value of the reward function makespan, which can guide the acquisition of a smaller reward function, that is, given a number of tasks, a faster scheduling solution can be obtained, thereby saving overall resources.

[0200] Table 1

[0201]

[0202] Table 2

[0203]

[0204] Reference Figure 7 , Figure 7 This is a diagram of the computing performance of the embodiment of the present application, where the horizontal axis represents the problem size and the vertical axis represents the computing time in seconds. The asterisk line is the result of this embodiment, the square line is the result of the heuristic NEH, and the circle line is the result of the PN. Figure 7 As can be seen from the figure, as the problem size increases, the running time of NEH and PN increases nonlinearly, especially the heuristic algorithm. However, the running time of this embodiment increases quasi-linearly. As a result, this embodiment is more suitable for large-scale scheduling problems in industrial production than other algorithms.

[0205] Table 3 and Table 4 show the generalization results on two data sets, small, middle and large respectively represent the training algorithm on training sets with only 25 tasks, 50 tasks and 100 tasks, and the number of tasks involved in the test set is given in the first column of each table. From the two tables, we can see that this embodiment can basically complete the training on a training set with only 50 tasks, and achieve a similar effect to the training set of 100 tasks on the test set of 5000 tasks. This means that the technical solution of the present application is a very lightweight algorithm in the application scenario, and does not need to waste too much time in offline training.

[0206] Table 3

[0207]

[0208] Table 4

[0209]

[0210] Reference Figure 8 and Figure 9a , Figure 8 and Figure 9a It is a comparative illustration of the computing performance of the embodiment of the present application and other existing technologies. The hyper-parameter test is mainly conducted on the sliding-window sampling in the expanded training method, in order to verify the necessity of the training method and the sliding window. From Tables 5 and 6, we can see that the sliding window in the technical solution of the present application is necessary, because when beta=0, the quality of the solution is always the worst when the number of tasks is greater than 100. However, the quality of the solution does not improve as the sliding window becomes larger, but the optimal result is obtained at a specific value. Therefore, the size of the sliding window in this embodiment is a hyper-parameter-based design.

[0211] Figure 8 and Figure 9a The comparison of solution time also verifies the necessity of sliding windows from another perspective. All experiments with sliding windows outperform those without sliding windows in terms of performance, that is, they get the optimal solution faster. Moreover, the experiment shows that the size of the sliding window is inversely proportional to the computing performance, that is, the smaller the sliding window, the shorter the time to get the solution.

[0212] Table 5

[0213]

[0214] Table 6

[0215]

[0216] Reference Figure 9b , Figure 9b A process diagram of a task scheduling method provided in this embodiment is shown as follows: Figure 9b As shown in FIG. 1 , the task scheduling method of this embodiment includes:

[0217] 901. Input system information;

[0218] The system information in the embodiment of the present application may include multiple tasks to be scheduled, multiple schedulable resource groups, and other scheduling information, such as the time when each schedulable resource group performs and completes each task, and some constraints. Figure 4 The description of step 401 in the corresponding embodiment will not be repeated here.

[0219] 902. Establish a two-level scheduler;

[0220] The two-level scheduler in the embodiment of the present application may include the first reinforcement learning model and the second reinforcement learning model in the above embodiment. For details, please refer to the description of step 402 and step 403 in the above embodiment, which will not be repeated here.

[0221] 903. Perform scheduling processing based on the upper layer of the scheduler and fix the relevant variable values;

[0222] The scheduler upper layer in the embodiment of the present application can be the first reinforcement learning model in the above embodiment. Regarding how to perform scheduling processing based on the scheduler upper layer, reference can be made to the description of step 402 in the above embodiment, which will not be repeated here.

[0223] 904. Perform scheduling processing based on the lower layer of the scheduler and fix the relevant variable values;

[0224] 905. Determine whether convergence has occurred; if so, proceed to step 906; if not, return to step 903;

[0225] The lower layer of the scheduler in the embodiment of the present application can be the second reinforcement learning model in the above embodiment. Regarding how to perform scheduling processing based on the lower layer of the scheduler, reference can be made to the description of step 403 in the above embodiment, which will not be repeated here.

[0226] 906. Output the scheduling result.

[0227] Reference Fig.10 , Fig.10 A structural diagram of a task scheduling device provided in this embodiment is shown as follows: Fig.10 As shown in FIG. 1 , the device 1000 of this embodiment includes:

[0228] The acquisition module 1001 is used to acquire a plurality of tasks to be scheduled and a plurality of schedulable resource groups; wherein each of the plurality of schedulable resource groups includes a plurality of schedulable resources;

[0229] A first reinforcement learning module 1002 is used to process the multiple tasks to be scheduled and the multiple schedulable resource groups through a first reinforcement learning model to obtain a first allocation relationship, wherein the first allocation relationship represents a corresponding relationship between the multiple tasks to be scheduled and the multiple schedulable resource groups;

[0230] The second reinforcement learning module 1003 is used to process the multiple tasks to be scheduled and the multiple schedulable resources through a second reinforcement learning model at least based on the first allocation relationship to obtain a second allocation relationship, wherein the second allocation relationship includes a corresponding relationship between the multiple tasks to be scheduled and the multiple schedulable resources; wherein, in the second allocation relationship, each of the multiple tasks to be scheduled and the schedulable resource group to which the corresponding schedulable resource belongs satisfies the first allocation relationship.

[0231] In an optional implementation, each of the multiple schedulable resource groups is a production line, and each of the multiple schedulable resources is a production equipment in the production line; or,

[0232] Each of the multiple schedulable resource groups is a time period, and each of the multiple schedulable resources is a sub-time period in the time period.

[0233] In an optional implementation, the second reinforcement learning module 1003 is specifically used to:

[0234] At least based on the first allocation relationship, part of the multiple tasks to be scheduled and the multiple schedulable resources are processed through a second reinforcement learning model to obtain a sub-allocation relationship, wherein the sub-allocation relationship represents the correspondence between part of the multiple tasks to be scheduled and the multiple schedulable resources.

[0235] In an optional implementation, the multiple tasks to be scheduled include multiple target tasks to be scheduled, the multiple schedulable resource groups include a target schedulable resource group, the target schedulable resource group includes multiple target schedulable resources, and in the first allocation relationship, the multiple target tasks to be scheduled correspond to the target schedulable resource group;

[0236] The second reinforcement learning module 1003 is specifically used for:

[0237] In the case that less than a preset number of target schedulable resources among the multiple target schedulable resources can be allocated to the at least one target task to be scheduled, a target constraint is determined, wherein the target constraint indicates that the at least one target task to be scheduled does not correspond to the target schedulable resource group.

[0238] In an optional implementation, the first reinforcement learning module 1002 is further used to process the multiple tasks to be scheduled and the multiple schedulable resource groups through a first reinforcement learning model based at least on the target constraint and the second allocation relationship to obtain an updated first allocation relationship, wherein the updated first allocation relationship represents a corresponding relationship between the multiple tasks to be scheduled and the multiple schedulable resource groups; wherein, in the updated first allocation relationship, the at least one target task to be scheduled corresponds to a schedulable resource group in the multiple schedulable resource groups except the target schedulable resource group;

[0239] The second reinforcement learning module 1003 is also used to process the multiple tasks to be scheduled and the multiple schedulable resources through the second reinforcement learning model at least based on the updated first allocation relationship to obtain an updated second allocation relationship, and the updated second allocation relationship represents the corresponding relationship between the multiple tasks to be scheduled and the multiple schedulable resources; wherein, in the updated second allocation relationship, each of the multiple tasks to be scheduled and the schedulable resource group to which the corresponding schedulable resource belongs satisfies the updated first allocation relationship.

[0240] In an optional implementation, the first reinforcement learning model and the second reinforcement learning model are trained separately.

[0241] In an optional implementation, the first reinforcement learning model includes a first reward function; the second reinforcement learning model includes a second reward function.

[0242] In an optional implementation, the first reinforcement learning module is further used to generate a first shared vector, wherein the first shared vector includes a plurality of state variables; each state variable may represent an allocation state of a task to be scheduled or an occupied state of a schedulable resource group;

[0243] The second reinforcement learning module is further used to process the multiple tasks to be scheduled and the multiple schedulable resources through a second reinforcement learning model based at least on the first shared vector and the first allocation relationship.

[0244] In an optional implementation, the second reinforcement learning module is further used to generate a second shared vector, wherein the second shared vector includes a plurality of state variables; wherein each state variable may represent an allocation state of a task to be scheduled, an occupied state of a schedulable resource group, or an occupied state of a schedulable resource;

[0245] The first reinforcement learning module is further used to process the multiple tasks to be scheduled and the multiple schedulable resource groups through a first reinforcement learning model based at least on the second shared vector, the target constraint and the second allocation relationship.

[0246] The present application provides a task scheduling device, the device comprising: an acquisition module, used to acquire multiple tasks to be scheduled and multiple schedulable resource groups; wherein each of the multiple schedulable resource groups includes multiple schedulable resources; a first reinforcement learning module, used to process the multiple tasks to be scheduled and the multiple schedulable resource groups through a first reinforcement learning model to obtain a first allocation relationship, the first allocation relationship represents the corresponding relationship between the multiple tasks to be scheduled and the multiple schedulable resource groups; a second reinforcement learning module, used to process the multiple tasks to be scheduled and the multiple schedulable resources through a second reinforcement learning model based on the first allocation relationship to obtain a second allocation relationship, the second allocation relationship represents the corresponding relationship between the multiple tasks to be scheduled and the multiple schedulable resources; wherein in the second allocation relationship, each of the multiple tasks to be scheduled and the schedulable resource group to which the corresponding schedulable resource belongs satisfies the first allocation relationship. In this embodiment, the first allocation relationship can be sampled, and the partial tasks to be scheduled obtained by the sampling can be scheduled and allocated through the second reinforcement learning model, so as to obtain the corresponding relationship between each partial task to be scheduled and the multiple schedulable resources. Through the above method, the second reinforcement learning model does not need to allocate resources for the global tasks to be scheduled, thereby improving the solution efficiency.

[0247] Next, an execution device provided by an embodiment of the present application is introduced. Fig.11 , Fig.11 This is a schematic diagram of a structure of an execution device provided in an embodiment of the present application. The execution device 1100 may be a mobile phone, a tablet, a laptop, a smart wearable device, a server, etc., which are not limited here. Fig.10 The task scheduling device described in the corresponding embodiment is used to implement Fig.10 The function of task scheduling in the corresponding embodiment. Specifically, the execution device 1100 includes: a receiver 1101, a transmitter 1102, a processor 1103 and a memory 1104 (wherein the number of the processor 1103 in the execution device 1100 can be one or more, Fig.11 In the example of FIG. 1 , the processor 1103 may include an application processor 11031 and a communication processor 11032. In some embodiments of the present application, the receiver 1101, the transmitter 1102, the processor 1103 and the memory 1104 may be connected via a bus or other means.

[0248] The memory 1104 may include a read-only memory and a random access memory, and provides instructions and data to the processor 1103. A portion of the memory 1104 may also include a non-volatile random access memory (NVRAM). The memory 1104 stores processor and operation instructions, executable modules or data structures, or subsets thereof, or extended sets thereof, wherein the operation instructions may include various operation instructions for implementing various operations.

[0249] The processor 1103 controls the operation of the execution device. In a specific application, the various components of the execution device are coupled together through a bus system, wherein the bus system includes not only a data bus but also a power bus, a control bus, and a status signal bus, etc. However, for the sake of clarity, various buses are referred to as bus systems in the figure.

[0250] The method disclosed in the above embodiment of the present application can be applied to the processor 1103, or implemented by the processor 1103. The processor 1103 can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 1103. The above processor 1103 can be a general processor, a digital signal processor (digital signal processing, DSP), a microprocessor or a microcontroller, and can further include an application specific integrated circuit (application specific integrated circuit, ASIC), a field programmable gate array (field-programmable gate array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The processor 1103 can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to execute, or the hardware and software modules in the decoding processor can be combined and executed. The software module may be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 1104, and the processor 1103 reads the information in the memory 1104 and completes the steps of the above method in combination with its hardware.

[0251] The receiver 1101 can be used to receive input digital or character information and generate signal input related to the relevant settings and function control of the execution device. The transmitter 1102 can be used to output digital or character information through the first interface; the transmitter 1102 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group; the transmitter 1102 can also include a display device such as a display screen.

[0252] In one embodiment of the present application, the processor 1103 is configured to execute Figure 4 The task scheduling method executed by the execution device in the corresponding embodiment.

[0253] The present application also provides a training device. Fig.12 , Fig.12 It is a structural diagram of a training device provided in an embodiment of the present application. Specifically, the training device 1200 is implemented by one or more servers. The training device 1200 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 1212 (for example, one or more processors) and memory 1232, and one or more storage media 1230 (for example, one or more mass storage devices) storing application programs 1242 or data 1244. Among them, the memory 1232 and the storage medium 1230 can be short-term storage or permanent storage. The program stored in the storage medium 1230 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the training device. Furthermore, the central processing unit 1212 can be configured to communicate with the storage medium 1230 to execute a series of instruction operations in the storage medium 1230 on the training device 1200.

[0254] The training device 1200 may also include one or more power supplies 1226, one or more wired or wireless network interfaces 1250, one or more input and output interfaces 1258; or, one or more operating systems 1241, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0255] In the embodiment of the present application, the central processing unit 1212 is used to execute the steps related to the reinforcement learning network training method in the above embodiment.

[0256] Also provided in an embodiment of the present application is a computer program product which, when executed on a computer, enables the computer to execute the steps executed by the aforementioned execution device, or enables the computer to execute the steps executed by the aforementioned training device.

[0257] A computer-readable storage medium is also provided in an embodiment of the present application, which stores a program for signal processing. When the computer-readable storage medium is run on a computer, it enables the computer to execute the steps executed by the aforementioned execution device, or enables the computer to execute the steps executed by the aforementioned training device.

[0258] The execution device, training device or terminal device provided in the embodiments of the present application may specifically be a chip, and the chip includes: a processing unit and a communication unit, wherein the processing unit may be, for example, a processor, and the communication unit may be, for example, an input / output interface, a pin or a circuit, etc. The processing unit may execute the computer execution instructions stored in the storage unit so that the chip in the execution device executes the data processing method described in the above embodiment, or so that the chip in the training device executes the data processing method described in the above embodiment. Optionally, the storage unit is a storage unit in the chip, such as a register, a cache, etc. The storage unit may also be a storage unit located outside the chip in the wireless access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc.

[0259] For details, please refer to Fig.13 , Fig.13 A schematic diagram of the structure of a chip provided in an embodiment of the present application, the chip can be expressed as a neural network processor NPU 1300, NPU 1300 is mounted on the host CPU (Host CPU) as a coprocessor, and the host CPU assigns tasks. The core part of the NPU is the operation circuit 1303, which is controlled by the controller 1304 to extract matrix data from the memory and perform multiplication operations.

[0260] In some implementations, the operation circuit 1303 includes multiple processing units (Process Engine, PE) inside. In some implementations, the operation circuit 1303 is a two-dimensional systolic array. The operation circuit 1303 can also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the operation circuit 1303 is a general-purpose matrix processor.

[0261] For example, assume there is an input matrix A, a weight matrix B, and an output matrix C. The operation circuit takes the corresponding data of matrix B from the weight memory 1302 and caches it on each PE in the operation circuit. The operation circuit takes the matrix A data from the input memory 1301 and performs matrix operation with matrix B, and the partial result or final result of the matrix is ​​stored in the accumulator 1308.

[0262] The unified memory 1306 is used to store input data and output data. The weight data is directly transferred to the weight memory 1302 through the direct memory access controller (DMAC) 1305. The input data is also transferred to the unified memory 1306 through the DMAC.

[0263] BIU stands for Bus Interface Unit, i.e., bus interface unit 1310 , which is used for interaction between AXI bus, DMAC and instruction fetch buffer (IFB) 1309 .

[0264] The bus interface unit 1310 (Bus Interface Unit, BIU for short) is used for the instruction fetch memory 1309 to obtain instructions from the external memory, and is also used for the storage unit access controller 1305 to obtain the original data of the input matrix A or the weight matrix B from the external memory.

[0265] DMAC is mainly used to transfer input data in the external memory DDR to the unified memory 1306 or to transfer weight data to the weight memory 1302 or to transfer input data to the input memory 1301.

[0266] The vector calculation unit 1307 includes multiple operation processing units, and further processes the output of the operation circuit 1303 when necessary, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. It is mainly used for non-convolutional / fully connected layer network calculations in neural networks, such as Batch Normalization, pixel-level summation, upsampling of feature planes, etc.

[0267] In some implementations, the vector calculation unit 1307 can store the processed output vector to the unified memory 1306. For example, the vector calculation unit 1307 can apply a linear function; or a nonlinear function to the output of the operation circuit 1303, such as linear interpolation of the feature plane extracted by the convolution layer, and then, for example, a vector of accumulated values ​​to generate an activation value. In some implementations, the vector calculation unit 1307 generates a normalized value, a pixel-level summed value, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 1303, for example, for use in a subsequent layer in a neural network.

[0268] An instruction fetch buffer 1309 connected to the controller 1304 is used to store instructions used by the controller 1304;

[0269] Unified memory 1306, input memory 1301, weight memory 1302 and instruction fetch memory 1309 are all on-chip memories. External memories are private to the NPU hardware architecture.

[0270] The processor mentioned in any of the above places may be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits for controlling the execution of the above program.

[0271] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0272] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0273] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0274] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

Claims

1. A task scheduling method, characterized in that: The method comprises: Acquire multiple tasks to be scheduled and multiple schedulable resource groups; wherein each of the multiple schedulable resource groups includes multiple schedulable resources; Processing the plurality of tasks to be scheduled and the plurality of schedulable resource groups through a first reinforcement learning model to obtain a first allocation relationship, wherein the first allocation relationship includes a correspondence relationship between the plurality of tasks to be scheduled and the plurality of schedulable resource groups, and the first allocation relationship indicates an ordering of the plurality of schedulable resource groups corresponding to any one of the plurality of tasks to be scheduled; At least based on the first allocation relationship, the multiple tasks to be scheduled and the multiple schedulable resources are processed by a second reinforcement learning model to obtain a second allocation relationship, wherein the second allocation relationship includes a corresponding relationship between the multiple tasks to be scheduled and the multiple schedulable resources, and the second allocation relationship indicates the order of the multiple schedulable resources corresponding to any one of the multiple tasks to be scheduled; wherein the first allocation relationship serves as an input to the second reinforcement learning model, and in the second allocation relationship, each of the multiple tasks to be scheduled and the schedulable resource group to which the corresponding schedulable resource belongs satisfy the first allocation relationship, and the first reinforcement learning model and the second reinforcement learning model are connected through a sliding window sampling.

2. The method according to claim 1, characterized in that Each of the plurality of schedulable resource groups is a production line, and each of the plurality of schedulable resources is a production equipment in the production line; or, Each of the multiple schedulable resource groups is a time period, and each of the multiple schedulable resources is a sub-time period in the time period.

3. The method according to claim 1, characterized in that The processing of the plurality of tasks to be scheduled and the plurality of schedulable resources by a second reinforcement learning model based on the first allocation relationship includes: At least based on the first allocation relationship, part of the multiple tasks to be scheduled and the multiple schedulable resources are processed through a second reinforcement learning model to obtain a sub-allocation relationship, and the sub-allocation relationship represents the corresponding relationship between the part of the tasks to be scheduled and the multiple schedulable resources.

4. The method according to claim 1, characterized in that: The multiple tasks to be scheduled include multiple target tasks to be scheduled, the multiple schedulable resource groups include a target schedulable resource group, the target schedulable resource group includes multiple target schedulable resources, and in the first allocation relationship, the multiple target tasks to be scheduled correspond to the target schedulable resource group; the method further includes: In the case that less than a preset number of target schedulable resources among the multiple target schedulable resources can be allocated to at least one target task to be scheduled, a target constraint is determined, wherein the target constraint indicates that the at least one target task to be scheduled does not correspond to the target schedulable resource group.

5. The method according to claim 4, characterized in that The method further comprises: Based at least on the target constraint and the second allocation relationship, the plurality of tasks to be scheduled and the plurality of schedulable resource groups are processed by a first reinforcement learning model to obtain an updated first allocation relationship, wherein the updated first allocation relationship represents a corresponding relationship between the plurality of tasks to be scheduled and the plurality of schedulable resource groups; wherein, in the updated first allocation relationship, the at least one target task to be scheduled corresponds to a schedulable resource group among the plurality of schedulable resource groups except the target schedulable resource group; At least based on the updated first allocation relationship, the multiple tasks to be scheduled and the multiple schedulable resources are processed by the second reinforcement learning model to obtain an updated second allocation relationship, and the updated second allocation relationship represents the corresponding relationship between the multiple tasks to be scheduled and the multiple schedulable resources; wherein, in the updated second allocation relationship, each of the multiple tasks to be scheduled and the schedulable resource group to which the corresponding schedulable resource belongs satisfies the updated first allocation relationship.

6. The method according to any one of claims 1 to 5, characterized in that: The first reinforcement learning model and the second reinforcement learning model are trained separately.

7. The method according to any one of claims 1 to 5, characterized in that: The first reinforcement learning model includes a first reward function; the second reinforcement learning model includes a second reward function.

8. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: Generate a first shared vector, wherein the first shared vector includes a plurality of state variables; each state variable may represent an allocation state of a task to be scheduled or an occupied state of a schedulable resource group; The processing of the plurality of tasks to be scheduled and the plurality of schedulable resources by using a second reinforcement learning model at least based on the first allocation relationship includes: Based at least on the first sharing vector and the first allocation relationship, the plurality of tasks to be scheduled and the plurality of schedulable resources are processed by a second reinforcement learning model.

9. The method according to claim 5, characterized in that The method comprises: Generate a second shared vector, wherein the second shared vector includes a plurality of state variables; each state variable may represent an allocation state of a task to be scheduled, an occupied state of a schedulable resource group, or an occupied state of a schedulable resource; The processing of the plurality of tasks to be scheduled and the plurality of schedulable resource groups by using a first reinforcement learning model at least based on the target constraint and the second allocation relationship includes: Based at least on the second sharing vector, the target constraint and the second allocation relationship, the plurality of tasks to be scheduled and the plurality of schedulable resource groups are processed by a first reinforcement learning model.

10. A system, characterized in that: include: a first reinforcement learning model and a second reinforcement learning model implemented by one or more computers; The first reinforcement learning model is used to obtain a plurality of tasks to be scheduled and a plurality of schedulable resource groups, and process the plurality of tasks to be scheduled and the plurality of schedulable resource groups to obtain a first allocation relationship; wherein each of the plurality of schedulable resource groups includes a plurality of schedulable resources, the first allocation relationship includes a corresponding relationship between the plurality of tasks to be scheduled and the plurality of schedulable resource groups, and the first allocation relationship indicates the order of the plurality of schedulable resource groups corresponding to any one of the plurality of tasks to be scheduled; The second reinforcement learning model is used to process the multiple tasks to be scheduled and the multiple schedulable resources through the second reinforcement learning model at least based on the first allocation relationship to obtain a second allocation relationship, wherein the second allocation relationship includes the corresponding relationship between the multiple tasks to be scheduled and the multiple schedulable resources, and the second allocation relationship indicates the order of the multiple schedulable resources corresponding to any one of the multiple tasks to be scheduled; wherein the first allocation relationship is used as the input of the second reinforcement learning model, and in the second allocation relationship, each of the multiple tasks to be scheduled and the schedulable resource group to which the corresponding schedulable resource belongs satisfies the first allocation relationship, and the first reinforcement learning model and the second reinforcement learning model are connected through a sliding window sampling.

11. The system according to claim 10, characterized in that Each of the plurality of schedulable resource groups is a production line, and each of the plurality of schedulable resources is a production equipment in the production line; or, Each of the multiple schedulable resource groups is a time period, and each of the multiple schedulable resources is a sub-time period in the time period.

12. The system according to claim 10, characterized in that The second reinforcement learning model is used to process part of the multiple tasks to be scheduled and the multiple schedulable resources through the second reinforcement learning model based at least on the first allocation relationship, so as to obtain a sub-allocation relationship, wherein the sub-allocation relationship represents the correspondence between the part of the tasks to be scheduled and the multiple schedulable resources.

13. The system according to claim 10, characterized in that The multiple tasks to be scheduled include multiple target tasks to be scheduled, the multiple schedulable resource groups include a target schedulable resource group, the target schedulable resource group includes multiple target schedulable resources, and in the first allocation relationship, the multiple target tasks to be scheduled correspond to the target schedulable resource group; The second reinforcement learning model is also used to determine a target constraint when less than a preset number of target schedulable resources among the multiple target schedulable resources can be allocated to at least one target task to be scheduled, and the target constraint indicates that the at least one target task to be scheduled does not correspond to the target schedulable resource group.

14. The system according to claim 13, characterized in that The first reinforcement learning model is also used to: Based at least on the target constraint and the second allocation relationship, the plurality of tasks to be scheduled and the plurality of schedulable resource groups are processed by a first reinforcement learning model to obtain an updated first allocation relationship, wherein the updated first allocation relationship represents a corresponding relationship between the plurality of tasks to be scheduled and the plurality of schedulable resource groups; wherein, in the updated first allocation relationship, the at least one target task to be scheduled corresponds to a schedulable resource group among the plurality of schedulable resource groups except the target schedulable resource group; The second reinforcement learning model is also used to: At least based on the updated first allocation relationship, the multiple tasks to be scheduled and the multiple schedulable resources are processed by the second reinforcement learning model to obtain an updated second allocation relationship, and the updated second allocation relationship represents the corresponding relationship between the multiple tasks to be scheduled and the multiple schedulable resources; wherein, in the updated second allocation relationship, each of the multiple tasks to be scheduled and the schedulable resource group to which the corresponding schedulable resource belongs satisfies the updated first allocation relationship.

15. The system according to any one of claims 10 to 14, characterized in that: The first reinforcement learning model and the second reinforcement learning model are trained separately.

16. The system according to any one of claims 10 to 14, characterized in that: The first reinforcement learning model includes a first reward function; the second reinforcement learning model includes a second reward function.

17. The system according to any one of claims 10 to 14, characterized in that: The first reinforcement learning model is further used to: generate a first shared vector, the first shared vector comprising a plurality of state variables; each state variable may represent an allocation state of a task to be scheduled or an occupied state of a schedulable resource group; The second reinforcement learning model is used to: process the multiple tasks to be scheduled and the multiple schedulable resources through the second reinforcement learning model based at least on the first sharing vector and the first allocation relationship.

18. The system according to any one of claims 10 to 14, characterized in that: The first reinforcement learning model is further used to: generate a second shared vector, the second shared vector comprising a plurality of state variables; wherein each state variable may represent an allocation state of a task to be scheduled, an occupied state of a schedulable resource group, or an occupied state of a schedulable resource; The second reinforcement learning model is used to: process the multiple tasks to be scheduled and the multiple schedulable resource groups through the first reinforcement learning model based at least on the second sharing vector, the target constraint and the second allocation relationship.

19. The system according to any one of claims 10 to 14, characterized in that: The first reinforcement learning model includes DDQN, and the second reinforcement learning model includes a pointer network PN.

20. The system according to any one of claims 10 to 14, characterized in that: The first reinforcement learning model and the second reinforcement learning model include recurrent neural networks.

21. The system according to claim 20, characterized in that The first reinforcement learning model includes two long short-term memory networks LSTM based on the attention mechanism, and the second reinforcement learning model includes a first recurrent neural network RNN ​​for encoding and a second RNN for decoding.

22. A task scheduling device, characterized in that: The device comprises: An acquisition module, used to acquire a plurality of tasks to be scheduled and a plurality of schedulable resource groups; wherein each of the plurality of schedulable resource groups includes a plurality of schedulable resources; A first reinforcement learning module, configured to process the plurality of tasks to be scheduled and the plurality of schedulable resource groups through a first reinforcement learning model to obtain a first allocation relationship, wherein the first allocation relationship includes a correspondence relationship between the plurality of tasks to be scheduled and the plurality of schedulable resource groups, and the first allocation relationship indicates an ordering of the plurality of schedulable resource groups corresponding to any one of the plurality of tasks to be scheduled; A second reinforcement learning module is used to process the multiple tasks to be scheduled and the multiple schedulable resources through a second reinforcement learning model at least based on the first allocation relationship to obtain a second allocation relationship, wherein the second allocation relationship includes a corresponding relationship between the multiple tasks to be scheduled and the multiple schedulable resources, and the second allocation relationship indicates the order of the multiple schedulable resources corresponding to any one of the multiple tasks to be scheduled; wherein the first allocation relationship is used as an input of the second reinforcement learning model, and in the second allocation relationship, each of the multiple tasks to be scheduled and the schedulable resource group to which the corresponding schedulable resource belongs satisfies the first allocation relationship, and the first reinforcement learning model and the second reinforcement learning model are connected through a sliding window sampling.

23. The device according to claim 22, characterized in that Each of the plurality of schedulable resource groups is a production line, and each of the plurality of schedulable resources is a production equipment in the production line; or, Each of the multiple schedulable resource groups is a time period, and each of the multiple schedulable resources is a sub-time period in the time period.

24. The device according to claim 22, characterized in that The second reinforcement learning module is specifically used for: At least based on the first allocation relationship, part of the multiple tasks to be scheduled and the multiple schedulable resources are processed through a second reinforcement learning model to obtain a sub-allocation relationship, and the sub-allocation relationship represents the corresponding relationship between the part of the tasks to be scheduled and the multiple schedulable resources.

25. The device according to claim 22, characterized in that The multiple tasks to be scheduled include multiple target tasks to be scheduled, the multiple schedulable resource groups include a target schedulable resource group, the target schedulable resource group includes multiple target schedulable resources, and in the first allocation relationship, the multiple target tasks to be scheduled correspond to the target schedulable resource group; The second reinforcement learning module is specifically used for: In the case that less than a preset number of target schedulable resources among the multiple target schedulable resources can be allocated to at least one target task to be scheduled, a target constraint is determined, wherein the target constraint indicates that the at least one target task to be scheduled does not correspond to the target schedulable resource group.

26. The device according to claim 25, characterized in that The first reinforcement learning module is further used to process the multiple tasks to be scheduled and the multiple schedulable resource groups through a first reinforcement learning model based at least on the target constraint and the second allocation relationship to obtain an updated first allocation relationship, wherein the updated first allocation relationship represents a corresponding relationship between the multiple tasks to be scheduled and the multiple schedulable resource groups; wherein, in the updated first allocation relationship, the at least one target task to be scheduled corresponds to a schedulable resource group among the multiple schedulable resource groups except the target schedulable resource group; The second reinforcement learning module is also used to process the multiple tasks to be scheduled and the multiple schedulable resources through the second reinforcement learning model at least based on the updated first allocation relationship to obtain an updated second allocation relationship, and the updated second allocation relationship represents the corresponding relationship between the multiple tasks to be scheduled and the multiple schedulable resources; wherein, in the updated second allocation relationship, each of the multiple tasks to be scheduled and the schedulable resource group to which the corresponding schedulable resource belongs satisfies the updated first allocation relationship.

27. The device according to any one of claims 22 to 26, characterized in that The first reinforcement learning model and the second reinforcement learning model are trained separately.

28. The device according to any one of claims 22 to 26, characterized in that The first reinforcement learning model includes a first reward function; the second reinforcement learning model includes a second reward function.

29. The device according to any one of claims 22 to 26, characterized in that The first reinforcement learning module is further used to generate a first shared vector, wherein the first shared vector includes a plurality of state variables; each state variable may represent an allocation state of a task to be scheduled or an occupied state of a schedulable resource group; The second reinforcement learning module is further used to process the multiple tasks to be scheduled and the multiple schedulable resources through a second reinforcement learning model based at least on the first shared vector and the first allocation relationship.

30. The device according to claim 26, characterized in that The second reinforcement learning module is further used to generate a second shared vector, wherein the second shared vector includes a plurality of state variables; each state variable may represent an allocation state of a task to be scheduled, an occupied state of a schedulable resource group, or an occupied state of a schedulable resource; The first reinforcement learning module is further used to process the multiple tasks to be scheduled and the multiple schedulable resource groups through a first reinforcement learning model based at least on the second shared vector, the target constraint and the second allocation relationship.

31. A computer storage medium, characterized in that The computer storage medium stores one or more instructions, which, when executed by one or more computers, cause the one or more computers to perform the operations of the method of any one of claims 1-9.

32. A computer storage medium, characterized in that The computer storage medium stores one or more instructions, which, when executed by one or more computers, cause the one or more computers to implement the system of any one of claims 10-21.

33. A system comprising at least one processor, at least one memory and at least one communication interface; the processor, the memory and the communication interface are connected via a communication bus and communicate with each other; The at least one memory is used to store code; The at least one processor is used to execute the code to execute the task scheduling method according to any one of claims 1 to 9 to obtain a task scheduling result; The at least one communication interface is used to communicate with a device or a communication network to send the task scheduling result to the device or the communication network.