A service resource scheduling method based on self-distillation hierarchical reinforcement learning

Abstract

The present application provides a service resource scheduling method based on self-distillation hierarchical reinforcement learning, which involves the intersection of artificial intelligence and complex networks. The method includes: based on multiple configured time nodes, obtaining the initial state vector corresponding to multiple service resource packages to be processed at any time node; using the encoder module in the self-distillation hierarchical reinforcement learning network to process the initial state vector of the service resource package to obtain the teacher strategy representation vector of the service resource package at the time node; using the strategy network module to process the teacher strategy representation vector to determine the target strategy representation vector of the service resource package at the time node; the present application improves the performance of the self-distillation hierarchical reinforcement learning network through self-distillation technology, reduces the amount of calculation, and improves the effectiveness of scheduling decisions.

Description

Technical Field

[0001] The present application relates to the intersection of artificial intelligence and complex networks, and specifically, to a service resource scheduling method based on self-distillation hierarchical reinforcement learning. Background Art

[0002] The resource allocation problem of service networks can be modeled as a shortest path optimization problem with constraints and fixed topology. Traditional methods are not advantageous in dealing with various constraints and information interactions in the optimization of shortest paths for multi-objective individuals. The solution is cumbersome, and even in most cases, the Nash equilibrium solution cannot be obtained. The method based on deep reinforcement learning has achieved certain results in the study of service resource allocation problems in service network environments. However, for the scheduling problem of large-scale service resources, the problem is complex and the decision-making is difficult. When searching for the optimal decision, there are problems and challenges in the large action search space. Hierarchical reinforcement learning is used to solve the problem of searching for the optimal action in the action space. Hierarchical reinforcement learning is designed as a multi-level decision-making framework, which divides the decision-making process into high-level decisions and low-level decisions. High-level decisions provide sub-goals to guide low-level operations in this structure. High-level sub-goals significantly reduce the complexity of the action space, thereby improving the efficiency of the decision-making process; however, hierarchical reinforcement learning is rarely used in resource scheduling problems, especially in a large number of resource scheduling tasks in complex service networks. In these complex service networks, scheduling tasks involving large-scale resource packages remains a difficult problem to be solved due to the increased computational load overhead and limited scalability of hierarchical reinforcement learning methods. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide a service resource scheduling method based on self-distillation hierarchical reinforcement learning, which is used to solve the above-mentioned problems existing in the prior art, reduce the amount of calculation in the server resource scheduling process, and improve the calculation accuracy.

[0004] The present application provides a service resource scheduling method based on self-distillation hierarchical reinforcement learning, which may include:

[0005] Based on multiple configured time nodes, obtaining initial state vectors corresponding to multiple service resource packages to be processed at any time node;

[0006] For any service resource package, the encoder module in the self-distillation hierarchical reinforcement learning network is used to process the initial state vector of the service resource package to obtain the teacher strategy representation vector;

[0007] The strategy fitting function configured in the strategy network module in the self-distillation hierarchical reinforcement learning network is used to process the teacher strategy representation vector to determine the target strategy representation vector of the service resource package at this time node.

[0008] In a possible implementation, the encoder module includes: a student encoder and a teacher encoder;

[0009] The student encoder includes a student local attention submodule; the teacher encoder includes a teacher local attention submodule.

[0010] In a possible implementation, the encoder module in the self-distillation hierarchical reinforcement learning network is used to process the initial state vector of the service resource package to obtain the teacher strategy representation vector, including:

[0011] Using the student local attention submodule, the initial state vector of the service resource package is processed to obtain the feature perception attention weight of the service resource package at the time node;

[0012] Processing the feature perception attention weight and the initial state vector to obtain a student strategy representation vector output by the student encoder;

[0013] The teacher local attention submodule is used to process the student strategy representation vector and the initial state vector to obtain the attention weight matrix of the service resource package at the time node;

[0014] The attention weight matrix and the initial state vector are processed to obtain a teacher strategy representation vector output by the teacher encoder.

[0015] In a possible implementation, the self-distillation hierarchical reinforcement learning network further includes a self-distillation module;

[0016] After determining the teacher strategy representation vector output by the teacher encoder, the method further includes:

[0017] The network parameters of the self-distillation hierarchical reinforcement learning network are optimized through the self-distillation loss function configured in the self-distillation module to obtain optimized network parameters.

[0018] In one possible implementation, the self-distillation loss function is:

[0019]

[0020] Among them, L SD (·) is the self-distillation loss function, M is the number of service resource packages at time node t, Q tea (·) and Q stu (·) are the strategy fitting function and the student strategy fitting function, and are the teacher strategy representation vector and the student strategy representation vector respectively, θ SD Network parameters for self-distilled hierarchical reinforcement learning networks.

[0021] In a possible implementation, the learning process of the self-distillation hierarchical reinforcement learning network includes an experience storage stage and a training stage.

[0022] In a possible implementation, during the experience storage phase, an experience cell data set is collected for each time node; the experience cell data set includes: an initial state vector, a target strategy representation vector, a reward vector, a dynamic neighbor node, and a transportation state of any service resource package at each time node.

[0023] In a possible implementation, during the training phase, the network parameters and encoder parameters of the self-distillation hierarchical reinforcement learning network are optimized through the configured final loss function to obtain optimized network parameters and encoder parameters.

[0024] In a possible implementation, the final loss function is:

[0025]

[0026] in, is the final loss function, Q tea (·) and Q stu (·) are the strategy fitting function and the student strategy fitting function, t is the time node, θ Q Q tea (·) and Q stu The parameter of (·), θ SD are the network parameters of the self-distilled hierarchical reinforcement learning network, and are two normalization hyperparameters, is the initial state vector of the service resource package at time node t is the expected function, L SD (·) is the self-distillation loss function, and They are target value and student target value respectively.

[0027] The present application provides a service resource scheduling method based on self-distillation hierarchical reinforcement learning, the method comprising: based on the configured time nodes, obtaining the initial state vectors corresponding to multiple service resource packages to be processed at any time node; for any service resource package, using the encoder module in the self-distillation hierarchical reinforcement learning network to process the initial state vector of the service resource package to obtain a teacher strategy representation vector; using the strategy fitting function configured by the strategy network module in the self-distillation hierarchical reinforcement learning network to process the teacher strategy representation vector and determine the target strategy representation vector of the service resource package at the time node; the present application improves the performance of the self-distillation hierarchical reinforcement learning network through self-distillation technology, reduces the amount of calculation, and thus improves the effectiveness of scheduling decisions. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0029] Figure 1 A system architecture diagram for a service resource scheduling method based on self-distillation hierarchical reinforcement learning provided in an embodiment of the present application;

[0030] Figure 2 A flow chart of a service resource scheduling method based on self-distillation hierarchical reinforcement learning provided in an embodiment of the present application;

[0031] Figure 3 A schematic diagram of the structure of a self-distillation hierarchical reinforcement learning network provided in an embodiment of the present application;

[0032] Figure 4 A schematic diagram of the structure of a service resource scheduling device based on self-distillation hierarchical reinforcement learning provided in an embodiment of the present application;

[0033] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0035] The service resource scheduling method based on self-distillation hierarchical reinforcement learning provided in the embodiment of the present application can be applied to Figure 1 In the system architecture shown in Figure 1 As shown, the system may include: a server and a terminal. The server may be a physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and basic cloud computing services such as big data and artificial intelligence platforms. The terminal may be a user equipment (UE) such as a mobile phone, smart phone, laptop, digital broadcast receiver, personal digital assistant (PDA), tablet computer (PAD), handheld device, vehicle-mounted device, wearable device, computing device or other processing device connected to a wireless modem, mobile station (MS), mobile terminal, etc. The terminal and the server may be directly or indirectly connected via wired or wireless communication, which is not limited in this application.

[0036] The terminal is used to obtain the service resource package to be processed and send the service resource package to the server.

[0037] The server is used to receive a service resource package to be processed to execute a service resource scheduling method based on self-distillation hierarchical reinforcement learning provided in the present application.

[0038] With the rapid development of modern service industry, the service industry is increasingly showing complex and diverse characteristics. This complexity is no longer just a one-way dependence on agriculture or industry, but a combination of multiple industrial chains, forming a tightly coupled complex interactive network. This transformation not only reshapes the internal structure of the service system, but also redefines the collaboration model between industries. In this context, service resources play a vital role in the service industry system. They are the core support for the normal operation of the service system, providing the guarantee of material resources and virtual information resources, thereby supporting the service system to achieve its core functions. Therefore, how to effectively dispatch service resources has become one of the key issues in service network research. The development of the service industry is of great significance to the economy and society. It has promoted the improvement of production efficiency, promoted innovation and development, and provided users with a more personalized and high-quality service experience. In this process, the efficient dispatch of service resources is the key to achieving sustained growth and competitiveness of the service industry. Through scientific resource dispatching, the utilization rate of resources can be maximized, costs can be reduced, and the response speed and quality of services can be improved, thereby enhancing the market competitiveness of enterprises.

[0039] The significance of service resource scheduling is to ensure that limited physical and virtual resources can be efficiently utilized to meet the needs of different types of service requests. In a complex and changing service environment, the rationality of resource scheduling directly affects the overall performance and service quality of the system. Especially in cloud computing and edge computing scenarios, the optimization of resource scheduling can significantly improve the response speed of the system, reduce network latency, and achieve fair allocation of resources among multiple service requests. At the same time, a good resource scheduling strategy can effectively reduce energy consumption and unnecessary resource waste, thereby achieving the goal of green computing.

[0040] At the macro level, facing a service network with a complex topological structure, rationally dispatching target resources and achieving efficient transportation are the key to improving the production, management and operation efficiency of the service network, and are also an important part of the intelligent service network. The formation of a complex service network reflects the deep integration of diversified and multi-channel elements in the primary and secondary industries. This network is no longer a traditional single transmission chain of resources-channels-users, but a complex ecosystem composed of multiple elements such as service sites, resources, suppliers, users, manufacturers, and distribution centers. Through the diverse connections between these nodes, the topological structure of the service network has become increasingly complex, with non-Euclidean characteristics, which can effectively express various relationships and interactions in the service process. In a transportation connection structure with a fixed topology, when different resource scheduling individuals choose a path, they often compete for resources due to the constraints of load and benefit cost functions. Therefore, how to improve the overall efficiency of resource scheduling in complex service networks, minimize transportation consumption, and complete resource scheduling quickly and accurately is the core issue of service network resource scheduling research.

[0041] Generally speaking, the resource allocation problem of a service network can be modeled as a constrained shortest path optimization problem with a fixed topology. The traditional shortest path optimization problem with the Floyd algorithm as the core can achieve certain results for simple topologies or simple constrained problems. However, this type of tradition is not advantageous in dealing with various constraints and information interactions in the shortest path optimization of multi-objective individuals. The solution is cumbersome, and even in most cases, the Nash equilibrium solution cannot be obtained. These traditional methods have certain effects in small-scale, static environments, but when faced with dynamic changes in resources, complex service dependencies, and the diversity of task requirements, their limitations gradually emerge. This also provides an opportunity for the emergence of new intelligent scheduling methods. In recent years, deep reinforcement learning has become an emerging tool for solving service resource scheduling problems due to its powerful ability to learn strategies in dynamic environments. Deep reinforcement learning combines the decision-making ability of reinforcement learning with the expression ability of deep neural networks, and can adaptively learn resource scheduling strategies in complex, diverse, and dynamic environments, thereby improving the overall performance and resource utilization of the system. Deep reinforcement learning is a learning method based on the interaction between agents and the environment. Agents obtain rewards by taking actions in the environment and continuously optimize their behavior strategies to maximize the cumulative rewards. In the context of service resource scheduling, the environment can be regarded as the entire computing resource pool, and the actions of the agents are scheduling decisions, such as assigning tasks to specific server nodes. The reward signal is usually related to system performance indicators, such as task response time, resource utilization, and overall service quality. Methods based on deep reinforcement learning have achieved certain results in the study of service resource allocation problems in service network environments. In the study of this problem, the benefit function and cost function are usually modeled as action transfer reward functions. In view of the complex topological connection structure of the service network and various constrained transfer restrictions, the action value Q network is used to calculate the decision value, and then the decision network is optimized through back propagation of the loss function. In the face of general decision problems, the deep Q learning algorithm calculates the action value function, constructs the network cost function based on the time difference principle, optimizes the feature extraction and decision network, outputs the action decision value value, and uses the greedy strategy action decision selection strategy to achieve end-to-end decision-making. Deep Q-learning mainly solves the problem of discrete finite-dimensional action decision-making. In general, the resource allocation problem is the decision-making problem of resource target units facing finite discrete paths. Therefore, deep Q-learning has become one of the technical means to solve the resource allocation problem. The results show that the method based on deep reinforcement learning has better results than the traditional analytical algorithm.

[0042] However, for the scheduling problem of large-scale service resources, the problem is complex and the decision-making is difficult. When searching for the optimal decision, there are problems and challenges in the large action search space. These limitations make it challenging to effectively apply deep reinforcement learning in large-scale resource scheduling tasks. Hierarchical reinforcement learning is used to solve the problem of searching for the optimal action in the action space. Hierarchical reinforcement learning is designed as a multi-level decision-making framework that divides the decision-making process into high-level decisions and low-level decisions. High-level decisions provide sub-goals to guide low-level operations in this structure. High-level sub-goals significantly reduce the complexity of the action space, thereby improving the efficiency of the decision-making process. However, hierarchical reinforcement learning is rarely applied to resource scheduling problems, especially in the large number of resource scheduling tasks in complex service networks. In these complex service networks, due to the increased computational load overhead and limited scalability of hierarchical reinforcement learning methods, it is still a difficult problem to be solved in scheduling tasks involving large-scale resource packages.

[0043] The resource allocation problem of service networks can be modeled as a shortest path optimization problem with constraints and fixed topology. Traditional methods are not advantageous in dealing with various constraints and information interactions in the optimization of shortest paths for multi-objective individuals. The solution is cumbersome, and even in most cases, the Nash equilibrium solution cannot be obtained. The method based on deep reinforcement learning has achieved certain results in the study of service resource allocation problems in service network environments. However, for the scheduling problem of large-scale service resources, the problem is complex and the decision-making is difficult. When searching for the optimal decision, there are problems and challenges in the large action search space. Hierarchical reinforcement learning is used to solve the problem of searching for the optimal action in the action space. Hierarchical reinforcement learning is designed as a multi-level decision-making framework, which divides the decision-making process into high-level decisions and low-level decisions. High-level decisions provide sub-goals to guide low-level operations in this structure. High-level sub-goals significantly reduce the complexity of the action space, thereby improving the efficiency of the decision-making process; however, hierarchical reinforcement learning is rarely used in resource scheduling problems, especially in a large number of resource scheduling tasks in complex service networks. In these complex service networks, scheduling tasks involving large-scale resource packages remains a difficult problem to be solved due to the increased computational load overhead and limited scalability of hierarchical reinforcement learning methods.

[0044] Therefore, this application proposes a service resource scheduling method based on self-distillation hierarchical reinforcement learning to solve the above-mentioned problems existing in the prior art, which can reduce the amount of calculation in the server resource scheduling process and improve the calculation accuracy.

[0045] The preferred embodiments of the present application are described below in conjunction with the drawings in the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application. In addition, the embodiments and features in the embodiments of the present application may be combined with each other if there is no conflict.

[0046] Figure 2 A flow chart of a service resource scheduling method based on self-distillation hierarchical reinforcement learning provided in an embodiment of the present application. Figure 2 As shown, the method may include:

[0047] Step S210: Based on the configured multiple time nodes, obtain the initial state vector corresponding to the multiple service resource packages to be processed at any time node.

[0048] Specifically, the resource scheduling problem is regarded as a discrete-time decision-making problem of multiple agents with topological constraints; that is, a large number of resource scheduling tasks are regarded as multiple service resource packages to be processed; each service resource package can be regarded as an agent that can move in a complex industrial network with a relatively fixed topology. The ultimate goal is to efficiently transfer multiple service resource packages from the initial resource station to the target resource station within a limited number of steps.

[0049] The configured overall processing time is divided according to the configured time step to obtain multiple time nodes, and the method provided by the present application is used at each time node to schedule multiple service resource packages; any service resource package can move along the transmission path or choose to stay at the initial resource station. Once a service resource package arrives at the target resource station, another service resource package with different characteristics located at the target resource station will be assigned to another scheduling task of the newly designated target resource station. Furthermore, the method provided by the present application refers to execution at each time node.

[0050] Viewing Resource Stations as Complex Service Networks The nodes in are defined as ,in, Represents the service network The number of resource stations in the system.

[0051] Assume that in these complex service networks There are M ( ) service resource packages flow, multiple service resource packages are composed of a collection Description. According to existing work, each service resource package has an initial state vector. When , the initial state vectors of multiple service resource packages form the initial state matrix, which is recorded as Assume that each agent can only observe the adjacent resource stations and the agent at the current resource station.

[0052] Therefore, the resource station Service resource package The initial state vector Includes: 1) Service resource package Attribute information, including location, size, type, priority, etc.; 2) Environmental information (information of resource stations), including its location resource station and the attribute information of adjacent resource stations; 3) Resource stations Or adjacent resource station Other service resource packages.

[0053] Step S220: For any service resource package, use the encoder module in the self-distillation hierarchical reinforcement learning network to process the initial state vector of the service resource package to obtain a teacher strategy representation vector.

[0054] Among them, the encoder module includes a student encoder and a teacher encoder;

[0055] The student encoder includes a student local attention submodule; the teacher encoder includes a teacher local attention submodule.

[0056] Specifically, step S221, for any service resource package, the student local attention submodule is used to process the initial state vector of the service resource package to obtain the service resource package Feature perception attention weight at this time node;

[0057] The process can be expressed as:

[0058]

[0059] in, Resource Pack for Services and The feature-aware attention weight at time node t, for Resource Station V i The service resource package of the adjacent resource station, is a multi-layer perceptron model, and are the first learning weight matrix and the second learning weight matrix in the student encoder, respectively. At time node t The initial state vector.

[0060] Step S222, processing the feature perception attention weight and the initial state vector to obtain a student strategy representation vector output by the student encoder;

[0061] It can also be understood as the student strategy representation vector of the agent to the student encoder;

[0062] The process can be expressed as:

[0063]

[0064] in, is the student strategy representation vector of the service resource package at time node t, is the third learned weight matrix in the student encoder.

[0065] Step S223: Using the teacher's local attention submodule, process the student's strategy representation vector and the initial state vector to obtain the attention weight matrix of the service resource package at the time node;

[0066] The process can be expressed as:

[0067]

[0068] in, is the attention weight matrix of the service resource package at time node t, and are the first and second learning weight matrices in the teacher encoder, respectively, and g(·) is the MLP model.

[0069] Step S224, process the attention weight matrix and the initial state vector to obtain the teacher strategy representation vector output by the teacher encoder.

[0070] The process can be expressed as:

[0071]

[0072] in, is the teacher strategy representation vector of the service resource package at time node t, is the third learning weight matrix in the teacher encoder.

[0073] The process of step S223 and step S224 can be understood as: taking the initial state vector of the service resource package and the student strategy representation vector as the input of the teacher encoder, effectively assisting in obtaining the decision representation (teacher strategy representation vector).

[0074] In some embodiments, the self-distillation hierarchical reinforcement learning network further comprises a self-distillation module;

[0075] After determining the strategy representation vector output by the teacher encoder, the method may further include:

[0076] The network parameters of the self-distillation hierarchical reinforcement learning network are optimized through the self-distillation loss function configured in the self-distillation module to obtain the optimized network parameters.

[0077] The self-distillation loss function can be expressed as:

[0078]

[0079] Among them, L SD (·) is the self-distillation loss function, M is the number of service resource packages at time node t, Q tea (·) and Q stu (·) are the strategy fitting function and the student strategy fitting function, and are the teacher strategy representation vector and the student strategy representation vector respectively, θ SD Network parameters for self-distilled hierarchical reinforcement learning networks.

[0080] Furthermore, the strategy fitting function and the student strategy fitting function are obtained by fitting the student strategy representation vector and the teacher strategy representation vector corresponding to the time node t respectively through the MLP model.

[0081] Step S230: Use the strategy fitting function configured in the strategy network module in the self-distillation hierarchical reinforcement learning network to process the teacher strategy representation vector and determine the target strategy representation vector of the service resource package at the time node.

[0082] Specifically, based on the deep Q-learning method, using the Q-network corresponding Evaluate the teacher strategy representation vector and obtain the service resource package The optimal strategy at time node t, that is, the target strategy representation vector.

[0083] The process can be expressed as:

[0084]

[0085] in, Resource Pack for Services The optimal strategy at time node t (target strategy representation vector).

[0086] In some embodiments, the learning process of the self-distillation hierarchical reinforcement learning network includes an experience storage phase and a training phase.

[0087] During the experience storage phase, the experience cell data set for each time node is collected. ;

[0088] The empirical cell data set includes: the initial state vector of any service resource package at each time node , target strategy representation vector , reward vector , Dynamic neighbor nodes (adjacent resource stations) and transport status c t , , in this case, express The target resource station has been reached. express The target resource station was not reached.

[0089] During the training phase, the network parameters and encoder parameters of the self-distillation hierarchical reinforcement learning network are optimized through the configured final loss function to obtain the optimized network parameters and encoder parameters.

[0090] The final loss function can be expressed as:

[0091]

[0092] Among them, L(·) is the final loss function, Q tea (·) and Q stu (·) are the strategy fitting function and the student strategy fitting function, t is the time node, θ Q Q tea (·) and Q stu The parameter of (·), θ SD are the network parameters of the self-distilled hierarchical reinforcement learning network, and are two normalization hyperparameters, is the initial state vector of the service resource package at time node t, is the expected function, L SD (·) is the self-distillation loss function, and They are target value and student target value respectively.

[0093] This approach uses the Q-learning method to incorporate the Q function into the loss function of the decision-making process.

[0094] The target value and student target value can be expressed as:

[0095]

[0096]

[0097] Among them, r it is the reward vector at time node t, γ is a hyperparameter, Q'tea(·) and Q'stu (·) are the target strategy fitting function and the target student strategy fitting function respectively; it should be noted that Q tea (·) and Q stu (·) can be understood as updating the fitting function of the corresponding parameters at each iteration during the iterative process of the self-distillation hierarchical reinforcement learning network; Q'tea(·) and Q'stu (·) can be understood as updating the fitting function of the corresponding parameters after each preset number of iterations during the iterative process of the self-distillation hierarchical reinforcement learning network.

[0098] In summary, combined Figure 3 As shown, the self-distillation hierarchical reinforcement learning network may include: an encoder module, a self-distillation module, a policy network module, and a cell data experience pool for storing a meta-experience cell data set.

[0099] It should be noted that after obtaining the optimal strategy, steps S210 to S230 are repeatedly executed, and the environment is interacted with to obtain the corresponding experience cell data set at each time node. , train the self-distilled hierarchical reinforcement learning network until the corresponding algorithm converges.

[0100] The present application provides a service resource scheduling method based on self-distillation hierarchical reinforcement learning, the method comprising: based on the configured time nodes, obtaining the initial state vectors corresponding to multiple service resource packages to be processed at any time node; for any service resource package, using the encoder module in the self-distillation hierarchical reinforcement learning network to process the initial state vector of the service resource package to obtain a teacher strategy representation vector; using the strategy fitting function configured by the strategy network module in the self-distillation hierarchical reinforcement learning network to process the teacher strategy representation vector and determine the target strategy representation vector of the service resource package at the time node; the present application improves the performance of the self-distillation hierarchical reinforcement learning network through self-distillation technology, reduces the amount of calculation, and thus improves the effectiveness of scheduling decisions.

[0101] Corresponding to the above method, the embodiment of the present application also provides a service resource scheduling device based on self-distillation hierarchical reinforcement learning, such as Figure 4 As shown, the device comprises:

[0102] An acquisition unit 410 is configured to acquire, according to multiple configured time nodes, initial state vectors corresponding to multiple service resource packages to be processed at any time node;

[0103] The processing unit 420 is used to process the initial state vector of any service resource package using the encoder module in the self-distillation hierarchical reinforcement learning network to obtain a teacher strategy representation vector;

[0104] Furthermore, the strategy fitting function configured in the strategy network module in the self-distillation hierarchical reinforcement learning network is used to process the teacher strategy representation vector to determine the target strategy representation vector of the service resource package at this time node.

[0105] The functions of each functional unit of a service resource scheduling device based on self-distillation hierarchical reinforcement learning provided in the above-mentioned embodiment of the present application can be realized through the above-mentioned method steps. Therefore, the specific working process and beneficial effects of each unit in a service resource scheduling device based on self-distillation hierarchical reinforcement learning provided in the embodiment of the present application are not repeated here.

[0106] The present application also provides an electronic device, such as Figure 5 As shown, it includes a processor 510 , a communication interface 520 , a memory 530 and a communication bus 540 , wherein the processor 510 , the communication interface 520 , and the memory 530 communicate with each other via the communication bus 540 .

[0107] Memory 530, for storing computer programs;

[0108] The processor 510 is used to execute the program stored in the memory 530 to implement the following steps:

[0109] Based on multiple configured time nodes, obtaining initial state vectors corresponding to multiple service resource packages to be processed at any time node;

[0110] For any service resource package, the encoder module in the self-distillation hierarchical reinforcement learning network is used to process the initial state vector of the service resource package to obtain the teacher strategy representation vector;

[0111] The strategy fitting function configured in the strategy network module in the self-distillation hierarchical reinforcement learning network is used to process the teacher strategy representation vector to determine the target strategy representation vector of the service resource package at this time node.

[0112] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0113] The communication interface is used for communication between the above electronic device and other devices.

[0114] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0115] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0116] The implementation methods and beneficial effects of the components of the electronic device in the above embodiments to solve the problems can be found in Figure 2 The various steps in the illustrated embodiment are implemented, therefore, the specific working process and beneficial effects of the electronic device provided by the embodiment of the present application are not repeated here.

[0117] In another embodiment provided in the present application, a computer-readable storage medium is also provided, which stores instructions. When the computer-readable storage medium is executed on a computer, the computer executes a service resource scheduling method based on self-distillation hierarchical reinforcement learning as described in any of the above embodiments.

[0118] In another embodiment provided in the present application, a computer program product comprising instructions is also provided, which, when executed on a computer, enables the computer to execute a service resource scheduling method based on self-distillation hierarchical reinforcement learning as described in any of the above embodiments.

[0119] Those skilled in the art will appreciate that the embodiments in the present application can be provided as methods, systems, or computer program products. Therefore, the embodiments in the present application can be in the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments in the present application can be in the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0120] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0121] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0122] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0123] Unless otherwise defined, the technical terms or scientific terms used in this application should be understood by people with ordinary skills in the field to which the present invention belongs. "First", "second" and similar words used in this application do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect", "couple" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0124] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they are aware of the basic creative concepts. Therefore, the present application embodiments are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present application embodiments.

[0125] Obviously, those skilled in the art can make various changes and modifications to the embodiments in the present application without departing from the spirit and scope of the embodiments in the present application. Thus, if these modifications and variations of the embodiments in the present application are within the scope of the embodiments in the present application and their equivalents, the embodiments in the present application are also intended to include these modifications and variations.

Claims

1. A service resource scheduling method based on self-distillation hierarchical reinforcement learning, characterized in that: The method comprises: Based on multiple configured time nodes, obtaining initial state vectors corresponding to multiple service resource packages to be processed at any time node; For any service resource package, the encoder module in the self-distillation hierarchical reinforcement learning network is used to process the initial state vector of the service resource package to obtain the teacher strategy representation vector; The strategy fitting function configured in the strategy network module in the self-distillation hierarchical reinforcement learning network is used to process the teacher strategy representation vector to determine the target strategy representation vector of the service resource package at the time node; Wherein, the encoder module includes: a student encoder and a teacher encoder; The student encoder includes a student local attention submodule; the teacher encoder includes a teacher local attention submodule; The encoder module in the self-distillation hierarchical reinforcement learning network is used to process the initial state vector of the service resource package to obtain the teacher strategy representation vector, including: Using the student local attention submodule, the initial state vector of the service resource package is processed to obtain the feature perception attention weight of the service resource package at the time node; Processing the feature perception attention weight and the initial state vector to obtain a student strategy representation vector output by the student encoder; The teacher local attention submodule is used to process the student strategy representation vector and the initial state vector to obtain the attention weight matrix of the service resource package at the time node; The attention weight matrix and the initial state vector are processed to obtain a teacher strategy representation vector output by the teacher encoder.

2. The method according to claim 1, characterized in that The self-distillation hierarchical reinforcement learning network also includes a self-distillation module; After determining the teacher strategy representation vector output by the teacher encoder, the method further includes: The network parameters of the self-distillation hierarchical reinforcement learning network are optimized through the self-distillation loss function configured in the self-distillation module to obtain optimized network parameters.

3. The method according to claim 2, characterized in that The self-distillation loss function is: Among them, L SD (·) is the self-distillation loss function, M is the number of service resource packages at time node t, Q tea (·) and Q stu (·) are the strategy fitting function and the student strategy fitting function, and are the teacher strategy representation vector and the student strategy representation vector respectively, θ SD Network parameters for self-distilled hierarchical reinforcement learning networks.

4. The method according to claim 1, characterized in that The learning process of the self-distillation hierarchical reinforcement learning network includes an experience storage stage and a training stage.

5. The method according to claim 4, characterized in that During the experience storage phase, an experience cell data set is collected for each time node; the experience cell data set includes: an initial state vector, a target strategy representation vector, a reward vector, a dynamic neighbor node, and a transportation state of any service resource package at each time node.

6. The method according to claim 5, characterized in that During the training phase, the network parameters and encoder parameters of the self-distillation hierarchical reinforcement learning network are optimized through the configured final loss function to obtain optimized network parameters and encoder parameters.

7. The method according to claim 6, characterized in that The final loss function is: in, is the final loss function, Q tea (·) and Q stu (·) are the strategy fitting function and the student strategy fitting function, t is the time node, θ Q Q tea (·) and Q stu The parameter of (·), θ SD are the network parameters of the self-distilled hierarchical reinforcement learning network, and are two normalization hyperparameters, is the initial state vector of the service resource package at time node t, is the expected function, L SD (·) is the self-distillation loss function, and They are target value and student target value respectively.

Images