Training method and device of task allocation system and readable storage medium

Abstract

The invention provides a task allocation system training method and device and a readable storage medium, and the method comprises the steps: carrying out the modeling processing of an ISCC task allocation scene, and obtaining an ISCC task allocation system; repeatedly executing the target process until the first reward value of the first reward function and the second reward value of the second reward function respectively reach the maximum value; the target process comprises the following steps: distributing a high-level task to a low-level agent according to a high-level agent algorithm; according to a low-layer agent algorithm, resource information contributed by the low-layer agent to the target task is determined, feedback information is sent to the high-layer agent, a second reward value is obtained, and a second network parameter in the low-layer agent algorithm is updated according to the second reward value; obtaining the first reward value, and updating a first network parameter in the high-level agent algorithm according to the first reward value; the first reward value is determined according to feedback information of the low-layer agent.

Description

Technical Field

[0001] The present application relates to the field of cloud computing technology, and in particular to a training method, device and readable storage medium for a task allocation system. Background Art

[0002] Among the related technologies, the 6th Generation Mobile Communication System (6G) will integrate communication, perception, and computing, and the computing force network (CFN) enabled by inter-sensing computing is a key technology. Perception, communication, and computing work together, and multiple levels of computing power are connected through the network to achieve flexible matching of real-time perception and computing resources. However, the increase in network nodes brings complex communication and perception problems, making it difficult to meet the needs of high-concurrency services. With the help of multi-level computing power and artificial intelligence technology, the integration of communication, perception, computing, and intelligent technologies can improve network perception accuracy and system efficiency to meet the needs of 6G emerging services.

[0003] It can be seen that Integrated Sensing, Communication and Computation (ISCC) and intelligent technology complement each other and jointly build a 6G network system with intelligent connection of all things. The terminal devices, network devices and application servers in the 6G system will have intelligent capabilities, and the terminals and networks can independently select artificial intelligence (AI) algorithm structures according to application scenarios and use cases. Faced with the diverse resources and business needs in the 6G network, achieving efficient resource scheduling has become the key to ensuring network performance and stability. However, there is no relevant solution on how to coordinate different nodes in the ISCC system to stably allocate resources. Summary of the invention

[0004] The purpose of this application is to provide a training method, device and readable storage medium for a task allocation system to solve the problem of how to coordinate different nodes in an ISCC system to stably allocate resources.

[0005] To achieve the above object, an embodiment of the present application provides a training method for a task allocation system, comprising:

[0006] The task allocation scenario of the first ISCC of Tongasuansuan is modeled and processed to obtain an ISCC task allocation system, wherein the task allocation system includes a high-level intelligent agent and a low-level intelligent agent, wherein the high-level intelligent agent includes an ISCC core cloud network, and the low-level intelligent agent includes an ISCC edge network and an ISCC terminal;

[0007] Repeat the target process until the first reward value of the first reward function of the high-level agent and the second reward value of the second reward function of the low-level agent reach maximum values ​​respectively;

[0008] Wherein, the target process includes:

[0009] Allocating high-level tasks to the low-level agents according to the configured high-level agent algorithm;

[0010] According to the configured low-level agent algorithm, determine the resource information contributed by the low-level agent to the target task, send feedback information to the high-level agent, and obtain the second reward value, and update the second network parameter in the low-level agent algorithm according to the second reward value; wherein the second reward value is related to the execution result of the target task, the target task is executed based on the resources contributed by the low-level agent, and the target task includes the high-level task; the feedback information is obtained according to the performance change information of the low-level agent, the task completion quality of the low-level agent for the high-level task, and the smoothing algorithm;

[0011] The first reward value is obtained, and a first network parameter in the high-level agent algorithm is updated according to the first reward value; wherein the first reward value is determined according to feedback information of the low-level agent.

[0012] Optionally, the allocating high-level tasks to the low-level agents according to the configured high-level agent algorithm comprises:

[0013] Allocating high-level tasks to the low-level agents according to the state information of the high-level agents and the configured high-level agent algorithm;

[0014] The status information of the high-level agent includes: task-related information of the high-level task, resource status information of the low-level agent, and physical characteristic parameters, wherein the physical characteristic parameters are used to indicate the physical characteristics between the low-level agent and the high-level agent.

[0015] Optionally, determining the resource information contributed by the low-level agent to the target task according to the configured low-level agent algorithm includes:

[0016] The resource information contributed by the low-level agent to the target task is determined based on the resource status information of the low-level agent, the allocation information of the high-level task, the resource requirement information of the high-level task, the time requirement information of the high-level task and the low-level agent algorithm.

[0017] Optionally, the method of the embodiment of the present application further includes:

[0018] Determining the task completion quality of the high-level task according to the completion time of the high-level task and the quality-source ratio of the high-level task;

[0019] The quality-to-source ratio is the ratio of the quality of the completed high-level task to the amount of resources consumed to complete the high-level task.

[0020] Optionally, the target task also includes a local task of the low-level agent.

[0021] Optionally, the method of the embodiment of the present application, obtaining the second reward value, includes:

[0022] The second reward value is obtained according to the quality of the high-level task completed by the low-level agent, a first parameter and a second parameter, wherein the first parameter is used to indicate whether the high-level task is completed, and the second parameter is used to indicate whether the local task is completed.

[0023] Optionally, the method of the embodiment of the present application further includes:

[0024] The third network parameter in the low-level agent algorithm is updated according to the policy gradient algorithm, wherein the policy gradient algorithm includes a regularization parameter, wherein the regularization parameter is obtained according to a regularization coefficient and a cooperative behavior parameter, and wherein the cooperative behavior parameter is obtained according to resource information contributed by the low-level agent to the target task.

[0025] The present application also provides a training device for a task allocation system, including:

[0026] The first processing module is used to model the task allocation scenario of the first ISCC of the inter-sensory computing to obtain an ISCC task allocation system, wherein the task allocation system includes a high-level intelligent agent and a low-level intelligent agent, wherein the high-level intelligent agent includes an ISCC core cloud network, and the low-level intelligent agent includes an ISCC edge network and an ISCC terminal;

[0027] A second processing module is used to repeatedly execute the target process until the first reward value of the first reward function of the high-level agent and the second reward value of the second reward function of the low-level agent reach maximum values ​​respectively;

[0028] Wherein, the target process includes:

[0029] Allocating high-level tasks to the low-level agents according to the configured high-level agent algorithm;

[0030] According to the configured low-level agent algorithm, determine the resource information contributed by the low-level agent to the target task, send feedback information to the high-level agent, and obtain the second reward value, and update the second network parameter in the low-level agent algorithm according to the second reward value; wherein the second reward value is related to the execution result of the target task, the target task is executed based on the resources contributed by the low-level agent, and the target task includes the high-level task; the feedback information is obtained according to the performance change information of the low-level agent, the task completion quality of the low-level agent for the high-level task, and the smoothing algorithm;

[0031] The first reward value is obtained, and a first network parameter in the high-level agent algorithm is updated according to the first reward value; wherein the first reward value is determined according to feedback information of the low-level agent.

[0032] Optionally, the second processing module is used to:

[0033] Allocating high-level tasks to the low-level agents according to the state information of the high-level agents and the configured high-level agent algorithm;

[0034] The status information of the high-level agent includes: task-related information of the high-level task, resource status information of the low-level agent, and physical characteristic parameters, wherein the physical characteristic parameters are used to indicate the physical characteristics between the low-level agent and the high-level agent.

[0035] Optionally, the second processing module is used to:

[0036] The resource information contributed by the low-level agent to the target task is determined based on the resource status information of the low-level agent, the allocation information of the high-level task, the resource requirement information of the high-level task, the time requirement information of the high-level task and the low-level agent algorithm.

[0037] Optionally, the second processing module is further used for:

[0038] Determining the task completion quality of the high-level task according to the completion time of the high-level task and the quality-source ratio of the high-level task;

[0039] The quality-to-source ratio is the ratio of the quality of the completed high-level task to the amount of resources consumed to complete the high-level task.

[0040] Optionally, the target task also includes a local task of the low-level agent.

[0041] Optionally, the second processing module is used to:

[0042] The second reward value is obtained according to the quality of the high-level task completed by the low-level agent, a first parameter and a second parameter, wherein the first parameter is used to indicate whether the high-level task is completed, and the second parameter is used to indicate whether the local task is completed.

[0043] Optionally, the second processing module is further used for:

[0044] The third network parameter in the low-level agent algorithm is updated according to the policy gradient algorithm, wherein the policy gradient algorithm includes a regularization parameter, wherein the regularization parameter is obtained according to a regularization coefficient and a cooperative behavior parameter, and wherein the cooperative behavior parameter is obtained according to resource information contributed by the low-level agent to the target task.

[0045] An embodiment of the present application also provides a training device for a task allocation system, including: a transceiver, a processor, a memory, and a program or instruction stored in the memory and executable on the processor; when the processor executes the program or instruction, the steps in the training method for the task allocation system as described above are implemented.

[0046] An embodiment of the present application also provides a readable storage medium having a program or instruction stored thereon, which, when executed by a processor, implements the steps of the training method for the task allocation system as described above.

[0047] An embodiment of the present application also provides a computer program product, including computer instructions, which, when executed by a processor, implement the steps of the training method for the task allocation system as described above.

[0048] The beneficial effects of the above technical solution of the present application are as follows:

[0049] In an embodiment of the present application, the first reward value corresponding to the high-level intelligent agent is obtained based on the performance change information of the low-level intelligent agent, the task completion quality of the low-level intelligent agent for the high-level task, and the smoothing algorithm. Based on this first reward value, the stability of the high-level intelligent agent's learning can be effectively guaranteed, and the high-level intelligent agent's learning can be prevented from being affected by the dynamic changes of the low-level intelligent agent, thereby effectively ensuring the stability between the high-level intelligent agent and the low-level intelligent agent, and enabling the ISCC task allocation system to stably allocate tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 A schematic diagram of the computing network for applying the hierarchical multi-agent scheduling algorithm based on hierarchical deep reinforcement learning to the ISCC resource scheduling and allocation system;

[0051] Figure 2 A schematic diagram corresponding to the design concept of the task allocation system of an embodiment of the present application;

[0052] Figure 3 A flowchart of a training method for a task allocation system according to an embodiment of the present application;

[0053] Figure 4 A schematic diagram of a two-level hierarchical structure of Hierarchical Multi-Agent Scheduling (HMAS) according to an embodiment of the present application;

[0054] Figure 5 A schematic diagram of a module of a training device of a task allocation system according to an embodiment of the present application;

[0055] Figure 6 A schematic diagram of the structure of a training device of a task allocation system according to an embodiment of the present application. DETAILED DESCRIPTION

[0056] In order to make the technical problems, technical solutions and advantages to be solved by the present application clearer, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0057] It should be understood that the references to "one embodiment" or "an embodiment" throughout the specification mean that the specific features, structures, or characteristics associated with the embodiment are included in at least one embodiment of the present application. Therefore, the references to "in one embodiment" or "in an embodiment" appearing throughout the specification do not necessarily refer to the same embodiment. In addition, these specific features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0058] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way are interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of one type, and the number of objects is not limited, for example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, for example, the protection scope of "A and / or B" covers at least three schemes, namely, Scheme 1: including A and excluding B; Scheme 2: including B and excluding A; Scheme 3: including both A and B. The character " / " generally indicates that the objects associated before and after are in an "or" relationship.

[0059] In the various embodiments of the present application, it should be understood that the size of the serial numbers of the following processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0060] Additionally, the terms "system" and "network" are often used interchangeably herein.

[0061] In the embodiments provided in the present application, it should be understood that "B corresponding to A" means that B is associated with A, and B can be determined according to A. However, it should also be understood that determining B according to A does not mean determining B only according to A, and B can also be determined according to A and / or other information.

[0062] In order to meet the vision of 6G green development and realize the efficient coordination and utilization of 6G network resources, this application proposes a Hierarchical Multi-Agent Scheduling (HMAS) framework based on deep reinforcement learning. The framework divides resource scheduling and collaborative tasks into two levels, namely high-level agent (HLA) and low-level agent (LLA). The HMAS framework uses the dynamic feedback adjustment mechanism (DFAM) to solve the non-stationary problem between HLA and LLA, improve the efficiency and stability of overall learning, and accelerate the convergence speed of the hierarchical reinforcement learning system. The cooperative reward mechanism (CRM) is adopted to improve the coordination effect of LLA. The algorithm has low complexity and can achieve efficient use of 6G resources while ensuring that tasks are completed on time.

[0063] Figure 1 In order to use the HMAS algorithm based on hierarchical deep reinforcement learning for the computing force networking (CFN) of the ISCC resource scheduling and allocation system. The ISCC architecture network is layered, where the ISCC core cloud network is HLA, which is responsible for task allocation; the ISCC edge network (including mobile edge computing servers, base stations, etc.) and ISCC terminals all belong to LLA, which is responsible for task execution. The entire system is trained to maximize the reward function. In order to solve the non-stationary problem between HLA and LLA, that is, HLA learning is affected by the dynamic changes of the LLA learning process, the HMAS method uses DFAM to process the reward feedback problem of HLA, thereby ensuring the stability of HLA learning. CRM is used to allow LLA to have a mutually beneficial idea while ensuring its own rewards, so that the LLA coordination effect can be better. The schematic diagram corresponding to the above design idea is shown below. Figure 2 As shown, the high-level agent assigns tasks to the low-level agents, and the low-level agents send feedback information to the high-level agents based on DFAM.

[0064] Based on the above design ideas, such as Figure 3As shown, the present application provides a training method for a task allocation system, including:

[0065] Step 301: Modeling the task allocation scenario of the first ISCC of the synaesthesia algorithm to obtain an ISCC task allocation system, wherein the task allocation system includes a high-level intelligent agent and a low-level intelligent agent, wherein the high-level intelligent agent includes an ISCC core cloud network, and the low-level intelligent agent includes an ISCC edge network and an ISCC terminal.

[0066] like Figure 2 As shown, the ISCC core cloud network includes a core cloud server, and the ISCC edge network includes a mobile edge computing server, a base station, etc. The above-mentioned high-level intelligent agent is responsible for task allocation, and the above-mentioned low-level intelligent agent is responsible for task execution.

[0067] Step 302: Repeat the target process until the first reward value of the first reward function of the high-level agent and the second reward value of the second reward function of the low-level agent reach maximum values ​​respectively.

[0068] Wherein, the target process includes:

[0069] Allocating high-level tasks to the low-level agents according to the configured high-level agent algorithm;

[0070] According to the configured low-level agent algorithm, determine the resource information contributed by the low-level agent to the target task, send feedback information to the high-level agent, and obtain the second reward value, and update the second network parameter in the low-level agent algorithm according to the second reward value; wherein the second reward value is related to the execution result of the target task, the target task is executed based on the resources contributed by the low-level agent, and the target task includes the high-level task; the feedback information is obtained according to the performance change information of the low-level agent, the task completion quality of the low-level agent for the high-level task, and the smoothing algorithm;

[0071] The first reward value is obtained, and a first network parameter in the high-level agent algorithm is updated according to the first reward value; wherein the first reward value is determined according to feedback information of the low-level agent.

[0072] Optionally, the smoothing algorithm is an exponential smoothing algorithm. The mechanism by which the lower-level agent sends the feedback information to the higher-level agent is the DFAM mechanism.

[0073] Optionally, the performance change information of the low-level agent is obtained based on a performance monitoring function, and the performance monitoring function can be determined based on the variance of the low-level agent.

[0074] Optionally, the execution result of the target task includes the quality of the high-level task completed by the low-level agent, a first parameter and a second parameter, wherein the first parameter is used to indicate whether the high-level task is completed, and the second parameter is used to indicate whether the local task is completed.

[0075] Optionally, updating the first network parameter in the high-level agent algorithm according to the first reward value includes: updating the parameters of the critic network in the high-level agent algorithm according to the first reward value. For example, a temporal-difference (TD) algorithm is used to update the parameters of the critic network in the high-level agent algorithm based on the first reward value.

[0076] Optionally, the method of the embodiment of the present application further includes: updating the parameters of the actor network in the high-level agent algorithm based on the policy gradient algorithm and the adaptive learning rate. The adaptive learning rate is obtained according to the performance change information of the low-level agent. The performance change information of the low-level agent can be obtained based on the performance monitoring function.

[0077] Optionally, updating the second network parameters in the low-level agent algorithm according to the second reward value includes: updating the parameters of the critic network in the low-level agent algorithm according to the second reward value. For example, updating the parameters of the critic network in the low-level agent algorithm using the second reward value based on the TD algorithm.

[0078] Optionally, the method of the embodiment of the present application further includes:

[0079] The parameters of the actor network in the low-level agent algorithm are updated according to the policy gradient method, namely the third network parameters described below.

[0080] In an embodiment of the present application, the first reward value corresponding to the high-level intelligent agent is obtained based on the performance change information of the low-level intelligent agent, the task completion quality of the low-level intelligent agent for the high-level task, and the smoothing algorithm. Based on this first reward value, the stability of the high-level intelligent agent's learning can be effectively guaranteed, and the high-level intelligent agent's learning can be prevented from being affected by the dynamic changes of the low-level intelligent agent, thereby effectively ensuring the stability between the high-level intelligent agent and the low-level intelligent agent, and enabling the ISCC task allocation system to stably allocate tasks.

[0081] The above DFAM mechanism is described in detail below in conjunction with the embodiments.

[0082] like Figure 4As shown in the figure, HMAS is a two-level hierarchical structure, and the ultimate goal of the multi-agent system is to maximize the global reward. Both the high-level agent algorithm and the low-level agent algorithm adopt the Actor-Critic (AC) algorithm, in which the actor network (Actor network, i.e., Actor H ) performs task allocation according to the overall state of the environment, and the actor network of the low-level agent algorithm Take actions according to the target task. The critic network of the high-level agent algorithm (Critic network, Critic H )DFAM is used to improve the stability and efficiency of strategy learning. The critic network of the low-level agent algorithm is in, Represents the high-level agent H N The action at time t, Represents the low-level agent L N The action at time t, Represents the low-level agent L N The reward at time t, s t represents the state space at time t, specifically the LLA state space and / or HLA state space at time t, θ H represents the actor network parameters of the high-level agent; θ L Represents the actor network parameters of the low-level agent.

[0083] The Markov decision processes of HLA and LLA of HMAS are:

[0084] MDP H = H ,A H ,P H ,r H ,γ H >

[0085] MDP L = L ,A L ,P L ,r L ,γ L >

[0086] MDP H is the Markov decision process of HLA, S H is the HLA state space, A H is the HLA action space, P H is the HLA state transition function, r H is the HLA reward function (i.e., the first reward function mentioned above), γ H is the high-level discount factor. MDP L ​​is the Markov decision process of LLA, S L is the LLA state space, A L is the LLA action space, P L is the LLA state transition function, r L is the LLA reward function (i.e., the second reward function mentioned above), γ L is the low-level discount factor.

[0087] 1) High-level agents;

[0088] The role of HLA is to assign tasks to different LLAs so that they can work together as a team. HLA uses the actor-critic algorithm to learn the task assignment strategy, predict the performance of LLA under different tasks, and decide the task assignment based on these predictions. LLA also uses the actor-critic algorithm. The actor network is responsible for selecting actions, and the critic network is responsible for evaluating the value of actions and providing feedback signals. This algorithm allows the agent to learn how to better perform tasks based on the assignment of HLA.

[0089] Optionally, the method of the embodiment of the present application further includes:

[0090] Determining the task completion quality of the high-level task according to the completion time of the high-level task and the quality-source ratio of the high-level task;

[0091] The quality-to-source ratio is the ratio of the quality of the completed high-level task to the amount of resources consumed to complete the high-level task.

[0092] In the embodiment of the present application, feedback information is obtained based on the quality-source ratio, so that the high-level intelligent agent can obtain the ratio information of the quality of the completed high-level task and the amount of resources consumed to complete the high-level task.

[0093] The first reward function mentioned above, also called the global reward function, is defined as follows:

[0094]

[0095] Assume that the total number of tasks is I, that is, the total number of high-level tasks is I, F(i) represents the task completion quality of the low-level agent to complete the high-level task i, and Time i Indicates the completion time of the assigned high-level task i, Time i Depending on the maximum completion time of the assigned LLA group, ∈ is a small positive number used to avoid the case where the denominator is zero. The “quality-to-source ratio” is defined as the ratio between the quality of the completed task and the amount of resources consumed. represents the quality of task i completed by LLA (measured by task satisfaction), C i Indicates the amount of resources consumed to complete high-level task i (with respect to computing resources Considering that different tasks may have different importance and difficulty, a weight τ is assigned to each task i , β 1 Represents the weight factor, 0≤β 1 ≤1.

[0096] The update of LLA will have an impact on HLA. However, during the training process, LLA shows a dynamic change trend from inferior to superior. In this non-stationary environment, it is difficult for HLA to learn the optimal strategy. The exploration of HLA is crucial to the overall learning process. Its non-stationarity leads to uncertainty in the transfer process, affecting strategy optimization. HLA sets goals and expects LLA to achieve maximum rewards, but the dynamic changes of LLA during training may cause the reward feedback received by HLA to not match its true value, which is not conducive to HLA learning. Therefore, in order to solve the problem of mismatch between HLA and LLA during training, this application designs a mechanism to improve the robustness of the system, called DFAM. This mechanism needs to take into account the dynamic changes of LLA, while ensuring that HLA can obtain rewards that truly reflect the value of its behavior. Specifically, HLA's F(i) needs to go through the following three steps:

[0097] First, define a smoothing function S(F(i)) to smooth the task completion quality of the low-level agent, using exponential smoothing:

[0098] S(F(i))=αF(i)+(1-α)S(F prev (i));

[0099] Where α is the smoothing factor, 0≤α≤1, F prev (i) is F(i) of the previous time step.

[0100] Secondly, we model the policy uncertainty and define the performance monitoring function U(θ L ) is used to estimate the uncertainty of the LLA group. The LLA group includes at least one LLA. The function is determined based on the variance of the lower-level agent, and the following formula can be obtained:

[0101] F(i)=F(i)-β 2 U(θ L );

[0102] where β 2 is the weight factor, 0≤β 2 ≤1, used to control the impact of uncertainty on high-level strategy rewards, θ L are the parameters of the actor network of the low-level agent.

[0103] Finally, combining the above two methods, namely:

[0104] F(i)=S[F(i)-β 2 U(θ L )];

[0105] The final F(i) obtained here is the feedback information sent by the low-level agent to the high-level agent. Through this feedback information, the reward feedback received by HLA matches its true value, which is beneficial to the learning and training of HLA.

[0106] Optionally, the allocating high-level tasks to the low-level agents according to the configured high-level agent algorithm comprises:

[0107] Allocating high-level tasks to the low-level agents according to the state information of the high-level agents and the configured high-level agent algorithm;

[0108] The status information of the high-level agent includes: task-related information of the high-level task, resource status information of the low-level agent, and physical characteristic parameters, wherein the physical characteristic parameters are used to indicate the physical characteristics between the low-level agent and the high-level agent.

[0109] Optionally, the task-related information of the high-level task includes, but is not limited to, resource requirement information of the high-level task and time requirement information of the high-level task.

[0110] Optionally, the object characteristic parameters include but are not limited to distance information between the low-level intelligent agent and the high-level intelligent agent, and channel quality information.

[0111] Optionally, the resource requirement information of the high-level task includes the requirement of the high-level task for computing resources, network resources and storage resources.

[0112] Optionally, the time requirement information of the high-level task includes a maximum tolerance time of the high-level task, for example, a maximum delay for completing the high-level task.

[0113] Optionally, the resource status information of the low-level agent includes the remaining computing resources, network resources, storage resources and service resources of the low-level agent at the current moment.

[0114] In the embodiment of the present application, for different types of high-level tasks i, the state of the design environment input is in, The items in it represent the requirements of high-level task i for computing resources, network resources, storage resources and the maximum tolerable time of the task. Represents the nth agent (LLA n ), Each term in represents LLA nThe remaining computing resources, network resources, storage resources, service resources and physical distance from HLA at the current moment. The actor network performs actions, that is, outputs task allocation in Expressed as LLA n The proportion of tasks assigned, and

[0115] In the embodiment of the present application, according to different types of high-level tasks, high-level tasks are allocated based on the resource status information of the low-level agent and the distance information between the low-level agent and the high-level agent, so that tasks can be allocated according to the ability differences of the low-level agent, so that the low-level agent can perform the type of tasks that it is good at performing, thereby improving the efficiency of task execution. When allocating new tasks, based on the resource status information of the low-level agent, the resources occupied by the unfinished old tasks can be taken into account, and the current remaining resources of the low-level agent can be used to allocate tasks, effectively avoiding the problem that when the low-level agent has not completed the old task and receives a new task, there are not enough resources to perform the new task, thereby improving the efficiency of task execution.

[0116] In the embodiment of the present application, HLA uses an actor-critic architecture and adopts Deep Deterministic Policy Gradient (DDPG), which includes an actor network and a critic network. The actor network contains an online policy and a target policy. The network state-behavior function of the randomly initialized critic network

[0117] HLA will state sh i As input, the actor network of HLA is adopted To calculate the probability distribution of actions, and then perform action sampling:

[0118] HLA performs action a H , the environment will give a new state sh′ i and reward r H (sh i ,a H ).

[0119] like Figure 4 As shown, (sh i ,a H ,r H (sh i ,a H ),sh′ i ) is stored in the experience replay pool (referred to as the experience pool). Some samples are randomly extracted from the experience replay pool for training. Assume A batch of randomly sampled data is used to train the target network using temporal difference (TD). m It is expressed as:

[0120]

[0121] According to the immediate reward H , use TD algorithm to update the HLA critic network parameters ω H :

[0122]

[0123] ζ is the discount factor, 0<ζ<1. is a differential operation, which represents the difference between the return predicted by the HLA critic network and the actual return. The HLA critic network adjusts its parameters ω according to this difference. H , to more accurately predict future returns.

[0124] Finally, policy gradient is used to update the actor network parameters θ of HLA H .

[0125]

[0126] is a differential operation, Q H (sh i ,a H ) is the action value function of HLA, V H (sh i ) is the state value function of HLA.

[0127] Where η is the adaptive learning rate, 0<η<1, which can be dynamically adjusted according to the performance changes of LLA. When the performance of LLA fluctuates greatly, the learning rate of HLA can be reduced to reduce the policy update error caused by inaccurate reward feedback. As the performance of LLA gradually stabilizes, the learning rate of HLA can be gradually increased to speed up the learning speed. Use a performance monitoring function U(θ L ) to measure the performance of LLA and adjust the learning rate accordingly:

[0128]

[0129] The critic network calculates the TD loss and updates the actor network parameters. First, the critic network uses the Deep Q-learning Network (DQN), an experience replay-based algorithm, to optimize parameters. Experience replay is a method of learning strategies and value functions by replaying previously collected experience data. It can reduce the correlation of data, avoid overfitting, and make more effective use of data. The critic network calculates the optimal value of the state through Q-learning.

[0130] HLA's critic network uses DQN, an experience replay-based algorithm, to optimize parameters. This process is implemented through the TD algorithm:

[0131] δ H =r H +ζQ H (s′,a H ′;ω H )-Q H (s,a H ;ω H );

[0132]

[0133] Among them, r H is the immediate reward, ζ is the discount factor, and η is the adaptive learning rate.

[0134] TD Error represents the difference between the return predicted by the critic network and the actual return.

[0135] The critic network adjusts its parameters ω based on this difference H , to more accurately predict future returns.

[0136] The optimal value of the state is calculated, and the actor network uses this optimal value to iteratively update the parameters of the policy function, and then selects an action and obtains feedback and the next state. The policy function parameters are updated as follows:

[0137]

[0138] Optionally, determining the resource information contributed by the low-level agent to the target task according to the configured low-level agent algorithm includes:

[0139] The resource information contributed by the low-level agent to the target task is determined based on the resource status information of the low-level agent, the allocation information of the high-level task, the resource requirement information of the high-level task, the time requirement information of the high-level task and the low-level agent algorithm.

[0140] Optionally, the resource information contributed by the low-level agent to the target task includes proportion information of various types of resources contributed by the low-level agent to the target task.

[0141] In the embodiment of the present application, the input of the low-level agent algorithm consists of three parts: resource status information of the low-level agent, allocation information of the high-level task, and resource demand information of the high-level task. Among them, each item represents the computing resources, network resources, storage resources, and service resources that the low-level intelligent agent n has for the high-level task i.

[0142] Optionally, for the low-level agent n, the allocation information of the above high-level tasks is

[0143] Optionally, the resource requirement information of the high-level task is express.

[0144] In summary, for high-level tasks i, LLA n The input of the low-level agent algorithm is expressed as LLA m Output Respectively represent LLA n The proportion of various resources willing to contribute to high-level task i, when hour,

[0145] Optionally, the target task also includes a local task of the low-level agent.

[0146] In the embodiment of the present application, in addition to completing the high-level tasks assigned by the HLA, the LLA also needs to handle some sudden tasks, which are collectively referred to as regular tasks, i.e. local tasks, and these tasks also need to consume certain resources of the LLA. The LLA needs to execute the tasks assigned by the HLA and the local regular tasks at the same time.

[0147] In the embodiments of the present application, in the low-level intelligent agents of hierarchical reinforcement learning, the needs of local routine tasks and high-level tasks are taken into consideration at the same time, and the computing resources of the low-level intelligent agents are intelligently allocated based on the contribution degree as the execution action, so that the low-level intelligent agents can effectively guarantee the execution of local tasks while executing high-level tasks, and add the quality-source ratio as an evaluation indicator for feedback, so as to achieve lower energy efficiency while also efficiently processing tasks in parallel.

[0148] Optionally, the method of the embodiment of the present application, obtaining the second reward value, includes:

[0149] The second reward value is obtained according to the quality of the high-level task completed by the low-level agent, a first parameter and a second parameter, wherein the first parameter is used to indicate whether the high-level task is completed, and the second parameter is used to indicate whether the local task is completed.

[0150] In the embodiment of the present application, the quality of the high-level tasks completed by the low-level agents is measured by user satisfaction.

[0151] Optionally, the second reward function is:

[0152]

[0153] in, Indicates LLA n The quality of actually completing high-level task i (measured by user satisfaction), p i Indicates whether the assigned high-level task i is completed, q j Indicates whether the local task j is completed (0: No, 1: Yes), ξ is the adjustment factor, 0≤ξ≤1. i This is the first parameter mentioned above, p i 0 means that high-level task i is not completed, p i 1 means that high-level task i is completed, q j This is the second parameter mentioned above, q j 0 means that the local task j is not completed, q j A value of 1 indicates that local task j is completed.

[0154] Optionally, the method of the embodiment of the present application further includes:

[0155] The third network parameter in the low-level agent algorithm is updated according to the policy gradient algorithm, wherein the policy gradient algorithm includes a regularization parameter, wherein the regularization parameter is obtained according to a regularization coefficient and a cooperative behavior parameter, and wherein the cooperative behavior parameter is obtained according to resource information contributed by the low-level agent to the target task.

[0156] The third network parameter can be understood as the parameter of the actor network in the low-level agent algorithm.

[0157] In the embodiment of the present application, each LLA is also constructed using an actor-critic model. LLA updates its actor network according to the policy gradient method to maximize the expected return. Considering the "tit-for-tat" strategy, the policy gradient needs to include additional terms to encourage cooperative behavior. The critic network is updated by the TD method to accurately estimate the state value or action value.

[0158] set up (Expected reward) is the action taken in state sl The total benefit expected by the low-level agents is is the policy function (determines the action to take in state sl probability), For LLA n The parameters of the actor network, For LLA n Parameters of the critic network.

[0159]

[0160] Combined with the tit-for-tat strategy, a regularization term is added to the gradient update to encourage cooperation.

[0161]

[0162] represents cooperative behavior, that is, the above cooperative behavior parameters, and the judgment condition is and λ is the regularization coefficient.

[0163] in, is the regularization parameter mentioned above.

[0164] The critic network of LLA uses DQN, an algorithm based on experience replay, to optimize parameters. This process is implemented through the TD algorithm:

[0165]

[0166] is the immediate reward, γ is the discount factor, is the learning rate. TD error Represents the difference between the return predicted by LLA's critic network and the actual return. LLA's critic network will adjust its parameters based on this difference To more accurately predict future earnings.

[0167] In the embodiment of the present application, a collaborative reward mechanism (CRM) is used to improve the LLA coordination effect.

[0168] In the embodiment of the present application, firstly, a method of setting reserved resources is adopted for various task sources; secondly, a variety of different intelligent agents are set for various task types, with different types of task processing capabilities; finally, the completion of the previous tasks is considered to deal with the situation where the tasks overlap in the time dimension. Through the above methods, the execution efficiency of the tasks is effectively improved.

[0169] like Figure 5 As shown, the embodiment of the present application also provides a training device for a task allocation system, including:

[0170] The first processing module 501 is used to model the task allocation scenario of the first ISCC of the synaesthesia algorithm to obtain an ISCC task allocation system, wherein the task allocation system includes a high-level intelligent agent and a low-level intelligent agent, wherein the high-level intelligent agent includes an ISCC core cloud network, and the low-level intelligent agent includes an ISCC edge network and an ISCC terminal;

[0171] A second processing module 502 is used to repeatedly execute the target process until the first reward value of the first reward function of the high-level agent and the second reward value of the second reward function of the low-level agent reach maximum values ​​respectively;

[0172] Wherein, the target process includes:

[0173] Allocating high-level tasks to the low-level agents according to the configured high-level agent algorithm;

[0174] According to the configured low-level agent algorithm, determine the resource information contributed by the low-level agent to the target task, send feedback information to the high-level agent, and obtain the second reward value, and update the second network parameter in the low-level agent algorithm according to the second reward value; wherein the second reward value is related to the execution result of the target task, the target task is executed based on the resources contributed by the low-level agent, and the target task includes the high-level task; the feedback information is obtained according to the performance change information of the low-level agent, the task completion quality of the low-level agent for the high-level task, and the smoothing algorithm;

[0175] The first reward value is obtained, and a first network parameter in the high-level agent algorithm is updated according to the first reward value; wherein the first reward value is determined according to feedback information of the low-level agent.

[0176] Optionally, the second processing module is used to:

[0177] Allocating high-level tasks to the low-level agents according to the state information of the high-level agents and the configured high-level agent algorithm;

[0178] The status information of the high-level agent includes: task-related information of the high-level task, resource status information of the low-level agent, and physical characteristic parameters, wherein the physical characteristic parameters are used to indicate the physical characteristics between the low-level agent and the high-level agent.

[0179] Optionally, the second processing module is used to:

[0180] The resource information contributed by the low-level agent to the target task is determined based on the resource status information of the low-level agent, the allocation information of the high-level task, the resource requirement information of the high-level task, the time requirement information of the high-level task and the low-level agent algorithm.

[0181] Optionally, the second processing module is further used for:

[0182] Determining the task completion quality of the high-level task according to the completion time of the high-level task and the quality-source ratio of the high-level task;

[0183] The quality-to-source ratio is the ratio of the quality of the completed high-level task to the amount of resources consumed to complete the high-level task.

[0184] Optionally, the target task also includes a local task of the low-level agent.

[0185] Optionally, the second processing module is used to:

[0186] The second reward value is obtained according to the quality of the high-level task completed by the low-level agent, a first parameter and a second parameter, wherein the first parameter is used to indicate whether the high-level task is completed, and the second parameter is used to indicate whether the local task is completed.

[0187] Optionally, the second processing module is further used for:

[0188] The third network parameter in the low-level agent algorithm is updated according to the policy gradient algorithm, wherein the policy gradient algorithm includes a regularization parameter, wherein the regularization parameter is obtained according to a regularization coefficient and a cooperative behavior parameter, and wherein the cooperative behavior parameter is obtained according to resource information contributed by the low-level agent to the target task.

[0189] In an embodiment of the present application, the first reward value corresponding to the high-level intelligent agent is obtained based on the performance change information of the low-level intelligent agent, the task completion quality of the low-level intelligent agent for the high-level task, and the smoothing algorithm. Based on this first reward value, the stability of the high-level intelligent agent's learning can be effectively guaranteed, and the high-level intelligent agent's learning can be prevented from being affected by the dynamic changes of the low-level intelligent agent, thereby effectively ensuring the stability between the high-level intelligent agent and the low-level intelligent agent, and enabling the ISCC task allocation system to stably allocate tasks.

[0190] It should be noted that the device is a device corresponding to the above-mentioned method embodiment. All implementation methods of the above-mentioned method embodiment can be applied to the device embodiment and can achieve the same technical effect, which will not be repeated here.

[0191] The present application embodiment provides a training device for a task allocation system, such as Figure 6As shown, it includes a transceiver 610, a processor 600, a memory 620, and a program or instruction stored in the memory 620 and executable on the processor 600; when the processor 600 executes the program or instruction, the steps of the training method of the task allocation system are implemented. The training device of the task allocation system can be an electronic device, such as a terminal or a computer device or a user device.

[0192] The transceiver 610 is used to receive and send data under the control of the processor 600 .

[0193] Among them, Figure 6 In the embodiment, the bus architecture may include any number of interconnected buses and bridges, specifically one or more processors represented by processor 600 and various circuits of memory represented by memory 620 are linked together. The bus architecture may also link together various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein. The bus interface provides an interface. The transceiver 610 may be a plurality of components, namely, a transmitter and a receiver, providing a unit for communicating with various other devices on a transmission medium. For different user devices, the user interface 630 may also be an interface capable of externally and internally connecting required devices, and the connected devices include but are not limited to a keypad, a display, a speaker, a microphone, a joystick, and the like.

[0194] The processor 600 is responsible for managing the bus architecture and general processing, and the memory 620 can store data used by the processor 600 when performing operations.

[0195] An embodiment of the present application also provides a readable storage medium on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps in the training method of the task allocation system as described above are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0196] An embodiment of the present application also provides a computer program product, including computer instructions, which, when executed by a processor, implement the various processes of the embodiment of the training method of the above-mentioned task allocation system and can achieve the same technical effect. To avoid repetition, they are not repeated here.

[0197] The processor is a processor in the training device of the task allocation system described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0198] It should be further explained that the terminals described in this specification include but are not limited to smart phones, tablet computers, etc., and many of the functional components described are called modules in order to more particularly emphasize the independence of their implementation methods.

[0199] In the embodiment of the present application, module can be implemented with software so that it can be executed by various types of processors. For example, an executable code module of an identification can include one or more physical or logical blocks of computer instructions, for example, it can be constructed as an object, process or function. Nevertheless, the executable code of the identified module does not need to be physically located together, but can include different instructions stored in different positions, and when these instructions are logically combined together, it constitutes a module and realizes the specified purpose of the module.

[0200] In fact, executable code module can be a single instruction or many instructions, and can even be distributed on a plurality of different code segments, distributed among different programs, and distributed across a plurality of memory devices. Similarly, operating data can be identified in the module, and can be implemented and organized in the data structure of any appropriate type according to any appropriate form. The operating data can be collected as a single data set, or can be distributed in different locations (including on different storage devices), and can only be present on a system or network as an electronic signal at least in part.

[0201] When a module can be implemented by software, considering the level of existing hardware technology, a person skilled in the art can build a corresponding hardware circuit to implement the corresponding function of the module that can be implemented by software without considering the cost. The hardware circuit includes a conventional very large scale integration (VLSI) circuit or gate array and existing semiconductors such as logic chips, transistors, or other discrete components. The module can also be implemented by a programmable hardware device, such as a field programmable gate array, a programmable array logic, a programmable logic device, etc.

[0202] The above exemplary embodiments are described with reference to the accompanying drawings, and many different forms and embodiments are feasible without departing from the spirit and teachings of the present application. Therefore, the present application should not be constructed as a limitation of the exemplary embodiments proposed herein. More specifically, these exemplary embodiments are provided so that the present application will be perfect and complete, and the scope of the present application will be conveyed to those who are familiar with the technology. In these figures, the component sizes and relative sizes may be exaggerated for clarity. The terms used here are only based on the purpose of describing specific exemplary embodiments and are not intended to be limiting. As used herein, unless the text clearly indicates otherwise, the singular forms "one", "an" and "the" are intended to include these multiple forms. It will be further understood that the terms "comprising" and / or "including" when used in this specification indicate the presence of the features, integers, steps, operations, components and / or components, but do not exclude the presence or increase of one or more other features, integers, steps, operations, components, components and / or their groups. Unless otherwise indicated, when stated, a range of values ​​includes the upper and lower limits of that range and any subranges therebetween.

[0203] The above is a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles described in the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A training method for a task allocation system, characterized in that: include: The task allocation scenario of the first ISCC of Tongasuansuan is modeled and processed to obtain an ISCC task allocation system, wherein the task allocation system includes a high-level intelligent agent and a low-level intelligent agent, wherein the high-level intelligent agent includes an ISCC core cloud network, and the low-level intelligent agent includes an ISCC edge network and an ISCC terminal; Repeat the target process until the first reward value of the first reward function of the high-level agent and the second reward value of the second reward function of the low-level agent reach maximum values ​​respectively; Wherein, the target process includes: Allocating high-level tasks to the low-level agents according to the configured high-level agent algorithm; According to the configured low-level agent algorithm, determine the resource information contributed by the low-level agent to the target task, send feedback information to the high-level agent, and obtain the second reward value, and update the second network parameter in the low-level agent algorithm according to the second reward value; wherein the second reward value is related to the execution result of the target task, the target task is executed based on the resources contributed by the low-level agent, and the target task includes the high-level task; the feedback information is obtained according to the performance change information of the low-level agent, the task completion quality of the low-level agent for the high-level task, and the smoothing algorithm; The first reward value is obtained, and a first network parameter in the high-level agent algorithm is updated according to the first reward value; wherein the first reward value is determined according to feedback information of the low-level agent.

2. The method according to claim 1, characterized in that The high-level agent algorithm configured to allocate high-level tasks to the low-level agent includes: Allocating high-level tasks to the low-level agents according to the state information of the high-level agents and the configured high-level agent algorithm; The status information of the high-level agent includes: task-related information of the high-level task, resource status information of the low-level agent, and physical characteristic parameters, wherein the physical characteristic parameters are used to indicate the physical characteristics between the low-level agent and the high-level agent.

3. The method according to claim 1, characterized in that Determining resource information contributed by the low-level agent to the target task according to the configured low-level agent algorithm, including: The resource information contributed by the low-level agent to the target task is determined based on the resource status information of the low-level agent, the allocation information of the high-level task, the resource requirement information of the high-level task, the time requirement information of the high-level task and the low-level agent algorithm.

4. The method according to claim 1, characterized in that: Also includes: Determining the task completion quality of the high-level task according to the completion time of the high-level task and the quality-source ratio of the high-level task; The quality-to-source ratio is the ratio of the quality of the completed high-level task to the amount of resources consumed to complete the high-level task.

5. The method according to claim 1, characterized in that The target task also includes the local task of the low-level agent.

6. The method according to claim 5, characterized in that Obtaining the second reward value includes: The second reward value is obtained according to the quality of the high-level task completed by the low-level agent, a first parameter and a second parameter, wherein the first parameter is used to indicate whether the high-level task is completed, and the second parameter is used to indicate whether the local task is completed.

7. The method according to claim 1, characterized in that Also includes: The third network parameter in the low-level agent algorithm is updated according to the policy gradient algorithm, wherein the policy gradient algorithm includes a regularization parameter, wherein the regularization parameter is obtained according to a regularization coefficient and a cooperative behavior parameter, and wherein the cooperative behavior parameter is obtained according to resource information contributed by the low-level agent to the target task.

8. A training device for a task allocation system, characterized in that: include: The first processing module is used to model the task allocation scenario of the first ISCC of the inter-sensory computing to obtain an ISCC task allocation system, wherein the task allocation system includes a high-level intelligent agent and a low-level intelligent agent, wherein the high-level intelligent agent includes an ISCC core cloud network, and the low-level intelligent agent includes an ISCC edge network and an ISCC terminal; A second processing module is used to repeatedly execute the target process until the first reward value of the first reward function of the high-level agent and the second reward value of the second reward function of the low-level agent reach maximum values ​​respectively; Wherein, the target process includes: Allocating high-level tasks to the low-level agents according to the configured high-level agent algorithm; According to the configured low-level agent algorithm, determine the resource information contributed by the low-level agent to the target task, send feedback information to the high-level agent, and obtain the second reward value, and update the second network parameter in the low-level agent algorithm according to the second reward value; wherein the second reward value is related to the execution result of the target task, the target task is executed based on the resources contributed by the low-level agent, and the target task includes the high-level task; the feedback information is obtained according to the performance change information of the low-level agent, the task completion quality of the low-level agent for the high-level task, and the smoothing algorithm; The first reward value is obtained, and a first network parameter in the high-level agent algorithm is updated according to the first reward value; wherein the first reward value is determined according to feedback information of the low-level agent.

9. The device according to claim 8, characterized in that The second processing module is used for: Allocating high-level tasks to the low-level agents according to the state information of the high-level agents and the configured high-level agent algorithm; The status information of the high-level agent includes: task-related information of the high-level task, resource status information of the low-level agent, and physical characteristic parameters, wherein the physical characteristic parameters are used to indicate the physical characteristics between the low-level agent and the high-level agent.

10. The device according to claim 8, characterized in that The second processing module is used for: The resource information contributed by the low-level agent to the target task is determined based on the resource status information of the low-level agent, the allocation information of the high-level task, the resource requirement information of the high-level task, the time requirement information of the high-level task and the low-level agent algorithm.

11. The device according to claim 8, characterized in that The second processing module is also used for: Determining the task completion quality of the high-level task according to the completion time of the high-level task and the quality-source ratio of the high-level task; The quality-to-source ratio is the ratio of the quality of the completed high-level task to the amount of resources consumed to complete the high-level task.

12. The device according to claim 8, characterized in that The target task also includes the local task of the low-level agent.

13. The device according to claim 12, characterized in that The second processing module is used for: The second reward value is obtained according to the quality of the high-level task completed by the low-level agent, a first parameter and a second parameter, wherein the first parameter is used to indicate whether the high-level task is completed, and the second parameter is used to indicate whether the local task is completed.

14. The device according to claim 8, characterized in that The second processing module is also used for: The third network parameter in the low-level agent algorithm is updated according to the policy gradient algorithm, wherein the policy gradient algorithm includes a regularization parameter, wherein the regularization parameter is obtained according to a regularization coefficient and a cooperative behavior parameter, and wherein the cooperative behavior parameter is obtained according to resource information contributed by the low-level agent to the target task.

15. A training device for a task allocation system, comprising: A transceiver, a processor, a memory, and a program or instruction stored in the memory and executable on the processor; characterized in that when the processor executes the program or instruction, the steps in the training method of the task allocation system as described in any one of claims 1 to 7 are implemented.

16. A readable storage medium having a program or instruction stored thereon, characterized in that: When the program or instruction is executed by a processor, the steps of the training method for a task allocation system as described in any one of claims 1 to 7 are implemented.

17. A computer program product, characterized in that The method comprises computer instructions, which, when executed by a processor, implement the steps of the training method for the task allocation system according to any one of claims 1 to 7.

Images