A training method and device of a task allocation system and a readable storage medium

Abstract

The application provides a training method and device of a task allocation system and a readable storage medium. The method comprises: modeling a task allocation scene of an ISCC to obtain an ISCC task allocation system; repeatedly executing a target process until a first reward value of a first reward function and a second reward value of a second reward function respectively reach maximum values; the target process comprises: allocating a high-level task to a low-level agent according to a high-level agent algorithm; determining resource information contributed by the low-level agent for a target task, sending feedback information to the high-level agent, and obtaining the second reward value, updating a second network parameter in the low-level agent algorithm according to the second reward value; obtaining the first reward value, updating a first network parameter in the high-level agent algorithm according to the first reward value; and the first reward value is determined according to the feedback information of the low-level agent.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of cloud computing, and particularly relates to a training method and device of a task allocation system and a readable storage medium. BACKGROUND

[0002] In the related art, the 6th Generation Mobile Communication System (6G) will integrate communication, sensing and computing, and the Computing Force Networking (CFN) is a key technology. Sensing, communication and computing are mutually coordinated, multi-level computing power is connected through the network, and real-time sensing and flexible matching of computing resources are realized. However, the increase of network nodes brings complex communication and sensing problems, and it is difficult to meet the high-concurrency business demand. With the help of multi-level computing power and artificial intelligence technology, the integration of communication, sensing, computing and intelligent technology can improve the network sensing accuracy and system efficiency, and meet the demand of 6G emerging business.

[0003] Therefore, the Integrated Sensing, Communication and Computation (ISCC) and intelligent technology are complementary to each other, and jointly build the 6G network system of everything intelligent connection. The terminal device, network device and application server in the 6G system will have intelligent capability, and the terminal and network can select the Artificial Intelligence (AI) algorithm structure according to the application scene and use case. In the face of the diversity of resources and business demand in the 6G network, efficient resource scheduling becomes the key to ensure the network performance and stability. However, there is no related scheme for how to stably allocate resources among different nodes in the ISCC system. SUMMARY

[0004] The purpose of the present application is to provide a training method and device of a task allocation system and a readable storage medium, so as to solve the problem of how to stably allocate resources among different nodes in the ISCC system.

[0005] To achieve the above purpose, an embodiment of the present application provides a training method of a task allocation system, comprising:

[0006] The task allocation scene of the Integrated Sensing, Communication and Computation (ISCC) is modeled and processed to obtain an ISCC task allocation system, the task allocation system comprising a high-level agent and a low-level agent, the high-level agent comprising an ISCC core cloud network, and the low-level agent comprising an ISCC edge network and an ISCC terminal;

[0007] repeating the target process until a first reward value of a first reward function of the high-level agent and a second reward value of a second reward function of the low-level agent respectively reach maximum values;

[0008] The target process comprises:

[0009] allocating the high-level task to the low-level agent according to the configured high-level agent algorithm;

[0010] determining resource information contributed by the low-level agent for the target task, sending feedback information to the high-level agent, and obtaining the second reward value according to the configured low-level agent algorithm, updating a second network parameter in the low-level agent algorithm according to the second reward value; the second reward value is related to an execution result of the target task, the target task is executed based on the resources contributed by the low-level agent, and the target task includes the high-level task; the feedback information is obtained according to performance change information of the low-level agent, task completion quality of the low-level agent for the high-level task, and a smoothing algorithm;

[0011] obtaining the first reward value, and updating a first network parameter in the high-level agent algorithm according to the first reward value; the first reward value is determined according to the feedback information of the low-level agent.

[0012] Optionally, the allocating the high-level task to the low-level agent according to the configured high-level agent algorithm comprises:

[0013] allocating the high-level task to the low-level agent according to the state information of the high-level agent and the configured high-level agent algorithm;

[0014] The state information of the high-level agent includes task-related information of the high-level task, resource state information of the low-level agent, and physical characteristic parameters, and the physical characteristic parameters are used to indicate physical characteristics between the low-level agent and the high-level agent.

[0015] Optionally, the determining resource information contributed by the low-level agent for the target task according to the configured low-level agent algorithm comprises:

[0016] determining the resource information contributed by the low-level agent for the target task according to resource state information of the low-level agent, allocation information of the high-level task, resource requirement information of the high-level task, time requirement information of the high-level task, and the low-level agent algorithm.

[0017] Optionally, the method of the embodiment of the application further comprises:

[0018] determine a task completion quality of the high-level task according to the completion time of the high-level task and a quality source ratio of the high-level task;

[0019] The quality source ratio is a ratio of a quality of the completed high-level task to an amount of resources consumed for completing the high-level task.

[0020] Optionally, the target task further includes a local task of the low-level agent.

[0021] Optionally, the method of the embodiment of the application, the second reward value comprises:

[0022] The second reward value is obtained according to the quality of the high-level task completed by the low-level agent, a first parameter and a second parameter, the first parameter is used to indicate whether the high-level task is completed, and the second parameter is used to indicate whether the local task is completed.

[0023] Optionally, the method of the embodiment of the application further comprises:

[0024] The third network parameter in the low-level agent algorithm is updated according to a policy gradient algorithm, the policy gradient algorithm comprises a regularization parameter, the regularization parameter is obtained according to a regularization coefficient and a cooperative behavior parameter, and the cooperative behavior parameter is obtained according to resource information contributed by the low-level agent for the target task.

[0025] The embodiment of the application further provides a training device of a task allocation system, comprising:

[0026] The first processing module is configured to model a task allocation scene of an integrated sensing and communication (ISCC) to obtain an ISCC task allocation system, the ISCC task allocation system comprising a high-level agent and a low-level agent, the high-level agent comprising an ISCC core cloud network, and the low-level agent comprising an ISCC edge network and an ISCC terminal.

[0027] The second processing module is configured to repeatedly execute a target process until a first reward value of a first reward function of the high-level agent and a second reward value of a second reward function of the low-level agent respectively reach maximum values.

[0028] The target process comprises:

[0029] The high-level task is allocated to the low-level agent according to a configured high-level agent algorithm.

[0030] According to the configured low-level agent algorithm, determine resource information contributed by the low-level agent to a target task, send feedback information to a high-level agent, and obtain a second reward value, and update a second network parameter in the low-level agent algorithm according to the second reward value; wherein the second reward value is related to an execution result of the target task, the target task is executed based on the resources contributed by the low-level agent, and the target task includes the high-level task; the feedback information is obtained according to performance change information of the low-level agent, task completion quality of the low-level agent to the high-level task, and a smoothing algorithm;

[0031] Obtain the first reward value, and update a first network parameter in the high-level agent algorithm according to the first reward value; wherein the first reward value is determined according to the feedback information of the low-level agent.

[0032] Optionally, the second processing module is configured to:

[0033] According to the state information of the high-level agent and the configured high-level agent algorithm, assign a high-level task to the low-level agent;

[0034] The state information of the high-level agent includes task-related information of the high-level task, resource state information of the low-level agent, and physical characteristic parameters, and the physical characteristic parameters are used to indicate physical characteristics between the low-level agent and the high-level agent.

[0035] Optionally, the second processing module is configured to:

[0036] According to the resource state information of the low-level agent, the assignment information of the high-level task, the resource requirement information of the high-level task, the time requirement information of the high-level task, and the low-level agent algorithm, determine resource information contributed by the low-level agent to the target task.

[0037] Optionally, the second processing module is further configured to:

[0038] According to the completion time of the high-level task and the quality-source ratio of the high-level task, determine the task completion quality of the high-level task;

[0039] The quality-source ratio is a ratio of quality of the completed high-level task to an amount of resources consumed for completing the high-level task.

[0040] Optionally, the target task further includes a local task of the low-level agent.

[0041] Optionally, the second processing module is configured to:

[0042] The second reward value is obtained according to a quality of the high-level task completed by the low-level agent, a first parameter and a second parameter, the first parameter being used to indicate whether the high-level task is completed, and the second parameter being used to indicate whether the local task is completed.

[0043] Optionally, the second processing module is further configured to:

[0044] The third network parameter in the low-level agent algorithm is updated according to a policy gradient algorithm, the policy gradient algorithm including a regularization parameter, the regularization parameter being obtained according to a regularization coefficient and a cooperative behavior parameter, the cooperative behavior parameter being obtained according to resource information contributed by the low-level agent for the target task.

[0045] Embodiments of the present application also provide a training device of a task allocation system, comprising a transceiver, a processor, a memory, and a program or instructions stored in the memory and executable on the processor; the processor implements the steps in the training method of the task allocation system when executing the program or instructions.

[0046] Embodiments of the present application also provide a readable storage medium having a program or instructions stored thereon, the program or instructions being executable by a processor to implement the steps in the training method of the task allocation system.

[0047] Embodiments of the present application also provide a computer program product comprising computer instructions, the computer instructions being executable by a processor to implement the steps in the training method of the task allocation system.

[0048] The above technical solutions of the present application have the following beneficial effects:

[0049] In embodiments of the present application, the first reward value corresponding to the high-level agent is obtained based on performance change information of the low-level agent, a task completion quality of the low-level agent for the high-level task, and a smoothing processing algorithm. Based on the first reward value, the stability of the learning of the high-level agent can be effectively ensured, and the learning of the high-level agent is prevented from being affected by the dynamic changes of the low-level agent, thereby effectively ensuring the stability between the high-level agent and the low-level agent, and further enabling the ISCC task allocation system to stably allocate tasks. BRIEF DESCRIPTION OF DRAWINGS

[0050] Figure 1 A power network diagram of an ISCC resource scheduling and allocation system using a layered multi-agent scheduling algorithm based on layered deep reinforcement learning;

[0051] Figure 2 A schematic diagram corresponding to the design idea of the task allocation system of embodiments of the present application;

[0052] Figure 3 A flowchart of a training method of a task allocation system of an embodiment of the present application;

[0053] Figure 4 A schematic diagram of a two-level hierarchy of a hierarchical multi-agent scheduling (HMAS) of an embodiment of the present application;

[0054] Figure 5 A module schematic diagram of a training device of a task allocation system of an embodiment of the present application;

[0055] Figure 6 A structure schematic diagram of a training equipment of a task allocation system of an embodiment of the present application. DETAILED DESCRIPTION

[0056] In order to make the technical problems to be solved, technical solutions and advantages of the present application clearer, the following will be described in detail with reference to the accompanying drawings and specific embodiments.

[0057] It should be understood that the terms "one embodiment" or "an embodiment" mentioned throughout the specification mean that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner.

[0058] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than that illustrated or described herein, and the objects distinguished by "first", "second" are generally a category, not limited to the number of objects, for example, the first object can be one or more. In addition, "and / or" in the specification and claims means at least one of the connected objects, for example, the protection scope of "A and / or B" covers at least three schemes, namely, scheme one: including A and not including B; scheme two: including B and not including A; scheme three: including A and B. The character " / " generally represents that the objects before and after are in an "or" relationship.

[0059] In various embodiments of the present application, it should be understood that the size of the serial number of the following processes does not mean the order of execution, and the execution order of the processes should be determined by their functions and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0060] In addition, the terms "system" and "network" are often used interchangeably herein.

[0061] In the embodiments provided in the present application, it should be understood that "B corresponding to A" means that B is associated with A, and B can be determined according to A. However, it should also be understood that the determination of B according to A does not mean that B is determined only according to A, but B can also be determined according to A and / or other information.

[0062] In order to meet the vision of 6G green development, realize efficient coordination and utilization of 6G network resources, the present application provides a hierarchical multi-agent scheduling (HMAS) framework based on deep reinforcement learning. The framework divides resource scheduling and collaborative tasks into two levels, namely high-level agent (HLA) and low-level agent (LLA). The HMAS framework uses a dynamic feedback adjustment mechanism (DFAM) to solve the non-stationary problem between HLA and LLA, improve the efficiency and stability of the overall learning, and accelerate the convergence speed of the hierarchical reinforcement learning system. A cooperative reward mechanism (CRM) is used to improve the coordination effect of LLA. The algorithm has low complexity and can realize efficient use of 6G resources while ensuring timely completion of tasks.

[0063] Figure 1 In order to use the HMAS algorithm based on hierarchical deep reinforcement learning in the computing force networking (CFN) of the ISCC resource scheduling and allocation system. The ISCC architecture network is layered, where the ISCC core cloud network is the HLA, responsible for task allocation; the ISCC edge network (including mobile edge computing servers, base stations, etc.) and the ISCC terminal belong to the LLA, responsible for executing tasks. The entire system is trained to maximize the reward function. In order to solve the non-stationary problem between HLA and LLA, that is, the HLA learning is affected by the dynamic changes of the LLA learning process, the HMAS method uses DFAM to handle the reward feedback problem of HLA, thereby ensuring the stability of HLA learning. CRM is used to let LLA have a reciprocal idea while ensuring its own reward, so that LLA can coordinate better. The schematic diagram corresponding to the above design idea is shown in Figure 2 The high-level agent assigns tasks to the low-level agent, and the low-level agent sends feedback information to the high-level agent based on DFAM.

[0064] Based on the above design idea, as shown in Figure 3As shown, the present application provides a training method of a task allocation system, comprising:

[0065] Step 301: modeling the task allocation scene of the integrated ISCC, to obtain an ISCC task allocation system, the ISCC task allocation system comprising a high-level agent and a low-level agent, the high-level agent comprising an ISCC core cloud network, and the low-level agent comprising an ISCC edge network and an ISCC terminal.

[0066] As shown, the ISCC core cloud network comprises a core cloud server, and the ISCC edge network comprises a mobile edge computing server, a base station, etc. The high-level agent is responsible for task allocation, and the low-level agent is responsible for task execution. Figure 2

[0067] Step 302: repeatedly executing a target process until the first reward value of the first reward function of the high-level agent and the second reward value of the second reward function of the low-level agent respectively reach maximum values.

[0068] The target process comprises:

[0069] allocating a high-level task to the low-level agent according to a configured high-level agent algorithm;

[0070] determining resource information contributed by the low-level agent for the target task according to a configured low-level agent algorithm, sending feedback information to the high-level agent, obtaining the second reward value, updating a second network parameter in the low-level agent algorithm according to the second reward value, wherein the second reward value is related to an execution result of the target task, the target task is executed based on the resources contributed by the low-level agent, and the target task comprises the high-level task; the feedback information is obtained according to performance change information of the low-level agent, task completion quality of the low-level agent for the high-level task, and a smoothing algorithm;

[0071] obtaining the first reward value, and updating a first network parameter in the high-level agent algorithm according to the first reward value; wherein the first reward value is determined according to the feedback information of the low-level agent.

[0072] Optionally, the smoothing algorithm is an exponential smoothing algorithm. The mechanism of sending the feedback information by the low-level agent to the high-level agent is the DFAM mechanism.

[0073] Optionally, the performance change information of the low-level agent is obtained based on a performance monitoring function, which can be determined based on the variance of the low-level agent.

[0074] ​Optionally, the execution result of the target task includes a quality of the high-level task completed by the low-level agent, a first parameter and a second parameter, the first parameter being used to indicate whether the high-level task is completed, and the second parameter being used to indicate whether the local task is completed.

[0075] Optionally, updating the first network parameter in the high-level agent algorithm according to the first reward value includes updating a parameter of a critic network in the high-level agent algorithm according to the first reward value. For example, a parameter of the critic network in the high-level agent algorithm is updated based on the first reward value by using a temporal-difference (TD) algorithm.

[0076] Optionally, the method of the embodiment of the present application further includes updating a parameter of an actor network in the high-level agent algorithm based on a policy gradient algorithm and an adaptive learning rate. The adaptive learning rate is obtained according to the performance change information of the low-level agent. The performance change information of the low-level agent can be obtained based on a performance monitoring function.

[0077] Optionally, updating the second network parameter in the low-level agent algorithm according to the second reward value includes updating a parameter of a critic network in the low-level agent algorithm according to the second reward value. For example, a parameter of the critic network in the low-level agent algorithm is updated based on the TD algorithm using the second reward value.

[0078] Optionally, the method of the embodiment of the present application further includes:

[0079] The parameter of the actor network in the low-level agent algorithm is updated according to a policy gradient method, i.e., the third network parameter.

[0080] In the embodiment of the present application, the first reward value corresponding to the high-level agent is obtained based on the performance change information of the low-level agent, the quality of the high-level task completed by the low-level agent, and a smoothing algorithm. Based on the first reward value, the stability of the learning of the high-level agent can be effectively ensured, and the learning of the high-level agent is prevented from being affected by the dynamic change of the low-level agent, thereby effectively ensuring the stability between the high-level agent and the low-level agent, and further enabling the ISCC task allocation system to stably perform task allocation.

[0081] The DFAM mechanism is described in detail below in combination with embodiments.

[0082] As Figure 4As shown, HMAS is a two-level hierarchy structure, and the final goal of the multi-agent system is to maximize the global reward. Both the high-level agent algorithm and the low-level agent algorithm adopt the Actor-Critic (AC) algorithm, wherein the actor network (i.e. ) of the high-level agent algorithm performs task allocation according to the overall state of the environment, the actor network of the low-level agent algorithm makes actions according to the target task. The critic network (Critic network, ) of the high-level agent algorithm adopts DFAM to improve the stability and efficiency of policy learning, and the critic network of the low-level agent algorithm is . Among them, represents the action of the high-level agent at time t, represents the action of the low-level agent at time t, represents the reward of the low-level agent at time t, represents the state space at time t, specifically the LLA state space and / or the HLA state space at time t, represents the actor network parameter of the high-level agent; represents the actor network parameter of the low-level agent.

[0083] The Markov decision process of the HLA and the LLA of the HMAS is respectively:

[0084] ;

[0085] ;

[0086] is the Markov decision process of the HLA, is the HLA state space, is the HLA action space, is the HLA state transition function, is the HLA reward function (i.e. the first reward function described above), is the high-level discount coefficient. is the Markov decision process of the LLA, is the LLA state space, is the LLA action space, is the LLA state transition function, is the LLA reward function (i.e. the second reward function described above), is the low-level discount coefficient.

[0087] 1) high-level agent;

[0088] The role of the HLA (Host Agent Network) is to assign tasks to different LLAs (Lack Agent Networks) to facilitate collaborative work across the team. The HLA employs an actor-critic algorithm to learn task assignment strategies, predicting the performance of LLAs on different tasks and determining assignments based on these predictions. The LLA also uses an actor-critic algorithm. The actor network selects actions, while the critic network evaluates the value of those actions and provides feedback. This algorithm allows the agent to learn how to better perform tasks based on the HLA's assignments.

[0089] Optionally, the method in this application embodiment further includes:

[0090] The task completion quality of the high-level task is determined based on the completion time of the high-level task and the quality-source ratio of the high-level task.

[0091] The quality-source ratio is the ratio of the quality of the completed high-level task to the amount of resources consumed in completing the high-level task.

[0092] In this embodiment of the application, feedback information is obtained based on the quality-source ratio, enabling the high-level intelligent agent to know the ratio of the quality of the completed high-level task to the amount of resources consumed in completing the high-level task.

[0093] The first reward function mentioned above, also known as the global reward function, is defined as follows:

[0094] ;

[0095] ;

[0096] Where, assuming the total number of tasks That is, the total number of high-level tasks is , This indicates that a lower-level intelligent agent completes a higher-level task. The quality of task completion Indicates the assigned high-level tasks Completion time, Depending on the maximum completion time assigned to the LLA group, It is a small positive number used to avoid the denominator being zero. The "quality-to-resource ratio" is defined as the ratio between the quality of the completed task and the amount of resources consumed. Indicates the tasks completed by LLA The quality (measured by task satisfaction). Indicates completion of high-level tasks The amount of resources consumed (compared to computing resources) (Related). Considering that different tasks may have varying degrees of importance and difficulty, a weight is assigned to each task. , denotes a weight factor, .

[0097] The update of LLA has an impact on HLA, however, LLA presents a dynamic change trend from bad to good during training, in this non-stationary environment, it is difficult for HLA to learn the optimal strategy. The exploration of HLA is crucial to the overall learning process, its non-stationarity leads to the uncertainty of the transmission process, affecting the strategy optimization. HLA sets a goal and expects LLA to achieve the maximum reward, but the dynamic change of LLA in training may cause the reward feedback received by HLA to be unmatched with its true value, which is not conducive to HLA learning. Therefore, in order to solve the problem of mismatch between HLA and LLA during training, the present application designs a mechanism to improve the robustness of the system, called DFAM. This mechanism needs to take into account the dynamic change of LLA, while ensuring that HLA can obtain rewards that truly reflect the value of its behavior. Specifically, the HLA's needs to go through the following three steps:

[0098] First, define a smoothing function to smooth the task completion quality of the low-level agent, using exponential smoothing:

[0099] ;

[0100] where is a smoothing factor, , is the of the previous time step.

[0101] Second, model the strategy uncertainty, define the performance monitoring function to estimate the uncertainty of the LLA group. The LLA group includes at least one LLA, the function is based on the variance of the low-level agent, which can be updated based on the following formula:

[0102] ;

[0103] where is a weight factor, , used to control the influence of uncertainty on the high-level strategy reward, are the parameters of the actor network of the low-level agent.

[0104] Finally, combine the above two methods, that is:

[0105] ;

[0106] Here the final ​This refers to the feedback information sent from lower-level agents to higher-level agents. This feedback ensures that the rewards received by the HLA are matched to their true value, thus facilitating the HLA's learning and training.

[0107] Optionally, assigning high-level tasks to the low-level agents according to the configured high-level agent algorithm includes:

[0108] Based on the state information of the high-level agent and the configured high-level agent algorithm, the high-level task is assigned to the low-level agent.

[0109] The state information of the high-level intelligent agent includes: task-related information of the high-level task, resource state information of the low-level intelligent agent, and physical characteristic parameters, wherein the physical characteristic parameters are used to indicate the physical characteristics between the low-level intelligent agent and the high-level intelligent agent.

[0110] Optionally, the task-related information of the high-level task includes, but is not limited to, the resource requirement information and the time requirement information of the high-level task.

[0111] Optionally, the object feature parameters include, but are not limited to, distance information between the lower-level agent and the higher-level agent, and channel quality information.

[0112] Optionally, the resource requirement information of the high-level task includes the high-level task's requirements for computing resources, network resources, and storage resources.

[0113] Optionally, the time requirement information of the high-level task includes the maximum tolerable time of the high-level task, for example, the maximum delay in completing the high-level task.

[0114] Optionally, the resource status information of the low-level intelligent agent includes the remaining computing resources, network resources, storage resources, and service resources of the low-level intelligent agent at the current moment.

[0115] In this application embodiment, different types of high-level tasks are targeted. The design environment input state is .in, The terms in the table represent the requirements of high-level task i for computing resources, network resources, storage resources, and the maximum tolerable time of the task. Represents the nth agent ( ) state, Each item in the middle represents The current remaining computing, network, storage, and service resources, as well as the physical distance to HLA. The actor network executes actions, i.e., outputs task allocation. ,in Represented as The proportion of the assigned task, and .

[0116] In the embodiments of the present application, according to different types of high-level tasks, the high-level tasks are assigned based on the resource state information of the low-level agent and the distance information between the low-level agent and the high-level agent, so that the task allocation can be made according to the capability difference of the low-level agent, so that the low-level agent can execute the type of task that it is good at executing, and the task execution efficiency is improved. When assigning a new task, the resource state information of the low-level agent is considered, the resources occupied by the unfinished old task are considered, and the current remaining resources of the low-level agent are used for task allocation, so as to effectively avoid the problem that when the low-level agent does not complete the old task and receives a new task, there is not enough resource to execute the new task, thereby improving the execution efficiency of the task.

[0117] In the embodiments of the present application, the HLA uses the actor-critic architecture, adopts the Deep Deterministic Policy Gradient (DDPG), and contains two parts of actor network and critic network, wherein the actor network contains online policy and target policy. The network state-action function of the critic network is randomly initialized .

[0118] The HLA takes the state as input, adopts the actor network of the HLA to calculate the probability distribution of the action, and then samples the action: .

[0119] The HLA executes the action , at this time the environment gives a new state and a reward .

[0120] As Figure 4 shown, the is stored in the experience replay pool (referred to as experience pool). Some samples are randomly extracted from the experience replay pool for training, and it is assumed that is a batch of randomly sampled data, the target network is trained by temporal difference (TD), and the target network is expressed as:

[0121]

[0122] According to the immediate reward , the critic network parameters of the HLA are updated by TD algorithm :

[0123]

[0124]

[0125] is a discount factor, . Where is a differential operation, representing the difference between the critic network's predicted return and the actual return of HLA. The critic network of HLA adjusts its parameters according to this difference to more accurately predict future returns.

[0126] Finally, the actor network parameters of HLA are updated using policy gradient .

[0127]

[0128] is a differential operation, is the action value function of HLA, is the state value function of HLA.

[0129] Where is an adaptive learning rate, can be dynamically adjusted according to the performance changes of LLA. When the performance of LLA fluctuates greatly, the learning rate of HLA can be reduced to reduce the policy update error caused by inaccurate reward feedback. As the performance of LLA gradually stabilizes, the learning rate of HLA can be gradually increased to speed up learning. A performance monitoring function is used to measure the performance of LLA and adjust the learning rate accordingly:

[0130]

[0131] The critic network calculates the TD loss and updates the actor network parameters. First, the critic network uses the Deep Q-learning Network (DQN) algorithm based on experience replay to optimize parameters. Experience replay is a method of learning policy and value function by replaying previously collected experience data, which can reduce data correlation, avoid overfitting, and more effectively utilize data. The critic network calculates the optimal value of the state through Q-learning.

[0132] The critic network of HLA uses the DQN algorithm based on experience replay to optimize parameters, which is implemented through the TD algorithm:

[0133] ;

[0134] ;

[0135] where, is the immediate reward, is the discount factor, is the adaptive learning rate.

[0136] TD error represents the difference between the critic network's predicted return and the actual return.

[0137] The critic network adjusts its parameters based on this difference to more accurately predict future returns.

[0138] The optimal value of the state is calculated, and the actor network uses this optimal value to iteratively update the parameters of the policy function, then selects an action and obtains feedback and the next state. The policy function parameter update is as follows:

[0139] .

[0140] Optionally, according to the configured low-level agent algorithm, the resource information contributed by the low-level agent to the target task is determined, including:

[0141] According to the resource state information of the low-level agent, the allocation information of the high-level task, the resource requirement information of the high-level task, the time requirement information of the high-level task, and the low-level agent algorithm, the resource information contributed by the low-level agent to the target task is determined.

[0142] Optionally, the resource information contributed by the low-level agent to the target task includes proportion information of each type of resource contributed by the low-level agent to the target task.

[0143] In the embodiment of the application, the input of the low-level agent algorithm is composed of three parts: resource state information of the low-level agent, allocation information of the high-level task, and resource requirement information of the high-level task. Among them, the resource state information of the low-level agent is , where each item represents the computing resources, network resources, storage resources, and service resources that the low-level agent itself possesses for high-level task i.

[0144] Optionally, for low-level agent n, the above allocation information of the high-level task is .

[0145] Optionally, the resource requirement information of the high-level task is represented by .

[0146] In summary, for high-level task i, the input of the low-level agent algorithm is represented as . output , respectively, represent willingness to contribute to high-level tasks , the proportion of various resources, when , .

[0147] Optionally, the target task further includes a local task of the low-level agent.

[0148] In the embodiments of the present application, in addition to completing the high-level tasks assigned by the HLA, the LLA also needs to handle some unexpected tasks, which are collectively referred to as regular tasks, i.e. local tasks, which also need to consume certain LLA resources. The LLA needs to execute both the tasks assigned by the HLA and the local regular tasks.

[0149] In the embodiments of the present application, in the low-level agent of layered reinforcement learning, the requirements of local regular tasks and high-level tasks are considered at the same time, and the computing power resources of the low-level agent are intelligently allocated by taking the contribution degree as the execution action, so as to effectively guarantee the execution of local tasks while executing high-level tasks, and add the quality source ratio as an evaluation index for feedback, which can efficiently process tasks in parallel while achieving lower energy efficiency.

[0150] Optionally, the method of the embodiments of the present application, acquiring the second reward value, comprises:

[0151] According to the quality of the high-level task completed by the low-level agent, a first parameter and a second parameter, the second reward value is acquired, the first parameter is used to indicate whether the high-level task is completed, and the second parameter is used to indicate whether the local task is completed.

[0152] In the embodiments of the present application, the quality of the high-level task completed by the low-level agent is measured by user satisfaction.

[0153] Optionally, the second reward function is:

[0154] ;

[0155] wherein, represents the quality (measured by user satisfaction) of actually completing the high-level task, represents whether the assigned high-level task is completed, represents whether the local task is completed (0: No, 1: Yes), is an adjustment factor, is an adjustment factor, wherein, i.e. the first parameter mentioned above, 0 means that the high-level task i is not completed, 1 means that the high-level task i is completed, i.e. the second parameter mentioned above, 0 means that the local task is not completed, 1 means that the local task is completed.

[0156] Optionally, the method of the embodiment of the application further comprises:

[0157] updating a third network parameter in the low-level agent algorithm according to a policy gradient algorithm, the policy gradient algorithm comprising a regularization parameter, the regularization parameter being obtained according to a regularization coefficient and a cooperative behavior parameter, the cooperative behavior parameter being obtained according to resource information contributed by the low-level agent for the target task.

[0158] The third network parameter can be understood as a parameter of an actor network in the low-level agent algorithm.

[0159] In the embodiment of the application, each LLA is also constructed using an actor-critic model. The LLA updates its actor network according to a policy gradient method to maximize the expected return, and considering the tit-for-tat strategy, the policy gradient needs to include an additional term to encourage cooperative behavior. The critic network is updated by a TD method to accurately estimate the state value or action value.

[0160] Let be the total return expected by the low-level agent taking action in state , be the policy function (which determines the probability of taking action in state ), be the parameters of the actor network for , be the parameters of the critic network for .

[0161]

[0162] In combination with the tit-for-tat strategy, a regularization term is added in the gradient update to encourage cooperation.

[0163]

[0164] represents cooperative behavior, i.e. the cooperative behavior parameter mentioned above, and the judgment condition is and , . is a regularization coefficient.

[0165] wherein, is the regularization parameter described above.

[0166] The critic network of the LLA employs a DQN, an experience replay-based algorithm, to optimize the parameters, which is implemented through a TD algorithm:

[0167]

[0168]

[0169] is the immediate reward, is the discount factor, is the learning rate. The TD error represents the difference between the reward predicted by the critic network of the LLA and the actual reward. The critic network of the LLA adjusts its parameters to more accurately predict future rewards.

[0170] In the embodiments of the present application, a cooperative reward mechanism (CRM) is used to improve the coordination effect of the LLA.

[0171] In the embodiments of the present application, first, a reserved resource setting mode is adopted for multiple task sources; second, multiple different agents with different task processing capabilities are set for multiple task types; and finally, the completion of the previous task is considered to cope with the situation where the tasks exist in the time dimension. Through the above methods, the execution efficiency of the task is effectively improved.

[0172] As shown in Figure 5 , the present application also provides a training device of a task allocation system, which comprises:

[0173] A first processing module 501 is configured to model a task allocation scene of an integrated sensing and communication cloud (ISCC) to obtain an ISCC task allocation system, wherein the ISCC task allocation system comprises a high-level agent and a low-level agent, the high-level agent comprises an ISCC core cloud network, and the low-level agent comprises an ISCC edge network and an ISCC terminal.

[0174] A second processing module 502 is configured to repeatedly execute a target process until a first reward value of a first reward function of the high-level agent and a second reward value of a second reward function of the low-level agent respectively reach maximum values.

[0175] The target process comprises:

[0176] According to the configured high-level agent algorithm, the high-level task is allocated to the low-level agent;

[0177] According to the configured low-level agent algorithm, the low-level agent determines the resource information contributed by the low-level agent for the target task, sends feedback information to the high-level agent, and obtains the second reward value, and updates the second network parameter in the low-level agent algorithm according to the second reward value; wherein the second reward value is related to the execution result of the target task, the target task is executed based on the resources contributed by the low-level agent, and the target task includes the high-level task; the feedback information is obtained according to the performance change information of the low-level agent, the task completion quality of the low-level agent to the high-level task, and a smoothing algorithm;

[0178] The first reward value is obtained, and the first network parameter in the high-level agent algorithm is updated according to the first reward value; wherein the first reward value is determined according to the feedback information of the low-level agent.

[0179] Optionally, the second processing module is configured to:

[0180] According to the state information of the high-level agent and the configured high-level agent algorithm, the high-level task is allocated to the low-level agent;

[0181] The state information of the high-level agent includes: task-related information of the high-level task, resource state information of the low-level agent, and physical characteristic parameters, and the physical characteristic parameters are used to indicate the physical characteristics between the low-level agent and the high-level agent.

[0182] Optionally, the second processing module is configured to:

[0183] According to the resource state information of the low-level agent, the allocation information of the high-level task, the resource requirement information of the high-level task, the time requirement information of the high-level task, and the low-level agent algorithm, the resource information contributed by the low-level agent for the target task is determined.

[0184] Optionally, the second processing module is further configured to:

[0185] According to the completion time of the high-level task and the quality-source ratio of the high-level task, the task completion quality of the high-level task is determined;

[0186] The quality-source ratio is the ratio of the quality of the completed high-level task to the amount of resources consumed to complete the high-level task.

[0187] Optionally, the target task further includes a local task of the low-level agent.

[0188] Optionally, the second processing module is used for:

[0189] The second reward value is obtained based on the quality of the high-level task completed by the low-level agent, a first parameter, and a second parameter. The first parameter is used to indicate whether the high-level task is completed, and the second parameter is used to indicate whether the local task is completed.

[0190] Optionally, the second processing module is further configured to:

[0191] The third network parameters in the low-level agent algorithm are updated according to the policy gradient algorithm. The policy gradient algorithm includes a regularization parameter, which is obtained based on the regularization coefficient and the cooperative behavior parameter. The cooperative behavior parameter is obtained based on the resource information contributed by the low-level agent to the target task.

[0192] In this embodiment, the first reward value corresponding to the high-level agent is obtained based on the performance change information of the low-level agent, the task completion quality of the low-level agent for the high-level task, and the smoothing algorithm. Based on this first reward value, the stability of the learning of the high-level agent can be effectively guaranteed, and the learning of the high-level agent can be avoided from being affected by the dynamic changes of the low-level agent. This effectively ensures the stability between the high-level agent and the low-level agent, thereby enabling the ISCC task allocation system to allocate tasks stably.

[0193] It should be noted that this device is the same as the device described in the above method embodiments. All implementations of the above method embodiments can be applied to this device embodiment and achieve the same technical effect, which will not be repeated here.

[0194] This application provides a training device for a task allocation system, such as... Figure 6 As shown, the system includes a transceiver 610, a processor 600, a memory 620, and a program or instructions stored in the memory 620 and executable on the processor 600; when the processor 600 executes the program or instructions, it implements the steps of the training method for the task allocation system described above. The training device for this task allocation system can be an electronic device, such as a terminal, computer, or user equipment.

[0195] The transceiver 610 is used to receive and send data under the control of the processor 600.

[0196] Among them, Figure 6In particular embodiments, the bus architecture can include any number of interconnected buses and bridges, specifically, various circuitry linking the one or more processors represented by the processor 600 and the memory represented by the memory 620. The bus architecture can also link various other circuitry such as peripheral devices, voltage regulators, and power management circuitry, all of which are well known in the art and thus, not further described herein. The bus interface provides an interface to the transceiver 610. The transceiver 610 can be a plurality of elements, i.e., including a transmitter and a receiver, providing a means for communicating with various other apparatus over a transmission medium. The user interface 630 can also be an interface capable of coupling to various devices such as a keypad, a display, a speaker, a microphone, a joystick, etc., as are well known in the art, for a user of the various user devices.

[0197] The processor 600 is responsible for managing the bus architecture and general processing, and the memory 620 can store data used by the processor 600 in executing operations.

[0198] The embodiments of the present application further provide a readable storage medium, which has a program or instruction stored thereon, the program or instruction is executed by a processor to implement the steps in the training method of the task allocation system and achieve the same technical effects. To avoid repetition, details are not described here.

[0199] The embodiments of the present application further provide a computer program product, which includes computer instructions, the computer instructions are executed by a processor to implement various processes of the embodiments of the training method of the task allocation system and achieve the same technical effects. To avoid repetition, details are not described here.

[0200] The processor is the processor in the training device of the task allocation system in the above embodiments. The readable storage medium includes a computer readable storage medium, such as a computer readable only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a magnetic disk or an optical disk, etc.

[0201] It is further noted that the terminals described in this specification include but are not limited to smart phones, tablet computers, etc., and many functional components described are referred to as modules to more specifically emphasize the independence of their implementation.

[0202] In this embodiment, the module can be implemented in software so that it can be executed by various types of processors. For example, an identified executable code module may include one or more physical or logical blocks of computer instructions, which may be constructed as objects, procedures, or functions. Nevertheless, the executable code of the identified module does not need to be physically located together, but may include different instructions stored in different bits, which, when logically combined, constitute the module and achieve the module's intended purpose.

[0203] In practice, an executable code module can be a single instruction or many instructions, and can even be distributed across multiple different code segments, different programs, and across multiple memory devices. Similarly, operational data can be identified within the module and can be implemented in any suitable form and organized within any suitable data structure. This operational data can be collected as a single dataset or distributed across different locations (including different storage devices), and can exist, at least in part, solely as electronic signals within the system or network.

[0204] When a module can be implemented using software, considering the current level of hardware technology, modules that can be implemented in software can be implemented using hardware circuits by those skilled in the art to achieve the corresponding functions, without considering cost. These hardware circuits include conventional very-large-scale integrated circuits (VLSI) or gate arrays, as well as existing semiconductors such as logic chips and transistors, or other discrete components. Modules can also be implemented using programmable hardware devices, such as field-programmable gate arrays, programmable array logic, and programmable logic devices.

[0205] The foregoing exemplary embodiments are described with reference made to the drawings which are provided for the purpose of explanation and illustration and are not intended to limit the scope of the application. Indeed, various modifications and variations that fall within the spirit and scope of the application can become apparent to those skilled in the art upon reading this specification, and it is to be understood that such modifications and variations are intended to fall within the scope of the application. Further, it is to be understood that the phraseology and terminology employed herein are for the purpose of description and explanation and are not intended to be limiting. The use of "including" and "comprising" and variations thereof is meant to encompass the items listed thereafter and equivalents thereof as well as additional items and equivalents thereof. Unless otherwise specified, a range of values includes the beginning and end points of the range as well as any sub-ranges therebetween.

[0206] The above description is considered that of the preferred embodiments of the application only. Modifications and alterations will occur to others upon reading the preceding description and it is intended to include all such modifications and alterations insofar as they come within the scope of the claims.

Claims

1. A training method of a task allocation system, characterized by, The application relates to a task allocation system of an integrated sensing and computing (ISCC) system. The target process is repeatedly executed until a first reward value of a first reward function of the high-level agent and a second reward value of a second reward function of the low-level agent respectively reach maximum values. The target process comprises the following steps: According to a configured high-level agent algorithm, a high-level task is allocated to the low-level agent. According to a configured low-level agent algorithm, resource information contributed by the low-level agent for a target task is determined, feedback information is sent to the high-level agent, the second reward value is obtained, the second network parameter in the low-level agent algorithm is updated according to the second reward value, the second reward value is related to an execution result of the target task, the target task is executed based on the resources contributed by the low-level agent, and the target task comprises the high-level task; the feedback information is obtained according to performance change information of the low-level agent, task completion quality of the low-level agent for the high-level task and a smoothing processing algorithm; The first reward value is obtained, and the first network parameter in the high-level agent algorithm is updated according to the first reward value; the first reward value is determined according to the feedback information of the low-level agent. According to the state information of the high-level agent and the configured high-level agent algorithm, the high-level task is allocated to the low-level agent. The first reward function is: ; ； wherein the total number of high-level tasks is , denotes the task completion quality of a high-level task by a low-level agent, denotes the completion time of a high-level task , is a positive number, denotes the quality of a high-level task completed by a low-level agent, denotes the amount of resources consumed for completing a high-level task , denotes the weight assigned to a high-level task , denotes the weight factor, ; The second reward function is: ; wherein, denotes the nth low-level agent actually accomplishes the high-level task of quality, denotes the assigned high-level task is completed, denotes the local task is completed, is a tuning factor, .

2. The method of claim 1, wherein, The state information of the high-level agent comprises task-related information of the high-level task, resource state information of the low-level agent and physical characteristic parameters, and the physical characteristic parameters are used for indicating physical characteristics between the low-level agent and the high-level agent. According to the resource state information of the low-level agent, allocation information of the high-level task, resource demand information of the high-level task, time requirement information of the high-level task and the low-level agent algorithm, the resource information contributed by the low-level agent for the target task is determined. The target process further comprises the following steps:

3. The method of claim 1, wherein, According to the completion time of the high-level task and the quality-source ratio of the high-level task, the task completion quality of the high-level task is determined. The quality-source ratio is a ratio of quality of the completed high-level task to resource consumption amount for completing the high-level task.

4. The method of claim 1, wherein, The target task further comprises a local task of the low-level agent. The second reward value is obtained by the following steps: The second reward value is obtained according to quality of the high-level task completed by the low-level agent, a first parameter and a second parameter; the first parameter is used for indicating whether the high-level task is completed, and the second parameter is used for indicating whether the local task is completed.

5. The method of claim 1, wherein, The target process further comprises the following steps:

6. The method of claim 5, wherein, ​ ​ 7. The method of claim 1, wherein, ​ The third network parameter in the low-level agent algorithm is updated according to a policy gradient algorithm, the policy gradient algorithm includes a regularization parameter, the regularization parameter is obtained according to a regularization coefficient and a cooperative behavior parameter, and the cooperative behavior parameter is obtained according to resource information contributed by the low-level agent for the target task.

8. A training device for a task allocation system, characterized in that Comprise: The first processing module is used for modeling a task allocation scene of an integrated sensing and communication (ISCC), to obtain an ISCC task allocation system, the ISCC task allocation system comprising a high-level agent and a low-level agent, the high-level agent comprising an ISCC core cloud network, and the low-level agent comprising an ISCC edge network and an ISCC terminal; The second processing module is used for repeatedly executing a target process until a first reward value of a first reward function of the high-level agent and a second reward value of a second reward function of the low-level agent respectively reach maximum values; The target process comprises: allocating a high-level task to the low-level agent according to a configured high-level agent algorithm; determining resource information contributed by the low-level agent for a target task, sending feedback information to the high-level agent, and obtaining the second reward value according to a configured low-level agent algorithm, updating a second network parameter in the low-level agent algorithm according to the second reward value, wherein the second reward value is related to an execution result of the target task, the target task is executed based on resources contributed by the low-level agent, and the target task comprises the high-level task; the feedback information is obtained according to performance change information of the low-level agent, task completion quality of the low-level agent for the high-level task, and a smoothing processing algorithm; obtaining the first reward value, and updating a first network parameter in the high-level agent algorithm according to the first reward value, wherein the first reward value is determined according to the feedback information of the low-level agent; The first reward function is: ; ； wherein the total number of high-level tasks is , denotes the task completion quality of a low-level agent completing a high-level task , denotes the completion time of a high-level task , is a positive number, denotes the quality of a high-level task completed by a low-level agent, denotes the amount of resources consumed for completing a high-level task , denotes the weight assigned to a high-level task , denotes the weight factor, ; The second reward function is: ; wherein, represents the nth low-level agent actually accomplishes the high-level task quality, represents the assigned high-level task is completed, represents the local task is completed, is a tuning factor, .

9. The apparatus of claim 8, wherein, The second processing module is used for: allocating the high-level task to the low-level agent according to state information of the high-level agent and the configured high-level agent algorithm; The state information of the high-level agent comprises task-related information of the high-level task, resource state information of the low-level agent, and a physical characteristic parameter, the physical characteristic parameter being used for indicating physical characteristics between the low-level agent and the high-level agent.

10. The apparatus of claim 8, wherein, The second processing module is used for: determining the resource information contributed by the low-level agent for the target task according to resource state information of the low-level agent, allocation information of the high-level task, resource demand information of the high-level task, time requirement information of the high-level task, and a low-level agent algorithm.

11. The apparatus of claim 8, wherein, The second processing module is further used for: determining task completion quality of the high-level task according to a completion time of the high-level task and a quality-to-source ratio of the high-level task, wherein the quality-to-source ratio is a ratio of quality of the completed high-level task to an amount of resources consumed for completing the high-level task. The target task further comprises a local task of the low-level agent.

12. The apparatus of claim 8, wherein, The second processing module is used for:

13. The apparatus of claim 12, wherein, ​ The second reward value is obtained according to a quality of the high-level task completed by the low-level agent, a first parameter and a second parameter, the first parameter being used to indicate whether the high-level task is completed, and the second parameter being used to indicate whether the local task is completed.

14. The apparatus of claim 8, wherein, The second processing module is further configured to: The third network parameter in the low-level agent algorithm is updated according to a policy gradient algorithm, the policy gradient algorithm including a regularization parameter, the regularization parameter being obtained according to a regularization coefficient and a cooperative behavior parameter, the cooperative behavior parameter being obtained according to resource information contributed by the low-level agent for the target task.

15. A training apparatus for a task allocation system, comprising: A transceiver, a processor, a memory, and a program or instructions stored on the memory and executable on the processor; and wherein the processor implements the steps in the training method of the task allocation system according to any one of claims 1 to 7 when executing the program or instructions.

16. A readable storage medium, on which a program or instructions are stored, characterized in that, The program or instructions implement the steps in the training method of the task allocation system according to any one of claims 1 to 7 when executed by a processor.

17. A computer program product, characterised in that, The program or instructions implement the steps in the training method of the task allocation system according to any one of claims 1 to 7 when executed by a processor. The program or instructions implement the steps in the training method of the task allocation system according to any one of claims 1 to 7 when executed by a processor.

Images