Communication and computing resource joint allocation optimization method for multiple agents

Abstract

Disclosed in the present invention is a communication and computing resource joint allocation optimization method for multiple agents, capable of constructing a communication channel model, a computing model and a motion energy consumption model on the basis of a current industrial workshop environment, an edge server and agent parameters. Each agent interacts with the environment to obtain a current observation value, inputs the obtained observation value into an action network, obtains an action parameter corresponding to a next moment, calculates a current reward, updates neural network parameters, and trains current data, so as to obtain a communication and computing resource joint allocation strategy. The present invention can optimize the allocation method when the communication and computing resources are limited, so as to satisfy the requirements of low delay and high reliability in communication, and timely computing power, thereby reducing resource waste.

Description

A method for joint allocation optimization of communication and computing resources of multiple agents

[0001] The present application claims priority to the Chinese patent application filed on July 2, 2024 with the Chinese Patent Office and with the application number 2024108771167, and with the title of "A method for joint allocation optimization of communication and computing resources of multiple agents", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] The present application belongs to the technical field of agent resource allocation optimization methods, and specifically relates to a method for joint allocation optimization of communication and computing resources of multiple agents. BACKGROUND

[0003] Under the industrial workshop scenario, the Internet of Things technology for equipment develops rapidly, and the limited computing capacity and communication bandwidth of terminal devices limit the popularization and application of the Internet of Things technology.

[0004] In the prior art, the allocation technology of communication and computing resources generally takes a single agent as the allocation optimization object, and allocates communication and computing resources respectively. These methods are relatively simple and can achieve part of the optimization goals in the industrial workshop scenario. However, in the actual application scenario, there are a large number of robots in the industrial workshop, and the motion trajectories of multiple robots and the communication and computing environment are inconsistent. It is difficult to meet the further low-latency and high-reliability requirements in the actual application scenario by simply optimizing the resource allocation of all robots with a preset strategy. SUMMARY

[0005] The technical problem to be solved by the present application is to overcome the technical problem of insufficient optimization effect caused by separate optimization of the communication and computing resource allocation optimization method of the agent in the prior art, and to provide a method for joint allocation optimization of communication and computing resources of multiple agents.

[0006] The present application discloses a method for joint allocation optimization of communication and computing resources of multiple agents, comprising the following steps:

[0007] Step S1: establishing a communication and computing resource model, including the communication interruption probability of the agent, the required delay for computing, the total required delay for communication and computing, and the total energy consumption for communication and computing;

[0008] Step S2: constructing a MAPPO network, outputting the next time action value of the agent based on the MAPPO network, and the agent interacting with the environment according to the next time action value; calculating the energy consumption, communication interruption probability and delay of each agent according to the communication and computing resource model, and calculating a reward value set;

[0009] Step S3: constructing an optimization objective function of the action network and the comment network, updating the action policy and the value function of the action network and the comment network based on the optimization objective function and the observation value set.

[0010] Further, the step S1 comprises the following steps:

[0011] Step S1.1: calculating the communication interruption probability of the nth agent at t time;

[0012] Step S1.2: calculating the calculation required delay of the nth agent at t time;

[0013] Step S1.3: calculating the total calculation required delay of the nth agent at t time based on the calculation required delay;

[0014] Step S1.4: calculating the total energy consumption of the nth agent at t time for completing communication calculation.

[0015] Further, in the step S1:

[0016] The communication interruption probability of the nth agent at t time is:

[0017] Wherein, The signal-to-noise ratio at t time is Γ th The threshold value at t time is σ 2 The noise power is Pr LOS The line-of-sight probability is Pr NLOS The non-line-of-sight probability is β LOS The line-of-sight path loss is β NLOS The non-line-of-sight path loss is The communication transmission power of the agent at t time is Pt;

[0018] The calculation required delay of the nth agent at t time is:

[0019] Wherein, The task amount of the agent at t time is Tt, The communication bandwidth of the agent at t time is Bt, The number of CPU cores required for the agent to complete the task at t time is Ct, and e is the number of CPU cycles per second run by the edge server;

[0020] The total calculation required delay of the nth agent at t time for communication calculation is: total τ UL = τ cmp + τ Dl ;

[0021] Wherein, τ UL is the uplink transmission delay, τDl is the downlink transmission delay, and

[0022] The total energy consumption of the agent completing the communication calculation is: E total = E motors + E com ;

[0023] wherein E motors is the energy consumption of the motor when the agent moves, and E com is the energy consumption of the agent during the communication process.

[0024] Further, the step S2 comprises the following steps:

[0025] Step S2.1: Constructing the review network of the MAPPO network and the action network of each agent; the action network includes an action policy, the input is the observation value of the agent, and the output is the action value at the next time; the review network includes a value function, the input is the observation value set of all agents, and the output is the value function;

[0026] Step S2.2: The agent acquires the next action value and its Gaussian distribution function according to the action policy, randomly samples and outputs the action value at the next time, and then saves the action selection probability;

[0027] Step S2.3: Each agent interacts with the environment according to the action value at the next time, calculates and judges whether the speed at the next time and the position at the next time exceed the preset threshold;

[0028] Step S2.4: Calculate the energy consumption, communication interruption probability and delay of each agent according to the communication and calculation resource model, and calculate the reward value set based on the acceleration, position, energy consumption, communication interruption probability and delay of each agent.

[0029] Further, in the step S2, the observation value includes the current time slot energy consumption, the current time slot task amount, the current speed of the agent, and the current position of the agent; the action value includes the acceleration of the agent, the communication transmission power, the number of CPU cores required to complete the task, and the communication bandwidth.

[0030] Further, in the step S2, the reward is: r t = r a + r bd + r out + r delay + r energy ;

[0031] wherein r a and r bd are the constraint conditions of the acceleration of the agent and the boundary, and r outr is a reward for the communication interruption probability delay r is a reward for the total latency energy r is a reward for the energy consumption, and the calculation expression is: r delay = ξ delay ·τ total ; r energy = ξ energy ·E total ;

[0032] wherein, ξ a , ξ bd , ξ out , ξ delay , ξ energy are negative coefficients, respectively representing the proportion of the acceleration, the boundary condition, the communication interruption probability, the total latency, and the energy consumption in the total reward.

[0033] Further, the step S3 comprises the following steps:

[0034] Step S3.1: storing the observation value set, the observation value of each agent, the action value, the action selection probability, and the reward value set into the experience pool in each time slot;

[0035] Step S3.2: constructing the optimization objective function of the action network;

[0036] Step S3.3: constructing the optimization objective function of the critic network;

[0037] Step S3.4: when the data in the experience pool reaches the preset threshold, extracting random data to update the network parameters of the action network and the critic network, and calculating the optimization objective function thereof, and then updating the action policy and the value function with the updated network parameters, and then emptying the experience pool.

[0038] Further, the optimization objective function of the action network is:

[0039] wherein, θ is the network parameter of the action network, is the advantage function, ε is a hyperparameter for controlling clip, S represents the entropy of the action policy, σ is a hyperparameter for controlling the entropy coefficient, B represents the data size extracted in a single training, and n represents the number of agents. represents the difference function between the action policy at the next moment and the action policy at the current moment.

[0040] Further, the calculation expression of the difference function between the action policy at the next moment and the action policy at the current moment is:

[0041] wherein, is the action policy at the next moment, The action policy for the current time.

[0042] Further, the optimization objective function of the comment network is

[0043] Wherein, phi is the network parameter of the comment network, V φ The value function evaluated for state s i at time t, The discount reward, whose calculation expression is:

[0044] Wherein, r t is the reward at the current time; gamma t is the discount factor.

[0045] Beneficial effects: the application discloses a kind of communication and computing resource joint distribution optimization method of multi-agent, can according to current industrial workshop environment, edge server and agent parameter construct communication channel model, calculation model and motion energy consumption model. Each agent and environment interact to obtain current observation value, input the observation value obtained into action network, obtain the action parameter of next time, and calculate current reward, update neural network parameter and train current data, communication and computing resource joint distribution strategy.The application can optimize distribution method under the condition of limited communication, computing resource, realize the demand characteristics of low latency, high reliability, computing power and time, reduce resource waste. BRIEF DESCRIPTION OF DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0047] Fig. 1 is a schematic diagram of the overall flow method steps of the structure of the present application;

[0048] Fig. 2 is a schematic diagram of the MAPPO network structure of the present application;

[0049] Fig. 3 is a schematic diagram of the method flow of the present application. DETAILED DESCRIPTION

[0050] In order to make the above objectives, characteristics and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application are described in detail below in combination with the drawings. In the following description, a large number of specific details are set forth in order to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without departing from the spirit of the present application, so the present application is not limited to the specific embodiments disclosed below.

[0051] Referring to FIGS. 1 and 3, the embodiment discloses a multi-agent communication and computing resource joint allocation optimization method, comprising the following steps:

[0052] Step S1: establishing a communication and computing resource model, including the communication interruption probability of the agent (AMR), the required delay for computing, the total required delay for communication and computing, and the total energy consumption for communication and computing;

[0053] Step S2: constructing a MAPPO network, outputting the next time action value of the agent based on the MAPPO network, and the agent interacting with the environment according to the next time action value; calculating the energy consumption, communication interruption probability and delay of each agent according to the communication and computing resource model, and calculating a reward value set;

[0054] Step S3: constructing an optimization objective function of the action network and the comment network, and updating the action policy and the value function of the action network and the comment network based on the optimization objective function and the observation value set.

[0055] The multi-agent communication and computing resource joint allocation optimization method disclosed in the embodiment can construct a communication channel model, a computing model and a motion energy consumption model according to the current industrial workshop environment, edge server and agent parameters. Each agent interacts with the environment to obtain the current observation value, inputs the obtained observation value into the action network, obtains the action parameters at the next time, calculates the current reward, updates the neural network parameters and trains the current data, and finally obtains the communication and computing resource joint allocation strategy. The present application can optimize the allocation method under the condition of limited communication and computing resources, realize the demand characteristics of low delay, high reliability, computing power and time, and reduce resource waste.

[0056] In the embodiment, the number of agents is N, the mass is m, the maximum working speed is v max , the maximum transmission power of communication is p max , the base station position coordinates (x bs , y bs ) and the actual available computing core number C total of the edge server.

[0057] Specifically, the step S1 comprises the following steps:

[0058] Step S1.1: Calculate the communication interruption probability of the nth agent at t time;

[0059] The communication interruption probability of the nth agent at t time is:

[0060] wherein, is the signal-to-noise ratio at t time, Γ th is the threshold value at t time, σ 2 is the noise power, Pr LOS is the line-of-sight probability, Pr NLOS is the non-line-of-sight probability, β LOS is the line-of-sight path loss, β NLOS is the non-line-of-sight path loss, is the communication transmission power of the agent at t time;

[0061] Step S1.2: Calculate the calculation required delay of the nth agent at t time;

[0062] The calculation required delay of the nth agent at t time is:

[0063] wherein, is the task amount of the agent at t time, is the communication bandwidth of the agent at t time, is the number of CPU cores required to complete the task of the agent at t time, and e is the number of CPU cycles per second run by the edge server;

[0064] Step S1.3: Calculate the total communication calculation required delay of the nth agent at t time based on the calculation required delay;

[0065] The total communication calculation required delay of the nth agent at t time is: total = τ UL + τ cmp + τ Dl ;

[0066] In this embodiment, the value of τ Dl is generally small and can be ignored.

[0067] wherein, τ UL is the uplink transmission delay, τ Dl is the downlink transmission delay, and has:

[0068] Step S1.4: Calculate the total communication calculation energy consumption of the nth agent at t time.

[0069] The total communication calculation energy consumption of the agent is: E total = E motors+E com ;

[0070] wherein, E motors is the energy consumption of the motor when the agent moves, E com is the energy consumption in the communication process of the agent, and has:

[0071] wherein, is the communication power of the nth agent at time t; in this embodiment, the energy consumption of the motor when the agent moves is obtained by the agent's own energy consumption module.

[0072] Specifically, referring to FIG. 2, the step S2 includes the following steps:

[0073] Step S2.1: constructing a comment network (Critic) of the MAPPO network and an action network (Actor) of each agent; the action network includes an action policy The input is the observation value of the agent The output is the action value a of the next moment t ; the comment network includes a value function, the input is the observation value set S t of all agents, and the output is the value function V φ ;

[0074] The observation value includes the current time slot energy consumption, the current time slot task amount, the current speed of the agent, and the current position of the agent; the action value includes the acceleration of the agent, the communication transmission power, the number of CPU cores required to complete the task, and the communication bandwidth.

[0075] Step S2.2: the agent obtains the next action value and its Gaussian distribution function according to the action policy, randomly samples in it and outputs the action value at the next moment, and then saves the action selection probability;

[0076] In this embodiment, the action network obtains the next action value output by the policy network according to the current moment action policy , adds Gaussian noise to the output action value, so that the action value distribution is subject to Gaussian distribution with the network output action value as the mean value and σ 2 as the standard deviation. On this basis, an action value is saved

[0077] Step S2.3: each agent interacts with the environment according to the action value at the next moment, calculates and judges whether the speed at the next moment and the position at the next moment exceed the preset threshold value;

[0078] In this embodiment, the agent obtains the action value Interact with the environment, calculate and judge whether the current speed exceeds the limit, if greater than the maximum speed, set the direction speed to v max , if less than the minimum value, set it to the minimum value v min ; calculate and judge whether the current position exceeds the specified area of the map, if it exceeds the specified area, set it to the nearest limit value;

[0079] Step S2.4: Calculate the energy consumption, communication interruption probability and delay of each agent according to the communication and calculation resource model, and calculate the reward value set based on the acceleration, position, energy consumption, communication interruption probability, delay of each agent: r t = r a + r bd + r out + r delay + r energy ;

[0080] Wherein, r a and r bd are the constraint conditions of the agent acceleration and the boundary, r out is the reward of the communication interruption probability, r delay is the reward represented by the total delay, r energy is the reward of energy consumption, and its calculation expression is: r delay = ξ delay ·τ total ; r energy = ξ energy ·E total ;

[0081] Wherein, ξ a , ξ bd , ξ out , ξ delay , ξ energy are negative coefficients, respectively representing the proportion of the acceleration, boundary condition, communication interruption probability, total delay and energy consumption of the agent in the total reward; by presetting the values of each negative coefficient, the MAPPO network can iterate towards the optimization goal of expected communication interruption probability, total delay and energy consumption.

[0082] In this embodiment, is a binary acceleration constraint condition flag, when the absolute value of the acceleration of the agent exceeds the maximum acceleration a max , otherwise, is a binary boundary constraint condition, when the agent exceeds the boundary, otherwise, ξ a , ξ bdrespectively represent the proportion of acceleration and boundary condition of the agent in the total reward. out respectively represent the proportion of the interrupt probability in the total reward. delay respectively represent the proportion of the time delay in the total reward. energy respectively represent the proportion of the energy consumption in the total reward. The negative coefficients are used to adjust the proportion of each sub-reward, and the preset values are in the range of (0, 1).

[0083] Specifically, the step S3 comprises the following steps:

[0084] Step S3.1: storing the observation value set, the observation value of each agent, the action value, the action selection probability and the reward value set into an experience pool in each time slot;

[0085] Step S3.2: constructing an optimization objective function of the action network;

[0086] In the embodiment, the optimization objective function of the action network is:

[0087] wherein, θ is the network parameter of the action network, is an advantage function, in the embodiment, the advantage function is calculated by a generalized advantage estimation (GAE) method, ε is a hyperparameter for controlling clip, S represents the entropy of the action policy, in the embodiment, the value is 0.2, σ is a hyperparameter for controlling the entropy coefficient, B represents the data size extracted in a single training, and n represents the number of agents. represents a difference function between the action policy at the next moment and the action policy at the current moment.

[0088] In the embodiment, the calculation expression of the difference function between the action policy at the next moment and the action policy at the current moment is:

[0089] wherein, is the action policy at the next moment, is the action policy at the current moment.

[0090] Step S3.3: constructing an optimization objective function of the critic network;

[0091] The optimization objective function of the critic network is

[0092] wherein, φ is the network parameter of the critic network, V φ is the value function evaluated at the state s i , is a discounted reward, and the calculation expression is:

[0093] wherein, r tRtis the reward at the current time; γ t γ is a discount factor, used to balance the weight between the current reward and the future reward, and the value range is (0, 1).

[0094] Step S3.4: When the data in the experience pool reaches the preset threshold, the network parameters of the action network and the comment network are updated, and the optimization objective function is calculated. The action policy and the value function are updated with the updated network parameters. Then the experience pool is emptied.

[0095] In the embodiment, the optimization objective function of the action network is constructed. The improvement degree of the next moment and the current moment strategy is compared by judging the difference between the next moment and the current moment. When the difference function r is greater than 1, it means that the probability of the next moment action strategy under the state s is higher than the current moment action strategy, that is, the next moment action strategy considers that the action a is a better choice. Then the advantage function is calculated to make the network update towards the trend of the advantage function increasing.

[0096] The optimization objective function of the comment network is constructed. The difference between the value function and the discounted reward is judged to make the estimated value greater than the actual value, that is, to maximize the estimated future return. The network is trained in this way to make the network update towards the trend of the maximum return.

[0097] In the embodiment, the above steps S2-S3 are repeated. When the number of iterations reaches the preset iteration number, it is judged whether the iteration effect reaches the expected effect. If the reward converges, it is considered that the expected effect is reached, and the iteration of the optimization of the joint allocation of communication and computing resources is ended. Otherwise, the iteration number is cleared and the iteration is restarted.

[0098] In the embodiment, the action network iterates towards the direction of the larger advantage function in the above optimization process, that is, the intelligent agent action that gives a higher reward. Correspondingly, such actions in the comment network will also tend to a higher future return. Based on this, the policy is updated during the iteration process. Finally, the difference between the new and old policies gradually decreases, the reward changes are not volatile, and the reward converges.

[0099] The technical features of the above embodiments can be combined arbitrarily. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described. However, as long as the combination of these technical features does not exist contradictory, it should be considered as the scope of the present disclosure.

[0100] The above embodiments only express several implementation ways of the present application, and the description is relatively specific and detailed, but it should not be understood as a limitation to the patent scope of the application. It should be pointed out that for ordinary skilled persons in the art, several modifications and improvements can be made without departing from the concept of the present application, which all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.

Claims

1. A method for joint allocation and optimization of communication and computing resources among multiple agents, characterized in that, The method comprises the following steps: Step S1: establishing a communication and computing resource model, including the communication interruption probability of the agent, the computing required delay, the total communication and computing required delay, and the total communication and computing energy consumption; Step S2: constructing a MAPPO network, outputting the next time action value of the agent based on the MAPPO network, and the agent interacting with the environment according to the next time action value; calculating the energy consumption, communication interruption probability and delay of each agent according to the communication and computing resource model, and calculating a reward value set; Step S3: constructing an optimization objective function of the action network and the comment network, and updating the action strategy and the value function of the action network and the comment network based on the optimization objective function and the observation value set. 2.The method of claim 1, wherein, The step S1 comprises the following steps: Step S1.1: calculating the communication interruption probability of the nth agent at t time; Step S1.2: calculating the computing required delay of the nth agent at t time; Step S1.3: calculating the total communication and computing required delay of the nth agent at t time based on the computing required delay; Step S1.4: calculating the total communication and computing energy consumption of the nth agent at t time.

3. The method of claim 2, wherein, In the step S1: The nth agent's communication interruption probability at time t is: wherein Snr(t) is the signal to noise ratio at time t th Th(t) is the threshold at time t 2 Pn is the noise power LOS Pr is the line of sight probability NLOS β is the non line of sight probability LOS β is the line of sight path loss NLOS β is the non line of sight path loss is the communication transmission power of the agent at t time; The computing required latency of the nth agent at time t is: wherein the task amount of the agent at time t, a communication bandwidth of the agent at time t, is the number of CPU cores required by the agent to complete the task at t time, and e is the number of CPU cycles per second of the edge server; The total communication and computing required delay of the nth agent at t time is: τ total = τ UL + τ cmp + τ Dl ; where τ UL is the uplink transmission delay, τ Dl is the downlink transmission delay, and The total communication and computing energy consumption of the agent is: E total = E motors + E com ; wherein E motors is the energy consumption of the motor when the agent is moving, E com is the energy consumption in the communication process of the agent.

4. The method of claim 1, wherein, The step S2 comprises the following steps: Step S2.1: constructing the comment network of the MAPPO network and the action network of each agent; the action network comprises an action strategy, the input is the observation value of the agent, and the output is the next time action value; the comment network comprises a value function, the input is the observation value set of all agents, and the output is the value function; Step S2.2: the agent acquires the next step action value and its Gaussian distribution function according to the action strategy, randomly samples in the Gaussian distribution function and outputs the next time action value, and then saves the action selection probability; Step S2.3: each agent interacts with the environment according to the next time action value, calculates and judges whether the next time speed and the next time position exceed the preset threshold; Step S2.4: calculating the energy consumption, communication interruption probability and delay of each agent according to the communication and computing resource model, and calculating a reward value set based on the acceleration, position, energy consumption, communication interruption probability and delay of each agent.

5. The method of claim 4, wherein, In the step S2, the observation value comprises the current time slot energy consumption, the current time slot task quantity, the current speed of the agent and the current position of the agent; and the action value comprises the acceleration of the agent, the communication transmission power, the number of CPU cores required to complete the task and the communication bandwidth.

6. The method of claim 4, wherein, The step S2, the reward is: t = r a + r bd + r out + r delay + r energy ; where r a and r bd are the constraints of the agent acceleration and the boundary, r out is the reward of the communication interruption probability, r delay is the reward represented by the total delay, and r energy is the reward of the energy consumption, and its calculation expression is: r delay = ξ delay ·τ total ; r energy = ξ energy · E total ; where, ξ a , ξ bd , ξ out , ξ delay , ξ energy are negative coefficients, respectively representing the acceleration of the agent, the boundary condition, the communication interruption probability, the total delay, and the proportion of energy consumption in the total reward.

7. The method of claim 1, wherein, The step S3 comprises the following steps: Step S3.1: storing the observation value set, the observation value of each agent, the action value, the action selection probability and the reward value set into an experience pool at each time slot; Step S3.2: constructing an optimization objective function of the action network; Step S3.3: constructing an optimization objective function of the comment network. Step S3.4: When the data in the experience pool reaches the preset threshold, extract the network parameters of the random data update action network and the comment network, calculate the optimization objective function, and update the action policy and value function with the updated network parameters; then empty the experience pool.

8. The method of claim 7, wherein, The optimization objective function of the action network is: wherein θ is a network parameter of the action network, is the advantage function, ε is a hyperparameter used to control the clip, S represents the entropy of the action policy, σ is a hyperparameter used to control the entropy coefficient, B represents the size of the data extracted for a single training, and n represents the number of agents, represents the difference function between the action policy at the next moment and the current moment.

9. The method of claim 8, wherein, The next time and the current time action policy difference function calculation expression is: wherein action policy for the next time instance, is the action policy at the current moment. is the action policy at the next moment.

10. The method of claim 7, wherein, The optimization objective function of the review network is where φ is the network parameter of the review network, V φ is the state at time s i the value function being evaluated, For the discount reward, the computational expression is: where r t is the reward at the current time; γ t is the discount factor.

Images