Multi-agent dynamic decision-making method based on reinforcement learning

Abstract

The invention relates to a multi-agent dynamic decision-making method based on reinforcement learning, which is applied to an edge computing network comprising an edge server and a plurality of pieces of user equipment, and comprises the following steps of: taking each piece of user equipment as an agent; abstracting a task unloading process in the edge computing network into a multi-agent part observable Markov decision process; defining the task unloading benefit of any user equipment to be obtained by subtracting the edge unloading cost of the user equipment from the edge calculation benefit obtained by the user equipment; and solving the multi-agent part observable Markov decision process by using a DDPG algorithm combined with an attention mechanism, and obtaining an optimal offload decision for maximizing the task offload benefit of each user equipment under the condition that each user equipment reaches Nash equilibrium. According to the invention, the calculation demand of the user equipment in the Internet of Things and the dynamic change of the Internet of Things environment are comprehensively considered, and a flexible, efficient and real-time calculation unloading decision is provided for each user equipment.

Description

Technical Field

[0001] The present invention relates to the field of wireless communication technology, and in particular to a multi-agent dynamic decision-making method based on reinforcement learning. Background Art

[0002] The widespread use of Internet of Things (IoT) technologies has revolutionized the way we interact with the physical world. With the explosive growth in the number of IoT devices, the data processing tasks involved have become more complex and diverse. These tasks include cutting-edge applications such as video compression, augmented reality (AR) games, and autonomous driving, which are growing at an unprecedented rate in terms of data processing requirements. However, traditional mobile user equipment (UEs) are often subject to hardware limitations such as limited computing power and battery life, which makes it difficult for them to efficiently process such large amounts of data. To address this problem, edge computing technology has emerged and has gradually become an important support in the field of IoT. The core idea of ​​edge computing is to offload part of the computing tasks from UEs to edge servers (ESs) that are closer to the data source for processing, thereby effectively alleviating the computing burden of UEs, shortening the latency of data processing, and improving the user experience. In this way, user devices can use nearby computing resources to process complex tasks when local resources are insufficient.

[0003] However, with the increasing number of IoT devices and applications, it is increasingly common for multiple UEs to request computation offload at the same time, which brings new challenges. The computing resources of edge servers are limited. When a large number of UEs request at the same time, resource competition will be inevitable, resulting in reduced task processing efficiency and even task delays that cannot meet application requirements. In addition, since edge computing environments are usually dynamic and highly distributed, traditional centralized resource allocation methods have many shortcomings in such environments, such as high communication overhead, poor system scalability, and potential risks of user privacy leakage.

[0004] Existing research mainly focuses on how to optimize computing offloading and resource allocation in static or predictable IoT environments. Typically, these methods use centralized control methods, which may be effective in small-scale application scenarios, but are obviously inadequate when faced with dynamically changing and large-scale IoT environments. For example, the public document CN114863683A proposes a computing offloading strategy based on multi-objective optimization, and the public document CN114640966A uses a prediction method based on machine learning to make offloading decisions. Although these methods can provide certain optimization effects in specific scenarios, they generally rely on complete information about the environment and centralized computing models. In dynamic environments with incomplete information sharing, their effectiveness is often limited.

[0005] Given the complexity and dynamism of the IoT environment, traditional methods are unable to cope with partial information sharing and dynamic changes. Therefore, a new computation offloading strategy is urgently needed, which should have distributed characteristics, be able to dynamically adapt to environmental changes, and make efficient decisions without relying on global information. Summary of the invention

[0006] The technical problem to be solved by the present invention is to provide a multi-agent dynamic decision-making method based on reinforcement learning, which can comprehensively consider the computing needs of user devices in the Internet of Things and the dynamic changes of the Internet of Things environment, and provide flexible, efficient and real-time computing offloading decisions for each user device.

[0007] The technical solution adopted by the present invention to solve the technical problem is: to provide a multi-agent dynamic decision-making method based on reinforcement learning, which is applied to an edge computing network including an edge server and multiple user devices, including the following steps:

[0008] Taking each of the user devices as an agent, the task offloading process in the edge computing network is abstracted into a multi-agent partially observable Markov decision process;

[0009] The task offloading benefit of any user device is defined as the edge computing benefit obtained by the user device minus the edge offloading cost;

[0010] The DDPG algorithm combined with the attention mechanism is used to solve the multi-agent partially observable Markov decision process, and the optimal offloading decision is obtained so that each user device can achieve the maximum benefit of its own task offloading under the Nash equilibrium.

[0011] Furthermore, the task offloading benefit of the user equipment is expressed as

[0012]

[0013] in, is the task offloading benefit of user equipment i in time slot t, represents the local computing tasks of all user devices except user device i, and are the total amount of computing tasks of user equipment i and user equipment m in time slot t, and are the local computing tasks of user equipment i and user equipment m in time slot t, respectively, i is the computing resource share of the edge server obtained by user device i, F is the available computing resources of the edge server, Offload the cost at the edge of user equipment i at time slot t.

[0014] Furthermore, the edge offloading cost of the user equipment is the sum of its local computing energy consumption and its task transmission energy consumption.

[0015] Furthermore, the edge offloading cost of the user equipment is expressed as

[0016]

[0017] in, is the edge unloading cost of user equipment i in time slot t, k is the energy consumption factor, is the computing power of user equipment i in time slot t, is the local computing task of user device i in time slot t, is the total amount of computing tasks of user equipment i in time slot t, p i is the transmission power of user equipment i, is the transmission bandwidth of user equipment i in time slot t.

[0018] Furthermore, the optimal unloading decision satisfies the following constraints:

[0019] The amount of offloaded tasks of each of the user equipment does not exceed the total amount of its computing tasks;

[0020] The task execution delay of each of the user equipment is less than a set value.

[0021] Furthermore, the task execution delay of the user equipment is a larger value of its local calculation delay and its offloading calculation delay.

[0022] Furthermore, the offloading calculation delay of the user equipment is the sum of its offloading task transmission delay and the cloud computing delay of the offloading task on the edge server.

[0023] Furthermore, the offloading calculation delay of the mobile user equipment is expressed as

[0024]

[0025] in, and are the total amount of computing tasks of user equipment i and user equipment m in time slot t, and are the local computing tasks of user equipment i and user equipment m in time slot t, is the transmission bandwidth of user device i in time slot t, and F is the available computing resources of the edge server.

[0026] Furthermore, in the multi-agent partially observable Markov decision process, each of the user devices can only observe the historical unloading decisions and its own properties of other user devices in the first K time slots.

[0027] Furthermore, the DDPG algorithm combined with the attention mechanism includes:

[0028] The actor processes the observation information to generate unloading actions, and generates unloading decisions that can maximize the state value function based on the reward and state transition of the environment feedback based on the attention mechanism;

[0029] The critic performs feature extraction based on the attention mechanism on the observed information, and then estimates the state value function of each user device when the uninstallation decision is given according to the extracted feature information.

[0030] Beneficial Effects

[0031] Due to the adoption of the above technical solution, the present invention has the following advantages and positive effects compared with the prior art: the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 is a flow chart of an embodiment of the present invention;

[0033] Figure 2 is a diagram of a user equipment adaptive task offloading system according to an embodiment of the present invention;

[0034] Figure 3 It is a flow chart of the A-MADDPG algorithm of an implementation mode of the present invention. DETAILED DESCRIPTION

[0035] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the appended claims of the application equally.

[0036] An embodiment of the present invention relates to an intelligent decision-making algorithm based on multi-agent reinforcement learning, which is based on a distributed computing offloading framework for dynamic Internet of Things networks. The framework allows each UE to autonomously and in real time determine its optimal offloading strategy under resource-constrained conditions, thereby improving user experience and overall system performance.

[0037] like Figure 1 As shown, the specific technical route includes the following steps.

[0038] S1, establish system task offloading framework

[0039] Consider a dynamic IoT environment where we have a number of I mobile user equipments (UEs), represented as a set These devices are all located within the service range of the edge server (ES). In each time slot t, each device generates a computing task. i The size of the task generated in time slot t is MB. Due to the limited computing power of UEs, part of the task is offloaded to ES. i The size of the computational task processed by the local resources is in The task part that is offloaded to ES is We assume that each UE is forward-looking and aims to make long-term decisions, offloading data volume to maximize its processing benefit in this dynamic network environment where private information about other UEs is unknown.

[0040] S2, build a computational offloading model

[0041] like Figure 2 As shown in Figure 2, latency and energy consumption are key factors to be considered in the task offloading process. As mentioned above, the task to be processed is sent by the UE in time slot t. i The local computing resources and ES resources are used together for processing. Next, we analyze the delay and energy consumption caused by these two processing methods, and combine them to derive the UE i The total delay and energy consumption.

[0042] S2.1, Local Computing

[0043] Task delay: The delay incurred by this process includes local processing time and can be expressed as in Indicates UE i computing power.

[0044] Energy consumption: Accordingly, the energy consumption generated by local computing can be calculated as where κ is the energy dissipation factor.

[0045] S2.2, Edge Server Computing

[0046] Task delay: This delay is mainly determined by task offloading and edge server processing time. The offloading delay is defined as in It is UE i The total amount of data offloaded from UEs to ES is Let F be the available computing resources of ES. ES allocates computing resources according to the amount of data offloaded by each UE. i The computing resource share obtained is Therefore, UE i The processing delay at ES is UEi The total delay is

[0047] Energy consumption: UE i The energy consumption of is mainly caused by task transmission and is calculated as where p i It is UE i The transmission power in time slot t.

[0048] S2.3, Problem Statement

[0049] Taking into account the latency and energy consumption from local computing and ES computing, we can obtain the overall performance indicators of task computing.

[0050] Total task latency: Given that local processing and offload computation are performed simultaneously, UE i The combined delay of is determined by the longer delay in these two stages. Therefore, in time slot t, UE i Execution delay It can be defined as

[0051]

[0052] The first part is the transmission and processing time on the ES, and the second part is the UE i Local processing time.

[0053] Total cost: In time slot t, UE i The energy consumption of is composed of the energy consumption of the local and ES stages, expressed as

[0054]

[0055] Benefit function: The benefit function of each UE is designed as follows

[0056]

[0057] in, In addition to UE i The decision vector of all UEs outside i is a weight in the range [0,1], indicating that UE i The computing resource share obtained on ES, UE i You can adjust α according to your specific needs i The first term of the benefit function represents the computational benefit obtained from ES, and the second term represents the associated overhead, covering the cost of transmission and local computation.

[0058] Objectives and constraints: The main objective of each UE is to select its offloading strategy within time slot t. To maximize its benefits. The corresponding optimization objectives and constraints can be expressed as:

[0059]

[0060] Among them, C1 ensures that UE i The amount of data offloaded will not exceed its total task volume, and C2 ensures that UE i The task completion time must be within the predetermined maximum delay.

[0061] Game Description: The equations presented show the competition between UEs for computing and bandwidth resources in pursuit of their independent offloading benefits. Game theory provides a low-complexity method to analyze the relationship between these UEs pursuing individual interests. Under this game theory framework, UEs can independently find the best offloading strategy and reach a Nash equilibrium. Using this thinking, we constructed a game model that captures the competition for computing and bandwidth resources when UEs maximize the benefits of offloading. Specifically, the game can be defined as:

[0062]

[0063] in

[0064] is the set of players (i.e. UEs);

[0065] is the set of uninstall strategies for all players, where Indicates that the ES is unloaded at time slot t.

[0066] The proportion of data loaded, is a set of time slots;

[0067] is the set of unloading benefits of all players at time slot t, where is the UE at time slot t i The offloading benefit obtained is as described in the utility function.

[0068] This game The Nash equilibrium is defined as follows:

[0069] Strategy Collection It is a dynamic computation offloading game If a NE at the equilibrium point S t，* -x t，* Under this condition, any UE cannot further improve its benefits by adjusting its strategy alone, that is:

[0070]

[0071] S3, building a multi-agent POMDP

[0072] We play for the long term A multi-agent POMDP is constructed, denoted as in Define the state space, describes the action space, is the set of state transition probability functions, represents the reward space, refers to the observation space, is the observed transition set. In each time slot t, given the current state Each UE determines its best offloading strategy based on the current observation. Then, based on the benefit function, the UE obtains an immediate reward and enters the next state based on the state transition probability function. Subsequently, the observation space is updated based on the observation transfer function. Below we will introduce the various components of the POMDP model in detail.

[0073] State Space: The state space is partially known and is defined as Including UE i The data size of the computational task generated at time slot t UE i Current computing power

[0074] Action Space: The action space is represented as In this game, the action is the proportion of data unloaded to ES in time slot t.

[0075] State transition probability function set: The state transition probability function set is expressed as It describes the possibility of transitioning from one state to another via a specific operation, expressed as

[0076] Reward space: Reward space is used Indicates that and After all UEs take action at time slot t, each UE will be rewarded according to the benefit function and enter the next round of the game.

[0077] Observation space: The observation space of each UE is composed of Definition. In this space, Defined as It means that at the beginning of time slot t, each UE can only observe the past offloading strategies of other UEs in the first K time slots and its own attributes (bandwidth, weight, offloading strategy, etc.). It is worth noting that when t≤K, is randomly generated.

[0078] Observation transfer: It should be noted that the observation transfer process is random. After all UEs take action in time slot t, UE i Observation through

[0079] Long-term reward: Consider τ, which is expressed as This means that UE i A complete game process, which means that UE i From initial observation Start making uninstall decisions Get rewards Then move on to the next observation This cycle continues until the game terminates. Given that UEs are forward-looking, they aim to adopt strategies to maximize their long-term rewards, defined as the cumulative discounted rewards starting from time slot t, expressed as

[0080]

[0081] in is the discounted expected future reward with a discount factor γ∈[0,1].

[0082] Objective: To achieve the above objectives, we need to determine the strategy that can maximize the long-term utility of UEs. i The computation offloading strategy is expressed as Next, UE i The goal of strategy optimization is to find the optimal parameter θ that maximizes the expected cumulative return. i ,Right now Mathematically, this can be expressed as:

[0083]

[0084] in, It is UE i The initial observation probability distribution of .

[0085] S4, Design of A-MADDPG algorithm for solving multi-agent POMDP

[0086] S4.1, Optimization strategy of A-MADDPG algorithm

[0087] A control module is designed based on the multi-agent DDPG framework to generate the optimal offloading decision for each UE. This control module consists of an actor network, an actor update unit, a critic network, an attention mechanism unit, a critic update unit, and an experience replay buffer. The critic network includes a feature extraction subnetwork and a state value function estimator combined with an attention mechanism, while the actor network contains a feature extraction subnetwork and a strategy generation network. Specifically, the actor network generates offloading actions by processing observation information, and then guides the UE's decision based on the rewards and state transitions fed back by the environment.

[0088] In the critic network, the observation information is first mapped to a feature vector, and then the built-in attention mechanism module selects the feature information most relevant to the current decision to accurately estimate the state value function. In each training, the actor and critic networks adjust their parameters according to their respective gradient update rules.

[0089] The attention mechanism module plays a vital role in the critic network. By intelligently screening and weighing the relationship between current and historical observations, the attention mechanism can focus on analyzing the information that is most helpful for the current decision and enhance the estimation accuracy of the state value function. Compared with traditional recurrent neural networks, the attention mechanism can more effectively capture the key features of the observation information, thereby providing more accurate decision support in partially observable environments.

[0090] At the end of each time slot, the observations, unloading decisions, rewards, and observations for the next time slot are stored in the experience replay buffer for subsequent training. The specific algorithm flow is as follows: Figure 3 shown.

[0091] S4.2, Update mechanism of A-MADDPG algorithm

[0092] 1. State value function estimation update: The goal of the critic network is to estimate the state value function of each UE given the current strategy Where o is the observation of the agent and a is the action taken by the agent. After combining the attention mechanism, the estimation of the state value function can be expressed as:

[0093]

[0094] Among them, ψ Q and ψ Q′ are the parameters of the critic network and the target critic network, respectively, and γ is the discount factor. The attention mechanism enhances the representation of o by screening the most relevant observation information, thereby improving the representation of Q π The estimation accuracy.

[0095] 2. Policy network update: The goal of the actor network is to find the optimal policy π(o) so that the action in each state can maximize the state value function Q π (o, a). In A-MADDPG, the parameter update of the actor network is performed through the following loss function:

[0096]

[0097] Among them, θ π is the parameter of the actor network. The introduction of the attention mechanism in the policy network helps the agent to more accurately select appropriate actions in a partially observable environment, thereby optimizing the strategy. In each iteration, the gradient update formula of the critic network is:

[0098]

[0099] This gradient is used to update the parameters of the critic network to minimize the estimation error. The gradient update formula of the actor network is:

[0100]

[0101] This gradient is used to update the parameters of the actor network to enhance the performance of the policy, enabling the agent to make decisions more effectively in partially observable environments.

[0102] 3. Attention mechanism update: The attention mechanism is updated by calculating the correlation between the current observation and the historical observation of the agent. Assume o t is the current observation, h t is the historical observation feature extracted by the attention mechanism, then the update of the attention weight can be expressed as:

[0103]

[0104] By calculating the attention weight β t，i , we can get the weighted historical information h t , used for the next decision:

[0105]

[0106] This step enables each UE to intelligently focus on the historical information that is most important for current decision making.

[0107] 4. Experience replay buffer update: After each training time slot, the agent’s observation o t , uninstall decision a t 、Instant Rewards t and the observation o of the next time slot t+1Will be stored in the experience replay buffer for subsequent network training and strategy optimization:

[0108] Buffer←{(o t , a t , r t , o t+1 )}.

[0109] This updating method enables the model to continuously learn from historical experience and gradually optimize the unloading strategy.

[0110] In summary, in view of the obvious deficiency of how to effectively handle task offloading in the dynamic IoT edge computing in the existing technology in a partial information sharing environment, this invention proposes an intelligent decision-making algorithm based on deep reinforcement learning, which solves the following technical problems:

[0111] 1) Task offloading in a partial information sharing environment: In the IoT edge computing environment, since communication between devices is limited and only partial information can be shared, how to effectively offload tasks in this environment becomes an important challenge. The algorithm framework proposed in this paper can make full use of limited information resources and realize efficient task offloading decisions.

[0112] 2) Independent decision-making capability in a dynamic network environment: The IoT network is highly dynamic. The status of user devices, available resources, and network conditions are constantly changing. How to provide each user device with a method that can independently make the optimal computing offloading decision in this environment is another technical challenge that needs to be solved urgently. The present invention designs a multi-agent algorithm based on deep reinforcement learning, so that each user device can make autonomous decisions in a dynamically changing environment, thereby achieving the optimization of task offloading.

[0113] Specifically, the present invention adopts the A-MADDPG algorithm based on deep reinforcement learning, which combines the attention mechanism and the deep deterministic policy gradient (DDPG) algorithm to enable each user device to independently make the optimal computational offloading decision based on its historical information and current state. In addition, the present invention also introduces the multi-agent partially observable Markov decision process (POMDP) ​​into the modeling process of task offloading. Through this method, the limitations of partial information sharing are more effectively handled, and the adaptability and robustness of the algorithm in complex environments are enhanced.

[0114] Experiments show that the method of the present invention is significantly superior to existing methods in terms of computational offloading utility and convergence speed, and especially demonstrates excellent adaptability and efficiency in a dynamic IoT environment.

Claims

1. A multi-agent dynamic decision-making method based on reinforcement learning, applied to an edge computing network including an edge server and multiple user devices, characterized in that: The following steps are involved: Taking each of the user devices as an agent, the task offloading process in the edge computing network is abstracted into a multi-agent partially observable Markov decision process; The task offloading benefit of any user device is defined as the edge computing benefit obtained by it minus its edge offloading cost; the DDPG algorithm combined with the attention mechanism is used to solve the multi-agent partially observable Markov decision process to obtain the optimal offloading decision that maximizes the task offloading benefit of each user device under Nash equilibrium.

2. The method according to claim 1, characterized in that The task offloading benefit of the user equipment is expressed as in, is the task offloading benefit of user equipment i in time slot t, represents the local computing tasks of all user devices except user device i, and are the total amount of computing tasks of user equipment i and user equipment m in time slot t, and are the local computing tasks of user equipment i and user equipment m in time slot t, respectively, i is the computing resource share of the edge server obtained by user device i, F is the available computing resources of the edge server, Offload the cost at the edge of user equipment i at time slot t.

3. The method according to claim 1, characterized in that The edge offloading cost of the user equipment is the sum of its local computing energy consumption and its task transmission energy consumption.

4. The method according to claim 3, characterized in that: The edge offloading cost of the user equipment is expressed as in, is the edge offloading cost of user equipment i in time slot t, κ is the energy consumption factor, is the computing power of user equipment i in time slot t, is the local computing task of user device i in time slot t, is the total amount of computing tasks of user equipment i in time slot t, p i is the transmission power of user equipment i, is the transmission bandwidth of user equipment i in time slot t.

5. The method according to claim 1, characterized in that The optimal unloading decision satisfies the following constraints: The amount of offloaded tasks of each of the user equipment does not exceed the total amount of its computing tasks; The task execution delay of each of the user equipment is less than a set value.

6. The method according to claim 5, characterized in that The task execution delay of the user equipment is the larger value of its local calculation delay and its offloading calculation delay.

7. The method according to claim 6, characterized in that The offloading calculation delay of the user equipment is the sum of its offloading task transmission delay and the cloud computing delay of the offloading task on the edge server.

8. The method according to claim 7, characterized in that The offloading calculation delay of the mobile user equipment is expressed as in, and are the total amount of computing tasks of user equipment i and user equipment m in time slot t, and are the local computing tasks of user equipment i and user equipment m in time slot t, is the transmission bandwidth of user device i in time slot t, and F is the available computing resources of the edge server.

9. The method according to claim 1, characterized in that: In the multi-agent partially observable Markov decision process, each of the user devices can only observe the historical unloading decisions and its own properties of other user devices in the first K time slots.

10. The method according to claim 1, characterized in that The DDPG algorithm combined with the attention mechanism includes: The actor processes the observation information to generate unloading actions, and generates unloading decisions that can maximize the state value function based on the reward and state transition of the environment feedback based on the attention mechanism; The critic performs feature extraction based on the attention mechanism on the observed information, and then estimates the state value function of each user device when the uninstallation decision is given according to the extracted feature information.

Images