Calculation unloading optimization strategy based on multi-agent deep reinforcement learning

Abstract

The invention provides a calculation unloading optimization strategy based on multi-agent deep reinforcement learning, and relates to the technical field of intelligent algorithms. The core content of the calculation unloading optimization strategy based on multi-agent deep reinforcement learning is a multi-agent deep reinforcement learning algorithm designed for an MEC multi-user calculation unloading scene. According to the method, a task Q network structure for each task is constructed by utilizing a value decomposition technology, and a corresponding task Q value is generated on the basis of considering all task relationships of the intelligent agent. The task Q values can be further aggregated to obtain the Q value of the agent, so that the problem of high time and space complexity faced by the multi-task agent in a complex parameterized action space is effectively relieved. In addition, a self-attention mechanism is utilized in the hybrid network, and corresponding weights are dynamically distributed according to the characteristics of different users. Therefore, more accurate interaction representation among the intelligent agents is realized, and the flexibility and effectiveness of overall decision making are improved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent algorithm technology, and specifically to a computational offloading optimization strategy based on multi-agent deep reinforcement learning. Background Art

[0002] In recent years, Mobile Edge Computing (MEC) has been widely used in many key scenarios of the Internet of Things, especially in the fields of drone systems, Internet of Vehicles, and Medical Internet of Things, and has achieved remarkable progress and results. MEC provides high-quality service support for mobile users by offloading computationally intensive tasks to edge servers close to users and reasonably allocating computing and communication resources for these tasks. By processing at the edge of the network, MEC can respond to user needs in real time and ensure the timeliness and accuracy of data. With the help of MEC, not only can the computing load of mobile devices be significantly reduced, but also the data transmission delay can be effectively reduced, thereby greatly improving the user experience. In addition, the resource optimization strategy of MEC can be dynamically adjusted according to the current network status and user needs, further improving the flexibility and adaptability of the system. This allows mobile users to enjoy a smoother and more efficient service experience in various environments, whether it is complex urban traffic or remote medical scenarios.

[0003] With the continuous development of artificial intelligence technology, deep reinforcement learning has become the core means to realize MEC computing offloading algorithms. In this field, the Deep Q Network (DQN) algorithm can effectively solve the offloading decision problem in discrete action space by combining the perception ability of deep learning and the decision-making ability of reinforcement learning. For the continuous action space in resource allocation, the Deep Deterministic Policy Gradient (DDPG) algorithm shows significant advantages based on the Actor-Critic architecture. By generating the optimal strategy through the Actor network and evaluating the strategy effect by the Critic network, DDPG can dynamically adapt to the complex resource allocation requirements in MEC scenarios, ensuring that the system can achieve efficient and flexible resource management in complex environments.

[0004] There are two common solutions to the problem of parameterized action space that contains both discrete and continuous actions in computational offloading strategies: one is to achieve continuity of actions by finely dividing the discrete action space of the DQN algorithm; the other is to discretize continuous actions based on the parameterized DDPG algorithm to optimize the parameterized action space. However, these two solutions have certain limitations in algorithm time complexity, space complexity, and algorithm performance. In order to deal with the problem of parameterized action space, Xiong et al. from Tencent AI Lab proposed the ParameterizedDQN (P-DQN) algorithm in 2018, and Bester et al. from the University of Witwatersrand improved it in 2019 and launched the Multi-Pass Q-Networks (MP-DQN) algorithm. These two algorithms are different from traditional compromise solutions. They directly process parameterized action space at the model level and have stronger adaptability and efficiency. Among them, the P-DQN algorithm is based on the Actor-Critic architecture. The Actor network first generates continuous actions based on state information, and then inputs these continuous actions together with the state information into the Q network. By calculating the Q values ​​of multiple discrete actions, the discrete action with the maximum Q value is finally selected as the optimal decision. In contrast, the MP-DQN algorithm further optimizes this process. By inputting continuous actions related to different discrete actions into the Q network multiple times, the algorithm effectively eliminates the interference of irrelevant continuous actions on the selection of discrete actions, thereby significantly improving the accuracy and robustness of discrete action decisions, and effectively enhancing the efficiency and stability of the algorithm in complex action spaces.

[0005] The P-DQN and MP-DQN algorithms solve the problem of parameterized action space through Actor networks and Q networks based on the Actor-Critic architecture. However, due to the structural limitations of the Q network, these two algorithms are only applicable to smaller discrete action spaces. In the MEC computation offloading problem, multiple users usually process multiple tasks at the same time, and the decisions between tasks should remain independent of each other. As the number of tasks and the offloading decision space increase, the output dimension of the Q network grows exponentially, which will significantly increase the computational and storage burden required by the algorithm. In particular, the MP-DQN algorithm faces greater challenges in terms of spatial complexity and time complexity because it uses multi-batch input to eliminate the impact of irrelevant continuous actions on discrete actions. This increase in complexity may not only lead to a decrease in processing efficiency, but also limit its effective application in large-scale, multi-task environments.

[0006] The QMIX algorithm has certain limitations in the application of parameterized action spaces in multi-agent environments. The algorithm uses a deep recurrent Q network (DRQN) for distributed execution of the agent network, which introduces a recursive neural network structure into the DQN algorithm to process temporal state information, thereby alleviating the problem of limited local perception of the agent. However, DRQN ​​still cannot directly cope with scenarios that require continuous action decisions, especially in the complex resource allocation problem in MEC computing offloading.

[0007] In addition, although QMIX merges the individual Q values ​​of each agent into a global Q value through a hybrid network to achieve centralized training, this strategy fails to fully consider the personalized needs and task differences between agents. Each agent may face different task priorities, computing power, and network conditions, which requires the algorithm to flexibly adapt to individual needs to achieve efficient resource utilization and task scheduling. QMIX's shortcomings in dealing with these personalized needs may lead to uneven resource allocation of agents or poor task offloading decisions, thus affecting the overall system performance and user experience. Summary of the invention

[0008] 1. Technical issues to be resolved

[0009] In view of the shortcomings of the prior art, the present invention provides a computational offloading optimization strategy based on multi-agent deep reinforcement learning, which solves the defects and shortcomings in the prior art.

[0010] (II) Technical solution

[0011] To achieve the above objectives, the present invention is implemented through the following technical solutions: a computing offloading optimization strategy based on multi-agent deep reinforcement learning, the core content of which is a multi-agent deep reinforcement learning algorithm designed for MEC multi-user computing offloading scenarios, and the specific steps of the algorithm are as follows:

[0012] Step 1: Initialize the online network parameters θ and target network parameters θ′ of all agents and hybrid networks, and initialize the experience pools {B i |i∈U} and the system experience pool B, where U represents the set of agents;

[0013] Step 2: Each agent obtains local state information by perceiving the environment t,i And some global information Each agent is based on the current local perception information o t,i , generate the corresponding resource allocation strategy through the target strategy network

[0014] Step 3: Convert the continuous actions obtained in step 2 With status information t,i Combine and decompose them into task states corresponding to each task of the agent and task continuity actions where j∈T i Represents the task set and task status information of agent i It not only contains the agent's perception information o t,i , also includes the continuous actions of other tasks except the corresponding task j

[0015] Step 4: Task Information Passed to the MPTQ network corresponding to each task in the agent to generate the Q value matrix In this process, these task information will be input into the MPTQ network in multiple batches for the corresponding offloading decisions, thereby effectively eliminating the interference with irrelevant continuous action information. Each row in the Q value matrix represents the task Q value of all tasks. Different rows reflect the offloading decisions of the corresponding tasks. t,i,j The impact of different on the Q value of all tasks;

[0016] Step 5: Each MPTQ network selects the task Q value The maximum Q value of the agent after aggregation The offloading decision represented by the corresponding row is the optimal offloading decision for the corresponding task And the task Q value corresponding to the optimal offloading decision of all MPTQs is After aggregation, the optimal agent Q value is formed

[0017] Step 6: Each agent allocates the resource strategy obtained in step 2 The optimal unloading decision in step 5 Combined, as the action a of the agent t,i Interact with the environment to obtain the corresponding agent benefit r t,i and the next state o t+1,i , and some global information Each agent sends its experience information (o t,i ,a t,i ,r t,i ,o t+1,i ) are stored in their respective experience pools Bi and the system experience pool B;

[0018] Step 7: Each agent will obtain the optimal agent Q value from step 6 Local information of the agent Combined and input into the self-attention mechanism network to generate a weighted Q value for each agent Weighted Agent Q-Value With some global information Input into the hybrid network and finally get the global Q value

[0019] Step 8: Each agent draws from its own experience pool B i Sampling empirical data to train the Actor and MPMTQ online networks in MPMT-DQN, and updating their respective target networks through soft updates;

[0020] Step 9: Sample experience data from the system experience pool B to train the hybrid network, the self-attention mechanism network, and the MPMTQ network of each agent to ensure that the agent network maximizes the overall benefits of the system while maximizing individual benefits;

[0021] Step 10: Repeat steps 2 to 9 until the algorithm converges.

[0022] Preferably, each agent network adopts the MPMT-DQN model based on the Actor-Critic architecture, which consists of an Actor network and an MPMTQ network. At each time step t, the Actor online network receives the current local perception information o of the agent. t,i As input, it outputs the agent's computing resource and communication resource allocation strategy for all tasks. As shown in formula (1), the Actor network can be described as a deterministic strategy:

[0023]

[0024] in, represents the online network parameters of the Actor of agent i, and N0 represents Gaussian noise.

[0025] Preferably, the MPMTQ network receives local observation information o of the agent. t,i and continuous actions output by the Actor network As input, and output the Q value Q of the agent i , the MPMTQ online network is expressed as shown in formula (2):

[0026]

[0027] in, represents the MPMTQ online network parameters of agent i.

[0028] The MPMTQ network decomposes the agent Q value into The MPMTQ is composed of MPTQ networks, each of which specifically learns the Q value of its corresponding task. Different from the Q network structure in MP-DQN, the MPTQ network not only receives the task state o of task j, but also t,i,j and continuous action Also includes information from all other tasks And other state information of agent i To ensure that the potential dependencies between tasks are fully considered.

[0029] Preferably, the MPTQ network will include other task information and other status information Considered as part of the corresponding task perception state, the corrected task state Expressed as Therefore, the input of MPTQ can be expressed as shown in formula (3):

[0030]

[0031] Among them, N s Indicates the number of edge servers, Indicates that the offloading decision for this task is s t,i,j The corrected task status at the time.

[0032] Preferably, all MPTQ networks of each agent will output an m×n Q value matrix in represents the elements in the matrix, and i Q ∈[1,…,m],j Q ∈[1,…,n]. In the matrix In the example, the number of rows m corresponds to the number of input batches, i.e., the range size N of the task offloading strategy. s +1; the number of columns n corresponds to the number of tasks for agent i Each row vector Indicates the offloading strategy s for the corresponding task t,i,j The task Q value of all tasks under represents the task Q value of the corresponding task j in each row. Therefore, the output of MPTQ can be expressed as shown in formula (4):

[0033]

[0034] Preferably, the MPTQ network selects the offloading strategy corresponding to the maximum agent Q value is the optimal discrete action, rather than selecting the s corresponding to the maximum task Q value of task jt,i,j , to ensure that the overall benefit of the agent task is maximized, therefore, the optimal unloading strategy It can be expressed as shown in formula (5):

[0035]

[0036] in, Indicates in s t,i,j The Q value of the agent after aggregation, P i,j represents the priority of task j.

[0037] Preferably, by aggregating all MPTQ networks in the optimal offloading strategy The optimal task Q value for task j is The Q value Q of agent i can be obtained t,i , as shown in formula (6):

[0038]

[0039] Preferably, SAE-QMIX introduces a self-attention mechanism between the hybrid network and the agent network, which can dynamically adjust the weights of each local value to meet the personalized needs of the agent, thereby providing a more accurate global value of the system;

[0040] The agent Q value {Q t,i |i∈U} and the modified agent state irrelevant to user privacy The Q values ​​of each agent are input into the self-attention network to calculate the weight of each agent. The weight matrix It is expressed as shown in formula (7):

[0041]

[0042] in, and denote the query vector and key vector in the self-attention mechanism, respectively, and is the scaling factor. In addition, W q and W k They correspond to the weight matrices of mapping q and k respectively; then, the weighted agent Q value is shown in formula (8):

[0043]

[0044] Preferably, the weighted Q-values ​​of all agents are The input will continue to be fed into the hybrid network, whose weights W = [W1, W2] and biases b = [b1, b2] are generated by the weight and bias network; at the same time, the restricted global state Also as the input of the hybrid network to ensure the effectiveness and accuracy of the aggregation process, it is worth noting that Not related to user privacy, global Q value As shown in formula (9):

[0045]

[0046] Among them, ReLu(·) is the activation function.

[0047] (III) Beneficial effects

[0048] The present invention provides a computational offloading optimization strategy based on multi-agent deep reinforcement learning. It has the following beneficial effects:

[0049] 1. The present invention uses value decomposition technology to construct a task Q network structure for each task, and generates corresponding task Q values ​​based on all task relationships of the agent. These task Q values ​​can be further aggregated to obtain the Q value of the agent, effectively alleviating the high time and space complexity problems faced by multi-task agents in complex parameterized action spaces. In addition, by inputting continuous actions related to different discrete actions into the Q network multiple times, the interference of irrelevant continuous actions on the selection of discrete actions is eliminated, thereby achieving a more accurate decision-making process.

[0050] 2. In this invention, the hybrid network uses the self-attention mechanism to dynamically calculate the weight of each agent's Q value, and by evaluating the relationship between agents, effectively captures key local information and suppresses redundant data, thereby enhancing the efficiency of information transmission and the accuracy of decision-making to meet personalized needs in different states. This mechanism significantly enhances the system's responsiveness and decision-making accuracy when dealing with information asymmetry and local perception problems.

[0051] 3. The present invention allows the agents to independently optimize their own strategies through the MPMT-DRQN algorithm during the training phase. The RNN structure integrated in the MPMT-DQN effectively realizes the perception of temporal information and enhances the decision-making ability of the agent in a dynamic environment. At the same time, the behavior of the global system is coordinated through the SAE-QMIX algorithm to ensure that the personalized needs of the agent are met. In addition, the training of the agent network is carried out simultaneously with the training of the system network model. This parallel processing method enables the agent to maintain flexibility in local decision-making, while ensuring the maximization of the global system performance through centralized training, thereby significantly improving the coordination ability and adaptability of the multi-agent system in a complex environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is the overall algorithm structure diagram of the present invention;

[0053] Figure 2It is the structure diagram of the MPMT-DQN algorithm of the present invention;

[0054] Figure 3 This is the structural diagram of the SAE-QMIX algorithm of the present invention. DETAILED DESCRIPTION

[0055] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0056] Example:

[0057] like Figure 1-3 As shown, the embodiment of the present invention provides a computing offloading optimization strategy based on multi-agent deep reinforcement learning. The core content is a multi-agent deep reinforcement learning algorithm designed for MEC multi-user computing offloading scenarios. The specific steps of the algorithm are as follows:

[0058] Step 1: Initialize the online network parameters θ and target network parameters θ′ of all agents and hybrid networks, and initialize the experience pools {B i |i∈U} and the system experience pool B, where U represents the set of agents;

[0059] Step 2: If Figure 1 As shown in (b), each agent obtains local state information by perceiving the environment. t,i And some global information Each agent is based on the current local perception information o t,i , generate the corresponding resource allocation strategy through the target strategy network

[0060] Step 3: If Figure 1 As shown in (a), the continuous action obtained in step 2 With status information t,i Combine and decompose them into task states corresponding to each task of the agent and task continuity actions where j∈T i Represents the task set and task status information of agent i It not only contains the agent's perception information o t,i , also includes the continuous actions of other tasks except the corresponding task j

[0061]

[0062] Step 4: Task Information Passed to the MPTQ network corresponding to each task in the agent to generate the Q value matrix In this process, these task information will be input into the MPTQ network in multiple batches for the corresponding offloading decisions, thereby effectively eliminating the interference with irrelevant continuous action information. Each row in the Q value matrix represents the task Q value of all tasks. Different rows reflect the offloading decisions of the corresponding tasks. t,i,j The impact of different on the Q value of all tasks;

[0063] Step 5: Each MPTQ network selects the task Q value The maximum Q value of the agent after aggregation The offloading decision represented by the corresponding row is the optimal offloading decision for the corresponding task And the task Q value corresponding to the optimal offloading decision of all MPTQs is After aggregation, the optimal agent Q value is formed

[0064] Step 6: Each agent allocates the resource strategy obtained in step 2 The optimal unloading decision in step 5 Combined, as the action a of the agent t,i Interact with the environment to obtain the corresponding agent benefit r t,i and the next state o t+1,i , and some global information Each agent sends its experience information (o t,i ,a t,i ,r t,i ,o t+1,i ) are stored in their respective experience pools Bi and the system experience pool B;

[0065] Step 7: If Figure 1 As shown in (c), each agent will obtain the optimal agent Q value obtained in step 6 Local information of the agent Combined and input into the self-attention mechanism network to generate a weighted Q value for each agent Weighted Agent Q-Value With some global information Input into the hybrid network and finally get the global Q value

[0066] Step 8: Each agent draws from its own experience pool B i Sampling empirical data to train the Actor and MPMTQ online networks in MPMT-DQN, and updating their respective target networks through soft updates;

[0067] Step 9: Sample experience data from the system experience pool B to train the hybrid network, the self-attention mechanism network, and the MPMTQ network of each agent to ensure that the agent network maximizes the overall benefits of the system while maximizing individual benefits;

[0068] Step 10: Repeat steps 2 to 9 until the algorithm converges.

[0069] In this invention, in the architecture, MPMT-DQN effectively solves the complex action space problem of the agent in the process of multi-task computing offloading, while the SAE-QMIX algorithm optimizes the limited state perception and agent decision competition in multi-agents. This invention aims to provide real-time and high-quality computing service support for MEC users.

[0070] First, by Figure 2 (b) It can be seen that each agent network adopts the MPMT-DQN model based on the Actor-Critic architecture, which consists of an Actor network and an MPMTQ network. At each time step t, the Actor online network receives the current local perception information o of the agent. t,i As input, it outputs the agent's computing resource and communication resource allocation strategy for all tasks. As shown in formula (1), the Actor network can be described as a deterministic strategy:

[0071]

[0072] in, represents the online network parameters of the Actor of agent i, and N0 represents Gaussian noise.

[0073] The MPMTQ network receives local observation information o of the agent. t,i and continuous actions output by the Actor network As input, and output the Q value Q of the agent i The MPMTQ online network can be expressed as shown in formula (2):

[0074]

[0075] in, represents the MPMTQ online network parameters of agent i.

[0076] Specifically, the MPMTQ network decomposes the agent Q value into The Q value of each task can effectively solve the high time complexity and space complexity problems caused by the parameterized action space in multi-task computing offloading. Figure 2 As shown in (a), MPMTQ is composed of MPTQ networks are composed of two MPTQ networks, each of which is dedicated to learning the Q value of its corresponding task. Different from the Q network structure in MP-DQN, the MPTQ network not only receives the task state o of task j, but also receives the task state o of task j. t,i,j and continuous action Also includes information from all other tasks And other state information of agent i To ensure that the potential dependencies between tasks are fully considered.

[0077] The MPTQ network uses the Multi-Pass method of the Q network, but only processes the state and continuous action information related to its corresponding task offloading decision This eliminates the influence of irrelevant information without considering other tasks. In other words, the MPTQ network takes information from other tasks as input. and other status information Considered as part of the corresponding task perception state. Corrected task state It can be expressed as Therefore, the input of MPTQ can be expressed as shown in formula (3):

[0078]

[0079] Among them, N s Indicates the number of edge servers, Indicates that the offloading decision for this task is s t,i,j The corrected task status at the time.

[0080] All MPTQ networks of each agent will output an m×n Q value matrix in represents the elements in the matrix, and i Q ∈[1,…,m],j Q ∈[1,…,n]. In the matrix In the example, the number of rows m corresponds to the number of input batches, i.e., the range size N of the task offloading strategy. s +1; the number of columns n corresponds to the number of tasks for agent i Each row vector Indicates the offloading strategy s for the corresponding task t,i,j The task Q value of all tasks under represents the task Q value of the corresponding task j in each row. Therefore, the output of MPTQ can be expressed as shown in formula (4):

[0081]

[0082] The MPTQ network selects the offloading strategy corresponding to the maximum agent Q value is the optimal discrete action, rather than selecting the s corresponding to the maximum task Q value of task j t,i,j , to ensure that the overall benefit of the agent task is maximized. Therefore, the optimal offloading strategy It can be expressed as shown in formula (5):

[0083]

[0084] in, Indicates in s t,i,j The Q value of the agent after aggregation, P i,j represents the priority of task j.

[0085] Finally, by aggregating all MPTQ networks in the optimal offloading strategy The optimal task Q value for task j is The Q value Q of agent i can be obtained t,i , as shown in formula (6):

[0086]

[0087] In summary, the goal of the MPMTQ network is to evaluate the Q-value of agent i, and the MPMTQ network is trained by minimizing the squared difference between the task Q-value and the target task Q-value of all MPTQ networks. Therefore, the loss function of the MPTQ online network for agent i corresponding to task j is As shown in formula (7):

[0088]

[0089] in, Indicates the parameters of the MPTQ online network, Indicates the task observation after correction and task actions Under this condition, the Q value of task j output by the MPTQ network. At the same time, the target task Q value y t,i,j It can be expressed by the time difference algorithm, as shown in formula (8):

[0090]

[0091] in, represents the task action of task j in the next time step, as shown in formula (9).

[0092] The goal of the Actor network is to maximize the Q value Q of the agent as much as possible. t,i As shown in formula (10), the loss function of the Actor network is calculated by t,i,j |s t,i,j =0,…,N s}, maximize the Q value of all aggregated agents To achieve:

[0093]

[0094] With the help of MPMT-DQN algorithm, the present invention can effectively solve the problem of computational offloading of agents in multi-task parameterized action space. In order to solve the problem of limited agent perception and decision competition between agents in multi-agent scenarios, the present invention proposes a Self-Attention Enhanced QMIX (SAE-QMIX) algorithm based on centralized training and distributed execution. The core idea of ​​the algorithm is to use a hybrid network to achieve value decomposition, integrate the local value of individual agents, and derive the global value of the system.

[0095] In this framework, the MPMT-DQN algorithm is used as an agent network, enabling agents to perform decisions independently. In addition, by adding GRU layers to the Actor network and all MPTQ networks, the enhanced MPMT-DQN (now called MPMT-DRQN) will be able to better remember and process historical observations and actions, thereby effectively coping with uncertainty and dynamic changes in MEC scenarios.

[0096] In addition, SAE-QMIX introduces a self-attention mechanism between the hybrid network and the agent network, which can dynamically adjust the weights of each local value to meet the personalized needs of the agent, thereby providing a more accurate global value of the system. Figure 3 As shown in the figure, the agent Q value {Q t,i |i∈U} and the modified agent state irrelevant to user privacy The Q values ​​of each agent are input into the self-attention network to calculate the weight of each agent Q value. The weight matrix It is expressed as shown in formula (11):

[0097]

[0098] in, and denote the query vector and key vector respectively, and is the scaling factor. In addition, W q and W k Corresponding to the weight matrices of mapping q and k respectively. Then, the weighted agent Q value can be expressed as shown in formula (12):

[0099]

[0100] Weighted Q-values ​​of all agents The input will continue to be fed into the hybrid network, whose weights W = [W1, W2] and biases b = [b1, b2] are generated by the weight and bias network. At the same time, the restricted global state It is also used as the input of the hybrid network to ensure the effectiveness and accuracy of the aggregation process. It is worth noting that It has nothing to do with user privacy. Global Q value The expression of is shown in formula (13):

[0101]

[0102] Among them, ReLu(·) is the activation function.

[0103] Generally, the value decomposition algorithm needs to satisfy the Individual Global Max (IGM) condition to ensure the consistency between the local value maximization of all agents and the global value. In order to satisfy the IGM condition, it is necessary to ensure that the derivative of the global value to the Q value of all agents is greater than zero, as shown in formula (14):

[0104]

[0105] Among them, the hybrid network ensures For all input The derivative of is greater than zero. At the same time, the SoftMax function in the self-attention network ensures that all weights is positive, thus ensuring Q t,i The derivative of is also greater than zero. Therefore, it can be proved that the SAE-QMIX algorithm satisfies the IGM condition.

[0106] Finally, similar to the process of updating the agent network, the loss function of the hybrid network, the self-attention network, and the MPMTQ network of all agents is And the Actor network used to train the agent to maximize the global Q value The loss function They can be expressed as formula (15) and formula (16) respectively:

[0107]

[0108] in, represents the network parameters mentioned in formula (15), the target global Q value y t It can be expressed as formula (17):

[0109]

[0110] Finally, the multi-agent deep reinforcement learning algorithm architecture based on MPMT-DQN and SAE-QMIX separates the training of the agent network from the training process of SAE-QMIX. Specifically, the agent ensures the optimization of individual performance by independently training the MPMT-DQN algorithm, while the server is responsible for training SAE-QMIX to maximize the overall system performance. This training method can not only improve the decision-making ability of individual agents, but also ensure the coordination and consistency of the system in multi-agent collaboration, thereby effectively responding to complex environmental changes and decision-making competition, and providing efficient solutions for applications such as MEC computing services.

[0111] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprising a reference structure" do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0112] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A computation offloading optimization strategy based on multi-agent deep reinforcement learning, characterized by: The core content is a multi-agent deep reinforcement learning algorithm designed for MEC multi-user computing offloading scenarios. The specific steps of the algorithm are as follows: Step 1: Initialize the online network parameters θ and target network parameters θ′ of all agents and hybrid networks, and initialize the experience pools {B i |i∈U} and the system experience pool B, where U represents the set of agents; Step 2: Each agent obtains local state information by perceiving the environment t,i And some global information Each agent is based on the current local perception information o t,i , generate the corresponding resource allocation strategy through the target strategy network Step 3: Convert the continuous actions obtained in step 2 With status information t,i Combine and decompose them into task states corresponding to each task of the agent and task continuity actions where j∈T i Represents the task set and task status information of agent i It not only contains the agent's perception information o t,i , also includes the continuous actions of other tasks except the corresponding task j Step 4: Task Information Passed to the MPTQ network corresponding to each task in the agent to generate the Q value matrix In this process, these task information will be input into the MPTQ network in multiple batches for the corresponding offloading decisions, thereby effectively eliminating the interference with irrelevant continuous action information. Each row in the Q value matrix represents the task Q value of all tasks. Different rows reflect the offloading decisions of the corresponding tasks. t,i,j The impact of different on the Q value of all tasks; Step 5: Each MPTQ network selects the task Q value The maximum Q value of the agent after aggregation The offloading decision represented by the corresponding row is the optimal offloading decision for the corresponding task And the task Q value corresponding to the optimal offloading decision of all MPTQs is After aggregation, the optimal agent Q value is formed Step 6: Each agent allocates the resource strategy obtained in step 2 The optimal unloading decision in step 5 Combined, as the action a of the agent t,i Interact with the environment to obtain the corresponding agent benefit r t,i and the next state o t+1,i , and some global information Each agent sends its experience information (o t,i ,a t,i ,r t,i ,o t+1,i ) are stored in their respective experience pools Bi and the system experience pool B; Step 7: Each agent will obtain the optimal agent Q value from step 6 Local information of the agent Combined and input into the self-attention mechanism network to generate a weighted Q value for each agent Weighted Agent Q-Value With some global information Input to the hybrid network and finally get the global Q value Step 8: Each agent draws from its own experience pool B i Sampling empirical data to train the Actor and MPMTQ online networks in MPMT-DQN, and updating their respective target networks through soft updates; Step 9: Sample experience data from the system experience pool B to train the hybrid network, the self-attention mechanism network, and the MPMTQ network of each agent to ensure that the agent network maximizes the overall benefits of the system while maximizing individual benefits; Step 10: Repeat steps 2 to 9 until the algorithm converges.

2. According to claim 1, a computational offloading optimization strategy based on multi-agent deep reinforcement learning is characterized in that: Each agent network adopts the MPMT-DQN model based on the Actor-Critic architecture, which consists of an Actor network and an MPMTQ network. At each time step t, the Actor online network receives the current local perception information o of the agent. t,i As input, it outputs the agent's computing and communication resource allocation strategy for all tasks. As shown in formula (1), the Actor network can be described as a deterministic strategy: in, represents the online network parameters of the Actor of agent i, and N0 represents Gaussian noise.

3. The computation offloading optimization strategy based on multi-agent deep reinforcement learning according to claim 2, characterized in that: The MPMTQ network receives local observation information o of the agent. t,i and continuous actions output by the Actor network As input, and output the Q value Q of the agent i , the MPMTQ online network is expressed as shown in formula (2): in, represents the MPMTQ online network parameters of agent i. The MPMTQ network decomposes the agent Q value into The MPMTQ is composed of MPTQ networks, each of which specifically learns the Q value of its corresponding task. Different from the Q network structure in MP-DQN, the MPTQ network not only receives the task state o of task j, but also t,i,j and continuous action Also includes information from all other tasks And other state information of agent i To ensure that the potential dependencies between tasks are fully considered.

4. The computation offloading optimization strategy based on multi-agent deep reinforcement learning according to claim 3 is characterized in that: The MPTQ network will transfer other task information and other status information Considered as part of the corresponding task perception state, the corrected task state Expressed as Therefore, the input of MPTQ can be expressed as shown in formula (3): Among them, N s represents the number of edge servers, Indicates that the offloading decision for this task is s t,i,j The corrected task status at the time.

5. The computation offloading optimization strategy based on multi-agent deep reinforcement learning according to claim 4 is characterized in that: All MPTQ networks of each agent will output an m×n Q value matrix in represents the elements in the matrix, and i Q ∈[1,…,m],j Q ∈[1,…,n]. In the matrix In the example, the number of rows m corresponds to the number of input batches, i.e., the range size N of the task offloading strategy. s +1; the number of columns n corresponds to the number of tasks for agent i Each row vector Indicates the offloading strategy s for the corresponding task t,i,j The task Q value of all tasks under represents the task Q value of the corresponding task j in each row. Therefore, the output of MPTQ can be expressed as shown in formula (4):

6. The computation offloading optimization strategy based on multi-agent deep reinforcement learning according to claim 5, characterized in that: The MPTQ network selects the offloading strategy corresponding to the maximum agent Q value is the optimal discrete action, rather than selecting the s corresponding to the maximum task Q value of task j t,i,j , to ensure that the overall benefit of the agent task is maximized, therefore, the optimal unloading strategy It can be expressed as shown in formula (5): in, Indicates in s t,i,j The Q value of the agent after aggregation, P i,j represents the priority of task j.

7. The computation offloading optimization strategy based on multi-agent deep reinforcement learning according to claim 6, characterized in that: By aggregating all MPTQ networks in the optimal offloading strategy The optimal task Q value for task j is The Q value Q of agent i can be obtained t,i , as shown in formula (6):

8. The computation offloading optimization strategy based on multi-agent deep reinforcement learning according to claim 7, characterized in that: SAE-QMIX introduces a self-attention mechanism between the hybrid network and the agent network, which can dynamically adjust the weights of each local value to meet the personalized needs of the agent, thereby providing a more accurate global value of the system; The agent Q value {Q t,i |i∈U} and the modified agent state irrelevant to user privacy The Q values ​​of each agent are input into the self-attention network to calculate the weight of each agent. The weight matrix It is expressed as shown in formula (7): in, and denote the query vector and key vector in the self-attention mechanism, respectively, and is the scaling factor. In addition, W q and W k They correspond to the weight matrices of mapping q and k respectively; then, the weighted agent Q value is shown in formula (8):

9. The computation offloading optimization strategy based on multi-agent deep reinforcement learning according to claim 8, characterized in that: Weighted Q-values ​​of all agents The input will continue to be fed into the hybrid network, whose weights W = [W1, W2] and biases b = [b1, b2] are generated by the weight and bias network; at the same time, the restricted global state Also as the input of the hybrid network to ensure the effectiveness and accuracy of the aggregation process, it is worth noting that Not related to user privacy, global Q value As shown in formula (9): Among them, ReLu(·) is the activation function.

Images