A multi-agent deep reinforcement learning-based computing offloading optimization strategy

Abstract

The application provides a kind of computing offload optimization strategy based on multi-agent deep reinforcement learning, it is related to intelligent algorithm technical field.The computing offload optimization strategy based on multi-agent deep reinforcement learning, multi-agent deep reinforcement learning algorithm designed for MEC multi-user computing offload scene.The application, using value decomposition technology, constructs the task Q network structure for each task, and generates the corresponding task Q value based on considering the relationship of all tasks of the agent.These task Q values can be further aggregated to obtain the Q value of the agent, effectively alleviating the high time and space complexity problems faced by multi-task agents in complex parameterized action space.In addition, in the hybrid network, the self-attention mechanism is used to dynamically allocate the corresponding weight to the features of different users, thereby achieving more accurate interaction representation between agents and improving the flexibility and effectiveness of the overall decision.

Description

Technical Field

[0001] The present invention relates to the field of intelligent algorithm technology, and specifically to a computational offloading optimization strategy based on multi-agent deep reinforcement learning. Background Art

[0002] In recent years, Mobile Edge Computing (MEC) has been widely adopted in multiple key IoT scenarios, particularly in areas such as unmanned aerial vehicle systems, the Internet of Vehicles (IoV), and the Medical Internet of Things (IoT), achieving significant progress and success. MEC provides high-quality service support for mobile users by offloading computationally intensive tasks to edge servers close to users and rationally allocating computing and communication resources to these tasks. By performing processing at the network edge, MEC can respond to user needs in real time, ensuring the timeliness and accuracy of data. MEC not only significantly reduces the computational load on mobile devices but also effectively reduces data transmission latency, significantly improving the user experience. Furthermore, MEC's ​​resource optimization strategies can dynamically adjust based on current network conditions and user needs, further enhancing the system's flexibility and adaptability. This enables mobile users to enjoy a smoother and more efficient service experience in a variety of environments, from complex urban transportation to remote medical scenarios.

[0003] With the continuous development of artificial intelligence technology, deep reinforcement learning has become a core means of implementing MEC computing offload algorithms. In this field, the Deep Q Network (DQN) algorithm, by combining the perception capabilities of deep learning with the decision-making capabilities of reinforcement learning, can effectively solve the offload decision-making problem in discrete action spaces. For continuous action spaces in resource allocation, the Deep Deterministic Policy Gradient (DDPG) algorithm, based on the Actor-Critic architecture, demonstrates significant advantages. By generating optimal policies through an actor network and evaluating their effectiveness through a critic network, DDPG can dynamically adapt to the complex resource allocation requirements of MEC scenarios, ensuring efficient and flexible resource management in complex environments.

[0004] There are two common solutions to the problem of parameterized action spaces, where computational offloading strategies involve both discrete and continuous actions. One approach involves finely partitioning the discrete action space of the DQN algorithm to achieve continuous action. The other approach primarily utilizes the parameterized DDPG algorithm, discretizing continuous actions to optimize the parameterized action space. However, both approaches have limitations in terms of time and space complexity, as well as performance. To address the problem of parameterized action spaces, Xiong et al. from Tencent AI Lab proposed the ParameterizedDQN (P-DQN) algorithm in 2018. In 2019, Bester et al. from the University of the Witwatersrand improved upon this approach and introduced the Multi-Pass Q-Networks (MP-DQN) algorithm. These two algorithms differ from traditional compromises by directly processing the parameterized action space at the model level, offering greater adaptability and efficiency. The P-DQN algorithm, based on the Actor-Critic architecture, first generates continuous actions based on state information. These continuous actions are then fed into the Q network along with the state information. The Q values ​​of multiple discrete actions are calculated, and the discrete action with the maximum Q value is ultimately selected as the optimal decision. In contrast, the MP-DQN algorithm further optimizes this process. By repeatedly feeding continuous actions related to different discrete actions into the Q network, the algorithm effectively eliminates the interference of irrelevant continuous actions on discrete action selection, significantly improving the accuracy and robustness of discrete action decisions and effectively enhancing the algorithm's efficiency and stability in complex action spaces.

[0005] The P-DQN and MP-DQN algorithms address the problem of parameterized action spaces through an actor network and a Q network based on an actor-critic architecture. However, due to the structural limitations of the Q network, these two algorithms are only applicable to smaller discrete action spaces. In the MEC computational offloading problem, multiple users typically process multiple tasks simultaneously, and the decisions between tasks should remain independent of each other. As the number of tasks and the offloaded decision space increase, the output dimension of the Q network grows exponentially, which will significantly increase the computational and storage burden required by the algorithm. In particular, the MP-DQN algorithm faces greater challenges in terms of spatial and temporal complexity because it uses multiple batches of input to eliminate the influence of irrelevant continuous actions on discrete actions. This increase in complexity not only leads to a decrease in processing efficiency but also limits its effective application in large-scale, multi-task environments.

[0006] The QMIX algorithm has certain limitations in its application of parameterized action spaces in multi-agent environments. The algorithm's agent network for distributed execution uses a Deep Recurrent Q Network (DRQN). By introducing a recurrent neural network structure into the DQN algorithm to process temporal state information, it alleviates the problem of limited local perception of the agents. However, DRQN ​​still cannot directly address scenarios requiring continuous action decisions, particularly the complex resource allocation issues encountered in MEC computation offloading.

[0007] Furthermore, while QMIX uses a hybrid network to combine the individual Q-values ​​of each agent into a global Q-value for centralized training, this strategy fails to fully account for the individual needs and task differences between agents. Each agent may face different task priorities, computing power, and network conditions, requiring the algorithm to flexibly adapt to individual needs for efficient resource utilization and task scheduling. QMIX's shortcomings in addressing these individual needs can lead to uneven resource allocation among agents or poor task offloading decisions, impacting overall system performance and user experience. Summary of the Invention

[0008] (1) Technical problems solved

[0009] In response to the shortcomings of the existing technology, the present invention provides a computational offloading optimization strategy based on multi-agent deep reinforcement learning, which solves the defects and shortcomings of the existing technology.

[0010] (2) Technical solution

[0011] To achieve the above objectives, the present invention is implemented through the following technical solutions: a computation offloading optimization strategy based on multi-agent deep reinforcement learning, and a multi-agent deep reinforcement learning algorithm designed for MEC multi-user computation offloading scenarios. The specific steps of the algorithm are as follows:

[0012] Step 1: Initialize the online network parameters θ and target network parameters θ′ of all agents and hybrid networks, and initialize the experience pool of all agents and system experience pool in represents a collection of agents;

[0013] Step 2: Each agent obtains local state information by perceiving the environment t,i And some global information Each agent is based on the current local perception information o t,i , generate the corresponding resource allocation strategy through the target strategy network

[0014] Step three: combine the continuous actions obtained in step two with the state information o t,i , and decompose it into the task state information of each task of the agent and the continuous action of the task , where represents the task set of agent i, and the task state information contains not only the perception information o t,i of the agent, but also the continuous action of other tasks except the corresponding task j

[0015] Step four: the task information is passed to the MPTQ network corresponding to each task of the agent to generate a Q value matrix In this process, these task information will be input into the MPTQ network in batches for the corresponding offloading decision, so as to effectively eliminate the interference of irrelevant continuous action information, and each row in the Q value matrix represents the task Q value of all tasks Different rows reflect the influence on all task Q values due to different offloading decisions s t,i,j of the corresponding task;

[0016] Step five: each MPTQ network selects the task Q value after aggregation, the maximum agent Q value corresponding to the offloading decision represented by the corresponding row is the optimal offloading decision of the corresponding task , and the task Q value corresponding to the optimal offloading decision of all MPTQ after aggregation forms the optimal agent Q value

[0017] Step six: each agent combines the resource allocation strategy obtained in step two with the optimal offloading decision in step five as the action a t,i of the agent, and interacts with the environment to obtain the corresponding agent reward r t,i and the next state o t+1,i , as well as part of the global information Each agent stores its experience information (o t,i , a t,i , r t,i , o t+1,i ) in the respective experience pool and the system experience pool ;

[0018] Step seven: each agent combines the optimal agent Q value obtained in step six with the local information of the agent Combined and input into the self-attention mechanism network to generate the weighted Q value of each agent Weighted agent Q-value With some global information Input into the hybrid network and finally get the global Q value

[0019] Step 8: Each agent draws from its own experience pool Sampling empirical data to train the Actor and MPMTQ online networks in MPMT-DQN, and updating their respective target networks through soft updates;

[0020] Step 9: From the system experience pool Sampling empirical data to train the hybrid network, the self-attention mechanism network, and the MPMTQ network of each agent, to ensure that the agent network maximizes the overall benefit of the system while maximizing the individual benefit;

[0021] Step 10: Repeat steps 2 to 9 until the algorithm converges.

[0022] Preferably, each agent network adopts the MPMT-DQN model based on the Actor-Critic architecture, which consists of an Actor network and an MPMTQ network. At each time step t, the Actor online network receives the agent's current local perception information o t,i As input, it outputs the agent's computing and communication resource allocation strategy for all tasks. As shown in formula (1), the Actor network can be described as a deterministic strategy:

[0023]

[0024] in, represents the online network parameters of the Actor of agent i, represents Gaussian noise.

[0025] Preferably, the MPMTQ network receives the local observation information o of the agent t,i and continuous actions output by the Actor network As input, and output the Q value Q of the agent i , the MPMTQ online network is expressed as shown in formula (2):

[0026]

[0027] in, represents the MPMTQ online network parameters of agent i.

[0028] The MPMTQ network decomposes the agent Q value into The task Q value is used to effectively solve the high time complexity and space complexity problems caused by the parameterized action space in multi-task computing offloading. MPMTQ is composed of Each MPTQ network is dedicated to learning the Q value of its corresponding task. Different from the Q network structure in MP-DQN, the MPTQ network not only receives the task state o of task j, but also t,i,j and continuous action Also includes information from all other tasks and other state information of agent i To ensure that the potential correlations between tasks are fully considered.

[0029] Preferably, the MPTQ network will send other task information and other status information Considered as part of the corresponding task perception state, the corrected task state Expressed as Therefore, the input of MPTQ can be expressed as shown in formula (3):

[0030]

[0031] Among them, N s Indicates the number of edge servers, Indicates that the offloading decision for this task is s t,i,j The corrected task status at the time.

[0032] Preferably, all MPTQ networks of each agent will output an m×n Q value matrix in represents the elements in the matrix, and i Q ∈[1,…,m],j Q ∈[1,…,n]. In the matrix In the example, the number of rows m corresponds to the number of input batches, i.e., the range size N of the task offloading strategy. s +1; the number of columns n corresponds to the number of tasks of agent i Each row vector Indicates the offloading strategy s for the corresponding task t,i,j The task Q value of all tasks under represents the task Q value of task j in each row. Therefore, the output of MPTQ can be expressed as shown in formula (4):

[0033]

[0034] Preferably, the MPTQ network selects the offloading strategy corresponding to the maximum agent Q value is the optimal discrete action, rather than selecting the s corresponding to the maximum task Q value of task j t,i,j , to ensure that the overall benefit of the agent task is maximized, therefore, the optimal offloading strategy It can be expressed as shown in formula (5):

[0035]

[0036] in, Indicates that in s t,i,j The Q value of the agent after aggregation, P i,j represents the priority of task j.

[0037] Preferably, by aggregating all MPTQ networks in the optimal offloading strategy The optimal task Q value corresponding to task j is The Q value Q of agent i can be obtained t,i , as shown in formula (6):

[0038]

[0039] Preferably, SAE-QMIX introduces a self-attention mechanism between the hybrid network and the agent network, which can dynamically adjust the weight of each local value to meet the personalized needs of the agent, thereby providing a more accurate global value of the system;

[0040] The agent Q value obtained by MPMT-DRQN and the modified agent state that is irrelevant to user privacy Will be input into the self-attention network together to calculate the weight of each agent's Q value; agent Q value The weight matrix It is expressed as shown in formula (7):

[0041]

[0042] in, and denote the query vector and key vector in the self-attention mechanism, respectively, and is the scaling factor. In addition, W q and W k The weight matrices corresponding to the mappings q and k respectively; then, the weighted agent Q value is shown in formula (8):

[0043]

[0044] Preferably, the weighted Q values ​​of all agents are The input will continue to be fed into the hybrid network, whose weights W = [W1, W2] and biases b = [b1, b2] are generated by the weight and bias network; at the same time, the restricted global state Also as the input of the hybrid network to ensure the effectiveness and accuracy of the aggregation process, it is worth noting that Not related to user privacy, global Q value As shown in formula (9):

[0045]

[0046] Among them, ReLu(·) is the activation function.

[0047] (3) Beneficial effects

[0048] This invention provides a computation offloading optimization strategy based on multi-agent deep reinforcement learning. It has the following beneficial effects:

[0049] 1. This invention utilizes value decomposition techniques to construct a task Q-network structure for each task and generates corresponding task Q-values ​​based on all the relationships between the agent's tasks. These task Q-values ​​can be further aggregated to obtain the agent's Q-value, effectively alleviating the high time and space complexity faced by multi-task agents in complex parameterized action spaces. Furthermore, by inputting continuous actions related to different discrete actions into the Q-network multiple times, the interference of unrelated continuous actions on the selection of discrete actions is eliminated, thereby achieving a more accurate decision-making process.

[0050] 2. This hybrid network utilizes a self-attention mechanism to dynamically calculate the Q-value weights of each agent. By evaluating relationships between agents, it effectively captures key local information and suppresses redundant data, thereby enhancing information transfer efficiency and decision-making accuracy to meet personalized needs in different situations. This mechanism significantly enhances the system's responsiveness and decision-making accuracy when addressing information asymmetry and local perception issues.

[0051] 3. During the training phase, this invention allows agents to independently optimize their strategies using the MPMT-DRQN algorithm. The RNN structure integrated in MPMT-DQN effectively realizes the perception of temporal information, enhancing the decision-making ability of agents in dynamic environments. Simultaneously, the SAE-QMIX algorithm coordinates the behavior of the global system to ensure that the personalized needs of the agents are met. Furthermore, the training of the agent network is performed simultaneously with the training of the system network model. This parallel processing approach enables agents to maintain flexibility in local decision-making while ensuring maximum global system performance through centralized training, significantly improving the collaboration and adaptability of multi-agent systems in complex environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 This is the overall algorithm structure diagram of the present invention;

[0053] Figure 2 This is the structure diagram of the MPMT-DQN algorithm of the present invention;

[0054] Figure 3 This is the structural diagram of the SAE-QMIX algorithm of the present invention. DETAILED DESCRIPTION

[0055] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0056] Example:

[0057] like Figures 1-3 As shown, the embodiment of the present invention provides a computing offloading optimization strategy based on multi-agent deep reinforcement learning, and a multi-agent deep reinforcement learning algorithm designed for MEC multi-user computing offloading scenarios. The specific steps of the algorithm are as follows:

[0058] Step 1: Initialize the online network parameters θ and target network parameters θ′ of all agents and hybrid networks, and initialize the experience pool of all agents and system experience pool in represents a collection of agents;

[0059] Step 2: If Figure 1 As shown in (b), each agent obtains local state information by perceiving the environment. t,i And some global information Each agent is based on the current local perception information o t,i , generate the corresponding resource allocation strategy through the target strategy network

[0060] Step 3: If Figure 1 As shown in (a), the continuous action obtained in step 2 With status information t,i Combine and decompose it into task states corresponding to each task of the agent and task continuity actions in Represents the task set and task status information of agent i It not only contains the agent's perception information o t,i , also includes the continuous actions of other tasks except the corresponding task j

[0061] Step 4: Task Information Passed to the MPTQ network corresponding to each task in the agent to generate the Q value matrix In this process, these task information will be input into the MPTQ network in multiple batches for the corresponding offloading decisions, thereby effectively eliminating the interference with irrelevant continuous action information. Each row in the Q value matrix represents the task Q value of all tasks. Different rows reflect the offloading decisions of the corresponding tasks s t,i,j The impact of different on the Q value of all tasks;

[0062] Step 5: Each MPTQ network selects the task Q value The maximum Q value of the agent after aggregation The offloading decision represented by the corresponding row is the optimal offloading decision for the corresponding task Moreover, the task Q value corresponding to the optimal offloading decision of all MPTQs is After aggregation, the optimal agent Q value is formed

[0063] Step 6: Each agent allocates the resource strategy obtained in step 2 and the optimal unloading decision in step 5 Combined, as the action a of the intelligent agent t,i Interact with the environment to obtain the corresponding agent benefit r t,i and the next state o t+1,i , and some global information Each agent will use its experience information ( t,i ,a t,i ,r t,i ,o t+1,i ) stored in their respective experience pools Neutralization system experience pool middle;

[0064] Step 7: If Figure 1 As shown in (c), each agent will obtain the optimal agent Q value obtained in step 6 and local information of the agent Combined and input into the self-attention mechanism network to generate the weighted Q value of each agent Weighted agent Q-value With some global information Input into the hybrid network and finally get the global Q value

[0065] Step 8: Each agent draws from its own experience pool Sampling empirical data to train the Actor and MPMTQ online networks in MPMT-DQN, and updating their respective target networks through soft updates;

[0066] Step 9: From the system experience pool Sampling empirical data to train the hybrid network, the self-attention mechanism network, and the MPMTQ network of each agent, to ensure that the agent network maximizes the overall benefit of the system while maximizing the individual benefit;

[0067] Step 10: Repeat steps 2 to 9 until the algorithm converges.

[0068] In this paper, the MPMT-DQN architecture effectively addresses the complex action space problem faced by agents during multi-task computation offloading, while the SAE-QMIX algorithm optimizes the limited state perception and agent decision-making competition in multi-agent networks. This paper aims to provide real-time and high-quality computing services for MEC users.

[0069] First, by Figure 2 (b) It can be seen that each agent network adopts the MPMT-DQN model based on the Actor-Critic architecture, which consists of an Actor network and an MPMTQ network. At each time step t, the Actor online network receives the agent's current local perception information o t,i As input, it outputs the agent's computing and communication resource allocation strategy for all tasks. As shown in formula (1), the Actor network can be described as a deterministic strategy:

[0070]

[0071] in, represents the online network parameters of the Actor of agent i, represents Gaussian noise.

[0072] The MPMTQ network receives the local observation information o of the agent. t,i and continuous actions output by the Actor network As input, and output the Q value Q of the agent i The MPMTQ online network can be expressed as shown in formula (2):

[0073]

[0074] in, represents the MPMTQ online network parameters of agent i.

[0075] Specifically, the MPMTQ network decomposes the agent Q value into a multi-task Q-value to effectively solve the high time and space complexity problems caused by the parameterized action space in multi-task computation offloading. As shown in Fig. Figure 2 (a), the MPMTQ consists of MPTQ networks, each of which learns the Q-value of its corresponding task. Unlike the Q-network structure in MP-DQN, the MPTQ network not only receives the task state t,i,j and continuous action of task j, but also includes information from all other tasks and other state information of agent i to ensure that the potential relevance between tasks is fully considered.

[0076] The MPTQ network follows the Multi-Pass method of the Q-network, but only processes the state and continuous action information related to its corresponding task offloading decision so as to eliminate the influence of irrelevant information without considering other tasks. In other words, the MPTQ network regards the other task information and other state information as part of the perceived state of the corresponding task. The revised task state can be represented as Therefore, the input of MPTQ can be represented as shown in equation (3):

[0077]

[0078] where N s represents the number of edge servers, represents the revised task state when the offloading decision of this task is s t,i,j .

[0079] Each MPTQ network of each agent outputs a Q-value matrix of m x n where represents the element in the matrix, and i Q ∈[1,…,m], j Q ∈[1,…,n]. In the matrix , the number of rows m corresponds to the number of input batches, i.e., the size of the value range of the task offloading strategy N s +1; the number of columns n corresponds to the number of tasks of agent i Each row vector represents the task Q-value of all tasks under the offloading strategy s t,i,j of the corresponding task, where represents the task Q-value of task j in each row. Therefore, the output of MPTQ can be represented as shown in equation (4):

[0080]

[0081] The MPTQ network selects the offloading strategy corresponding to the maximum agent Q value is the optimal discrete action, rather than selecting the s corresponding to the maximum task Q value of task j t,i,j , to ensure that the overall benefit of the agent task is maximized. Therefore, the optimal offloading strategy It can be expressed as shown in formula (5):

[0082]

[0083] in, Indicates that in s t,i,j The Q value of the agent after aggregation, P i,j represents the priority of task j.

[0084] Finally, by aggregating all MPTQ networks in the optimal offloading strategy The optimal task Q value corresponding to task j is The Q value Q of agent i can be obtained t,i , as shown in formula (6):

[0085]

[0086] In summary, the goal of the MPMTQ network is to evaluate the Q-value of agent i and train the MPMTQ network by minimizing the squared difference between the task Q-value and the target task Q-value of all MPTQ networks. Therefore, the loss function of the MPTQ online network for agent i corresponding to task j is As shown in formula (7):

[0087]

[0088] in, Indicates the parameters of the MPTQ online network, Indicates the task observation after correction and task actions Under this condition, the Q value of task j output by the MPTQ network. At the same time, the target task Q value y t,i,j It can be expressed by the time difference algorithm, as shown in formula (8):

[0089]

[0090] in, represents the task action of task j in the next time step, as shown in formula (9).

[0091] The goal of the Actor network is to maximize the Q value Q of the agent as much as possible. t,iAs shown in formula (10), the loss function of the Actor network is calculated by t,i,j |s t,i,j =0,…,N s}, maximize the Q value of all aggregated agents To achieve:

[0092]

[0093] Leveraging the MPMT-DQN algorithm, this paper effectively addresses the computational offloading problem of agents in multi-task parameterized action spaces. To address the limited perception of agents and the competitive decision-making between agents in multi-agent scenarios, this paper proposes a Self-Attention Enhanced QMIX (SAE-QMIX) algorithm based on centralized training and distributed execution. The core concept of this algorithm is to utilize a hybrid network to achieve value decomposition, integrating the local values ​​of individual agents to derive the global value of the system.

[0094] In this framework, the MPMT-DQN algorithm is used as the agent network, enabling agents to independently make decisions. Furthermore, by adding GRU layers to the Actor network and all MPTQ networks, the enhanced MPMT-DQN (now called MPMT-DRQN) is able to better memorize and process historical observations and actions, effectively handling the uncertainty and dynamic changes in MEC scenarios.

[0095] In addition, SAE-QMIX introduces a self-attention mechanism between the hybrid network and the agent network, which can dynamically adjust the weight of each local value to meet the personalized needs of the agent, thereby providing a more accurate global value of the system. Figure 3 As shown, the agent Q value obtained by MPMT-DRQN and the modified agent state that is irrelevant to user privacy The Q values ​​of each agent are input into the self-attention network to calculate the weight of each agent's Q value. The weight matrix It is expressed as shown in formula (11):

[0096]

[0097] in, and denote the query vector and key vector respectively, and is the scaling factor. In addition, W q and W kThe weight matrices corresponding to the mappings q and k are then expressed as follows:

[0098]

[0099] Weighted Q-values ​​of all agents The input will continue to be fed into the hybrid network, whose weights W = [W1, W2] and biases b = [b1, b2] are generated by the weight and bias network. At the same time, the restricted global state It is also used as the input of the hybrid network to ensure the effectiveness and accuracy of the aggregation process. It has nothing to do with user privacy. Global Q value The expression of is shown in formula (13):

[0100]

[0101] Among them, ReLu(·) is the activation function.

[0102] Typically, the value decomposition algorithm needs to satisfy the Individual Global Max (IGM) condition to ensure consistency between the local value maximization of all agents and the global value. To satisfy the IGM condition, it is necessary to ensure that the derivative of the global value with respect to the Q value of all agents is greater than zero, as shown in formula (14):

[0103]

[0104] Among them, the hybrid network ensures For all inputs The derivative of is greater than zero. At the same time, the SoftMax function in the self-attention network ensures that all weights is positive, thus ensuring Q t,i The derivative of is also greater than zero. Therefore, it can be proved that the SAE-QMIX algorithm meets the IGM condition.

[0105] Finally, similar to the process of updating the agent network, the loss function of the hybrid network, the self-attention network, and the MPMTQ network of all agents is And the Actor network used to train the agent to maximize the global Q value The loss function They can be expressed as formula (15) and formula (16) respectively:

[0106]

[0107] in, represents the network parameters mentioned in formula (15), the target global Q value yt It can be expressed as formula (17):

[0108]

[0109] Ultimately, the multi-agent deep reinforcement learning algorithm architecture based on MPMT-DQN and SAE-QMIX separates the training of the agent network from the training of SAE-QMIX. Specifically, the agents independently train the MPMT-DQN algorithm to optimize their individual performance, while the server trains SAE-QMIX to maximize overall system performance. This training approach not only improves the decision-making capabilities of individual agents but also ensures coordination and consistency across multi-agent collaboration, effectively coping with complex environmental changes and decision-making competition, providing a highly efficient solution for applications such as MEC computing services.

[0110] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the statement "comprising a reference structure" does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0111] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A computation offloading optimization strategy based on multi-agent deep reinforcement learning, characterized by: A multi-agent deep reinforcement learning algorithm designed for MEC multi-user computing offloading scenarios. The specific steps of the algorithm are as follows: Step 1: Initialize the online network parameters θ and target network parameters θ′ of all agents and hybrid networks, and initialize the experience pool of all agents and system experience pool in represents a collection of agents; Step 2: Each agent obtains local state information by perceiving the environment t,i And some global information Each agent is based on the current local perception information o t,i , generate the corresponding resource allocation strategy through the target strategy network Step 3: Convert the continuous action obtained in step 2 With status information t,i Combine and decompose it into task states corresponding to each task of the agent and task continuity actions in Represents the task set and task status information of agent i It not only contains the agent's perception information o t,i , also includes the continuous actions of other tasks except the corresponding task j Step 4: Task Information Passed to the MPTQ network corresponding to each task in the agent to generate the Q value matrix In this process, these task information will be input into the MPTQ network in multiple batches for the corresponding offloading decisions, thereby effectively eliminating the interference with irrelevant continuous action information. Each row in the Q value matrix represents the task Q value of all tasks. Different rows reflect the offloading decisions of the corresponding tasks s t,i,j The impact of different on the Q value of all tasks; Step 5: Each MPTQ network selects the task Q value The maximum agent Q value Q after aggregation i st*,i,j The offloading decision represented by the corresponding row is the optimal offloading decision for the corresponding task Moreover, the task Q value corresponding to the optimal offloading decision of all MPTQs is After aggregation, the optimal agent Q value is formed Step 6: Each agent allocates the resource strategy obtained in step 2 and the optimal unloading decision in step 5 Combined, as the action a of the intelligent agent t,i Interact with the environment to obtain the corresponding agent benefit r t,i and the next state o t+1,i , and some global information Each agent will use its experience information ( t,i ,a t,i ,r t,i ,o t+1,i ) stored in their respective experience pools Neutralization system experience pool middle; Step 7: Each agent will obtain the optimal agent Q value from step 6 and local information of the agent Combined and input into the self-attention mechanism network to generate the weighted Q value of each agent Weighted agent Q-value With some global information Input into the hybrid network and finally get the global Q value Step 8: Each agent draws from its own experience pool Sampling empirical data to train the Actor and MPMTQ online networks in MPMT-DQN, and updating their respective target networks through soft updates; Step 9: From the system experience pool Sampling empirical data to train the hybrid network, the self-attention mechanism network, and the MPMTQ network of each agent, to ensure that the agent network maximizes the overall benefit of the system while maximizing the individual benefit; Step 10: Repeat steps 2 to 9 until the algorithm converges.

2. The computation offloading optimization strategy based on multi-agent deep reinforcement learning according to claim 1, characterized in that: Each agent network adopts the MPMT-DQN model based on the Actor-Critic architecture, which consists of an Actor network and an MPMTQ network. At each time step t, the Actor online network receives the agent's current local perception information o t,i As input, it outputs the agent's computing and communication resource allocation strategy for all tasks. As shown in formula (1), the Actor network is described as a deterministic strategy: in, represents the online network parameters of the Actor of agent i, represents Gaussian noise.

3. The computation offloading optimization strategy based on multi-agent deep reinforcement learning according to claim 2, characterized in that: The MPMTQ network receives the local observation information o of the agent. t,i and continuous actions output by the Actor network As input, and output the Q value Q of the agent i , the MPMTQ online network is expressed as shown in formula (2): in, represents the MPMTQ online network parameters of agent i; The MPMTQ network decomposes the agent Q value into The task Q value is used to effectively solve the high time complexity and space complexity problems caused by the parameterized action space in multi-task computing offloading. MPMTQ is composed of Each MPTQ network is dedicated to learning the Q value of its corresponding task. Different from the Q network structure in MP-DQN, the MPTQ network not only receives the task state o of task j, but also t,i,j and continuous action Also includes information from all other tasks and other state information of agent i To ensure that the potential correlations between tasks are fully considered.

4. The computation offloading optimization strategy based on multi-agent deep reinforcement learning according to claim 3 is characterized by: The MPTQ network will transfer other task information and other status information Considered as part of the corresponding task perception state, the corrected task state Expressed as Therefore, the input representation of MPTQ is as shown in formula (3): Among them, N s Indicates the number of edge servers, Indicates that the offloading decision for this task is s t,i,j The corrected task status at the time.

5. The computation offloading optimization strategy based on multi-agent deep reinforcement learning according to claim 4 is characterized in that: All MPTQ networks of each agent will output an m×n Q value matrix in represents the elements in the matrix, and i Q ∈[1,…,m],j Q ∈[1,…,n]; in the matrix In the example, the number of rows m corresponds to the number of input batches, i.e., the range size N of the task offloading strategy. s +1; the number of columns n corresponds to the number of tasks of agent i ; Each row vector Indicates the offloading strategy s for the corresponding task t,i,j The task Q value of all tasks under represents the task Q value of task j in each row; therefore, the output of MPTQ is expressed as shown in formula (4):

6. The computation offloading optimization strategy based on multi-agent deep reinforcement learning according to claim 5, characterized in that: The MPTQ network selects the offloading strategy corresponding to the maximum agent Q value is the optimal discrete action, rather than selecting the s corresponding to the maximum task Q value of task j t,i,j , to ensure that the overall benefit of the agent task is maximized, therefore, the optimal offloading strategy It is expressed as shown in formula (5): in, Indicates that t,i,j The Q value of the agent after aggregation, P i,j represents the priority of task j.

7. The computation offloading optimization strategy based on multi-agent deep reinforcement learning according to claim 6, characterized in that: By aggregating all MPTQ networks in the optimal offloading strategy The optimal task Q value corresponding to task j is Get the Q value Q of agent i t,i , as shown in formula (6):

8. The computation offloading optimization strategy based on multi-agent deep reinforcement learning according to claim 7, characterized in that: SAE-QMIX introduces a self-attention mechanism between the hybrid network and the agent network, which can dynamically adjust the weights of each local value to meet the personalized needs of the agent, thereby providing a more accurate global value of the system; The agent Q value obtained by MPMT-DRQN and the modified agent state that is irrelevant to user privacy Will be input into the self-attention network together to calculate the weight of each agent's Q value; agent Q value The weight matrix It is expressed as shown in formula (7): in, and denote the query vector and key vector in the self-attention mechanism, respectively, and is the scaling factor; in addition, W q and W k The weight matrices corresponding to the mappings q and k respectively; then, the weighted agent Q value is shown in formula (8):

9. The computation offloading optimization strategy based on multi-agent deep reinforcement learning according to claim 8, characterized in that: Weighted Q-values ​​of all agents The input will continue to be fed into the hybrid network, whose weights W = [W1, W2] and biases b = [b1, b2] are generated by the weight and bias network; at the same time, the restricted global state Also as the input of the hybrid network to ensure the effectiveness and accuracy of the aggregation process, it is worth noting that Not related to user privacy, global Q value As shown in formula (9): Among them, ReLu(·) is the activation function.

Images