A Computation Offloading Method Based on Meta-Reinforcement Learning in Cell-Free Massive MIMO

Abstract

The present invention discloses a computation offloading method based on meta-reinforcement learning in cell-free massive MIMO, including: 1. Initializing the offloading task distribution, model weight parameters, and the number of model training cycles; 2. Each node receives the offloading task requests from users; 3. The node makes offloading decisions and RIS phase adjustment actions according to the offloading tasks; 4. The users under each node execute the offloading decisions, calculate the corresponding time delays, obtain rewards and the next state, and store them in the training pool; 5. The node samples the subtask trajectories in the task samples, updates the network model parameters, and transfers to the next training; 6. After the end of one training cycle, the nodes centrally calculate the loss function values of all tasks in the training pool and update the network model parameters twice. The present invention reduces the time delay of user task execution, improves the adaptability of users to the dynamic network, and improves the user service quality.

Description

Technical Field

[0001] The present invention belongs to the field of computation offloading of edge networks in mobile communication systems, and in particular relates to a computation offloading method based on meta-reinforcement learning in non-cellular large-scale MIMO. Background Art

[0002] With the development of mobile communications, the explosive growth of smart devices and global mobile data has placed tremendous pressure on traditional wireless access networks, giving rise to cellular-free wireless access networks (WANs). These networks deploy computing, storage, and network resources closer to user terminals, allowing users to offload computing tasks to nearby nodes, thereby alleviating network pressure and reducing task execution latency. However, when users offload their tasks to nodes, the transmission rate drops sharply in areas with weak signal coverage, leading to task execution delays and performance degradation. Reconfigurable smart surfaces (RIS) address this problem by adjusting the phase shift of reflective elements to enhance the signal propagation environment, increase transmission rates, and reduce task execution latency and energy consumption. Because the network environment of RIS-assisted WANs is dynamic and time-varying, rational computation offloading strategies are necessary to improve user service quality.

[0003] Existing computational offloading methods fall into two main categories: traditional optimization-based methods and reinforcement learning-based methods. Traditional methods transform the network environment into a fixed mathematical model, assuming a complete system model and the state transition probabilities of each state. This idealistic transformation lacks flexibility and is unsuitable for complex and dynamic network environments. To address these complex network environments, reinforcement learning is increasingly being used. However, traditional reinforcement learning suffers from poor adaptability to sudden changes, making it unsuitable for dynamic and changing networks. Furthermore, traditional reinforcement learning suffers from a low sampling rate, requiring retraining for each new environment, which is time-consuming. Summary of the Invention

[0004] The purpose of the present invention is to provide a computation offloading method based on meta-reinforcement learning in non-cellular massive MIMO to solve the technical problems of high latency in user task execution and poor adaptability to dynamic network environments.

[0005] In order to solve the above technical problems, the specific technical solutions of the present invention are as follows:

[0006] A method for computation offloading based on meta-reinforcement learning in cell-free massive MIMO includes the following steps:

[0007] Step 1: Initialize the model parameters of meta-reinforcement learning, including the offload task distribution ρ(V), where V is the set of user offload tasks, the inner and outer layer training rates α and β, and the policy function π θ (ai |s i ) parameter θ, where s i is the offloading state, a i is the decision execution action; initialize the number of training cycles Q;

[0008] Step 2: Each node receives the offloading tasks of all users within its coverage area, places all the user offloading tasks in the training pool, samples the user offloading tasks in the training pool, and constructs a task sample group;

[0009] Step 3: Randomly sample the user offloading task V i from the task sample group to construct the offloading state s i , and make a task offloading decision and decision execution action a i for the user according to the offloading state s i =(d i , ψ i ), where d i ∈{0,1} represents the task offloading decision, d i =1 means that the user task is offloaded to the node, d i =0 means that the user's task is offloaded locally; ψ i represents the phase allocation of the RIS;

[0010] Step 4: Calculate the latency of the decision execution action a i to obtain the reward r i and the next offloading state s i+1 , and store the signal (s i , a i , s i+1 , a i+1 ) in the training pool;

[0011] Step 5: The node samples the user offloading task V θ according to the policy function π i (a i |s i ) to form a set D i of subtask trajectories; use the set D i of subtask trajectories to calculate the gradient function, and use the calculated gradient function to update the model parameters of the meta-reinforcement learning, and transfer to the next training;

[0012] Step 6: After a training cycle ends, the node updates the model parameters of the meta-reinforcement learning by second-order gradient.

[0013] Preferably, the specific steps of the said Step 2 include the following steps:

[0014] Step 2.1: The node receives the offloading tasks of all users within its coverage area and places all the user offloading tasks in the training pool. The user offloading task is represented as a directed acyclic graph:

[0015] Γ = (V, Ε),

[0016] where V is the set of user offloading tasks, and Ε represents the set of dependency relationships between user offloading tasks. represents the dependency relationship between user offloading task V i and user offloading task V j ;

[0017] Step 2.2: The node sorts all the user offloading tasks in the training pool according to the time weight to obtain the set D of all sorted user offloading tasks G ;

[0018] Step 2.3: The node randomly samples user offloading tasks from the training pool to construct a task sample group.

[0019] Preferably, the specific steps of step 3 are as follows:

[0020] Step 3.1: The node receives the user offloading task V in the sampled task sample group i , and constructs the current task offloading state s i :

[0021] s i = (D G , A 1:i , R dl , R ul ),

[0022] where i ∈ I = {1, 2,..., I} is the set of user offloading task label numbers, A 1:i represents the offloading decision sequence of the first i offloading tasks of the user under the node, and R ul and R dl are the transmission rates of the uplink and downlink of the node and the users within the coverage area, and the calculation method is as follows:

[0023]

[0024]

[0025] where B is the bandwidth of the communication system, and are the signal-to-noise ratios of user k in the uplink and downlink;

[0026] Step 3.2: Obtain the offloading decision of the user offloading task and the RIS phase allocation:

[0027] a i=(d i , ψ i );

[0028] Step 3.3. The user of the node service performs action a i =(d i , ψ i ), d i ∈{0, 1} represents the task offloading decision of the user, d i =1 means the user's task is offloaded to the node, otherwise the user's task is offloaded locally, ψ i represents the phase allocation of the RIS.

[0029] Preferably, step 4 specifically includes the following steps:

[0030] Step 4.1. If the offloading task V i of the user under the node is executed locally, the local computing delay is:

[0031]

[0032] where C i is the number of CPU cycles required for the user to calculate and offload task V i , and f k is the CPU frequency of user k;

[0033] Step 4.2. If the offloading task V i of the user under the node is offloaded to the node for execution, the computing delay is:

[0034]

[0035] where f a is the CPU frequency of node a;

[0036] Step 4.3. According to steps 4.1 and 4.2, the delay of an offloading task V i is:

[0037] T i =FT i -FT i-1 ;

[0038] where FT i is the execution completion time of the offloading task V i ;

[0039] Step 4.4. Based on the delay of the offloading task V i , calculate the reward obtained by the decision execution action a i as:

[0040] r i =-T i ;

[0041] Step 4.5: The node executes action a i After that, a new state s is obtained i+1 , and the signal (s i , a i , r i , s i+1 ) is stored in the training pool

[0042] Preferably, step 5 specifically includes the following steps

[0043] Step 5.1: Use the policy function π i in the user-offloaded task V θ (a i |s i ) to randomly sample L subtasks D i =(τ i1 , τ i2 ,..., τ iL )

[0044] Step 5.2: The gradient function is calculated as follows

[0045]

[0046] where c is the loss coefficient is the pruning function is the value loss function, and their calculation methods are as follows respectively

[0047]

[0048]

[0049] where E is the mean calculation function, τ is the subtask is the user-offloaded task distribution, ε is the truncation factor, l i (θ) is the policy ratio is the Generalized Advantage Estimator (GAE), V π (s i ) is the value function is the true value function, and l i (θ) and are calculated as follows respectively

[0050]

[0051]

[0052] where π θ is the policy function is the policy function before parameter update, λ is the parameter for adjusting the deviation, and the parameter δ i is defined as:

[0053] δ i = r i (s i ) + γV π (s i+1 ) - V π (s i ),

[0054] where γ is the discount factor, γ ∈ [0, 1];

[0055] Step 5.3, Use the calculated gradient function to update the model parameters of meta-reinforcement learning:

[0056]

[0057] where α is the learning rate for inner-loop gradient update.

[0058] Preferably, the step 6 specifically includes the following steps:

[0059] Step 6.1, After a training cycle ends, the node recalculates the gradient J(θ) according to the updated model parameters of meta-reinforcement learning, and the calculation method is as follows:

[0060]

[0061] Through the already calculated gradient, the model parameters of meta-reinforcement learning are updated twice:

[0062] θ i ← θ - β▽ θ J(θ),

[0063] where β is the learning rate for outer-loop gradient update.

[0064] A computational offloading method based on meta-reinforcement learning in cell-free massive MIMO of the present invention has the following advantages:

[0065] 1. In the present invention, reinforcement learning is adopted at the node for computational offloading, which can perceive the environment with less information and can adapt to the complex network environment in the RIS-assisted cell-free radio access network, with high flexibility.

[0066] 2. The present invention uses proximal policy optimization (PPO) in reinforcement learning for offloading decision-making and resource allocation, which has good applicability to the continuous actions of RIS phase allocation, and avoids the problems of high complexity and slow convergence caused by the discretized high-dimensional action space.

[0067] 3. The present invention combines meta-learning with reinforcement learning, and utilizes the high adaptability and fast learning rate of meta-learning to achieve rapid adaptation to dynamic environments, greatly shorten training time, and significantly reduce user latency. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 This is a flow chart of a method for computation offloading based on meta-reinforcement learning in non-cellular massive MIMO according to the present invention;

[0069] Figure 2 This is a simulation result diagram comparing the present invention with the deep Q learning algorithm (DQN) and the deep deterministic policy gradient algorithm (DDPG). DETAILED DESCRIPTION

[0070] In order to better understand the purpose, structure and function of the present invention, the following is a further detailed description of a computation offloading method based on meta-reinforcement learning in a RIS-assisted non-cellular wireless access network of the present invention in conjunction with the accompanying drawings.

[0071] like Figure 1 FIG2 is a flow chart of a method for computation offloading based on meta-reinforcement learning in non-cellular massive MIMO according to the present invention, comprising the following steps:

[0072] S1: Initialize the model weight parameters of meta-reinforcement learning, including the offload task distribution ρ(V), where V is the set of user offloaded tasks, the inner and outer layer training rates α and β, and the policy function π θ (a i |s i ) parameter θ, where s i is the state, a i is the action; initialize the number of training cycles Q;

[0073] S2: Each node receives task offloading requests from all users within its coverage area, places all tasks in the training pool, samples the tasks in the training pool, and constructs a task sample group, specifically including:

[0074] S2-1: The node receives the offloading tasks of all users in the coverage area and puts all the tasks in the training pool. The user offloading tasks can be represented by a directed acyclic graph as follows:

[0075] Γ=(V,Ε),

[0076] Where E represents the set of dependencies between tasks;

[0077] S2-2: The node sorts all tasks in the training pool according to the time weight, and obtains the set D of all user tasks after sorting. G ;

[0078] S2-3: The node randomly samples tasks from the training pool and constructs a task sample group.

[0079] S3: User uninstalled tasks V in the randomly sampled task sample group i , build uninstall status s i , and according to the uninstall status s i Make task offloading decisions and decision execution actions for users i =(d i ,ψ i ), where d i ∈{0,1} represents the task offloading decision, d i =1 means the user task is offloaded to the node, d i =0 means the user's task is offloaded locally; i Indicates the phase allocation of RIS, including:

[0080] S3-1: The node receives the task sample v in the sampling task sample group i , build the current task offloading status:

[0081] s i =(D G ,A 1:i ,R dl ,R ul ),

[0082] Where i∈I={1,2,...,I} is the set of user task labels, A 1:i represents the offloading decision sequence of the first i computing tasks of the user under the node, R ul and R dl is the transmission rate of the uplink and downlink of the node and the users within the coverage area, which is calculated as follows:

[0083]

[0084]

[0085] Where B is the bandwidth of the communication system, and is the signal-to-noise ratio of user k in the uplink and downlink;

[0086] S3-2: The node obtains the task offloading decision and RIS phase allocation action based on the current task offloading status and strategy:

[0087] a i =(d i ,ψ i );

[0088] S3-3: The user of the node service performs action a i =(di , ψ i ), d i ∈ {0, 1} represents the user's task offloading decision, where d i = 1 indicates that the user offloads the task to the node, and vice versa, the user executes the task locally, ψ i represents the phase allocation of the RIS.

[0089] S4: The user executes action a according to the node's decision i , and calculates the latency of executing the action respectively to obtain the reward r i and the next state s i+1 , and stores the signal (s i , a i , s i+1 , a i+1 ) of this transition process in the task pool, specifically including:

[0090] S4-1: If the user task v i is executed locally, the local computing latency is:

[0091]

[0092] where C i is the number of CPU cycles required for the user to calculate task v i , and f k is the CPU frequency of user k;

[0093] S4-2: If the user task under the node is offloaded to the node for execution, the task computing latency is:

[0094]

[0095] where C i is the number of CPU cycles required to run the user task v i , and f a is the CPU frequency of node a;

[0096] S4-3: According to steps 4.1 and 4.2, the latency of a task v i is:

[0097] T i = FT i - FT i-1 ;

[0098] where FT i is the execution completion time of the user task v i ;

[0099] S4-4: Based on the latency of the user task execution, calculate the action a executed by the node iThe obtained reward is:

[0100] r i = -T i ;

[0101] S4-5: The node executes action a i and obtains a new state s i+1 , and stores the signal (s i , a i , r i , s i+1 ) generated by this conversion process in the training pool.

[0102] S5: The node samples task samples v θ (a i |s i ) according to the policy function π i to form a set D of subtask trajectories i ; Use D i to calculate the gradient function, and use the calculated gradient function to update the network model parameters, and transfer to the next training, specifically including:

[0103] S5-1: Randomly sample L subtasks from the task samples v i using the policy π θ (a i |s i )

[0104] S5-2: The calculation method of the gradient function is as follows:

[0105]

[0106] where c is the loss coefficient, is the pruning function, is the value loss function, and their calculation methods are as follows respectively:

[0107]

[0108]

[0109] where E is the mean calculation function, τ is the subtask, is the user offloading task distribution, ε is the truncation factor, l i (θ) is the policy ratio, is the Generalized Advantage Estimator (GAE), V π (s i ) is the value function, is the true value function, l i (θ) and The calculation methods are as follows:

[0110]

[0111]

[0112] Among them, π θ is the policy function, is the policy function before parameter update, λ is the parameter for adjusting the deviation, δ i is defined as:

[0113] δ i =r i (s i )+γv π (s i+1 )-v π (s i ),

[0114] Where γ is the discount factor, γ∈[0,1];

[0115] S5-3: Using the calculated gradient function Update network model parameters:

[0116]

[0117] Among them, α is the learning rate of the inner loop gradient update.

[0118] S6: After a training cycle, the node uses quadratic gradient to update the network model parameters.

[0119] S6-1: After a training cycle, the node recalculates the gradient based on the updated model parameters. The calculation method is as follows:

[0120]

[0121] Update the model parameters twice using the calculated gradient:

[0122] θ i ←θ-β▽ θ J(θ),

[0123] Where β is the learning rate of the outer loop gradient update.

[0124] By the attached Figure 2 The simulation results show that compared with the deep Q-learning algorithm (DQN) and the deep deterministic policy gradient algorithm (DDPG), the meta-learning method of the present invention has a faster convergence speed and a lower average system delay, indicating that the method proposed in the present invention has the best performance and significantly reduces the delay of user tasks.

[0125] It is understood that the present invention is described by way of some embodiments. Those skilled in the art will appreciate that various changes or equivalent substitutions can be made to these features and embodiments without departing from the spirit and scope of the present invention. Additionally, under the teachings of the present invention, these features and embodiments can be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of the present application belong to the scope protected by the present invention.

Claims

1. A computation offloading method based on meta-reinforcement learning in non-cellular massive MIMO, characterized by: The following steps are involved: Step 1: Initialize the model parameters of meta-reinforcement learning, including the offload task distribution ρ(V), where V is the set of user offload tasks, the inner and outer layer training rates α and β, and the policy function π θ (a i |s i ) parameter θ, where s i In the uninstall state, a i Execute actions for decision making; initialize the number of training cycles Q; Step 2: Each node receives the offloading tasks of all users within its coverage area, places all user offloading tasks in the training pool, samples the user offloading tasks in the training pool, and constructs a task sample group; Step 3: Randomly sample the user uninstalled tasks V in the task sample group i , build uninstall status s i , and according to the uninstall status s i Make task offloading decisions and decision execution actions for users i =(d i ,ψ i ), where d i ∈{0,1} represents the task offloading decision, d i =1 means the user task is offloaded to the node, d i =0 means the user's task is offloaded locally; i Indicates the phase assignment of RIS; Step 4: Calculate the decision execution action a i delay, and get reward r i With the next uninstall state s i+1 , and the signal (s i ,a i ,s i+1 ,a i+1 ) is stored in the training pool; Step 5: The node uses the policy function π θ (a i |s i ) Sample user uninstall task V i , forming a subtask trajectory set D i ; Use subtask trajectory set D i Calculate the gradient function and use the calculated gradient function to update the model parameters of the meta-reinforcement learning and move on to the next training; Step 6: After a training cycle, the node uses quadratic gradient to update the model parameters of the meta-reinforcement learning.

2. The method for computation offloading based on meta-reinforcement learning in non-cellular massive MIMO according to claim 1, characterized in that: The step 2 specifically includes the following steps: Step 2.1: The node receives the offloading tasks of all users in the coverage area and puts all the user offloading tasks into the training pool. The user offloading tasks are represented by a directed acyclic graph as follows: Γ=(V,Ε), Where V is the set of user uninstalled tasks, E represents the set of dependencies between user uninstalled tasks, Indicates that the user uninstalls the task V i and user uninstall task V j Dependencies between them; Step 2.2: The node sorts all user unloaded tasks in the training pool according to the time weight, and obtains the set D of all user unloaded tasks after sorting. G ; Step 2.3: The node randomly samples user offload tasks from the training pool to construct a task sample group.

3. The method for computation offloading based on meta-reinforcement learning in non-cellular massive MIMO according to claim 2, characterized in that: The step 3 specifically includes the following steps: Step 3.1, the node receives the user unloading task V in the sampling task sample group i , build the current task offloading state s i : s i =(D G ,A 1:i ,R dl ,R ul ), Where i∈I={1,2,...,I} is the set of user offload task labels, A 1:i represents the offloading decision sequence of the first i offloading tasks of the user under the node, R ul and R dl is the transmission rate of the uplink and downlink of the node and the users within the coverage area, which is calculated as follows: Where B is the bandwidth of the communication system, and is the signal-to-noise ratio of user k in the uplink and downlink; Step 3.2: Obtain the user's offloading decision and RIS phase allocation for the offloading task: a i =(d i ,ψ i ); Step 3.3: The user of the node service performs action a i =(d i ,ψ i ), d i ∈{0,1} represents the user’s task offloading decision, d i =1 means the user's task is offloaded to the node, otherwise the user's task is offloaded to the local, ψ i Indicates the phase assignment of RIS.

4. The method for computation offloading based on meta-reinforcement learning in non-cellular massive MIMO according to claim 3, characterized in that: The step 4 specifically includes the following steps: Step 4.1: If the user's uninstall task V under the node i If executed locally, the local computing latency is: Among them, C i Calculate the offload task V for the next user i The number of CPU cycles required, f k is the CPU frequency of user k; Step 4.2: If the user's uninstall task V under the node i Offloading to the node for execution, the computational latency is: Among them, f a is the CPU frequency of node a; Step 4.3: According to steps 4.1 and 4.2, obtain an offloading task V i The delay is: T i =FT i -FT i-1 ; Among them, FT i To uninstall the task V i The execution completion time; Step 4.4: Based on the offloading task V i The delay of computing decision execution action a i The rewards obtained are: r i ＝-T i ； Step 4.5: Node executes action a i After that, we get the new state s i+1 , the signal (s i ,a i ,r i ,s i+1 ) are stored in the training pool.

5. The method for computation offloading based on meta-reinforcement learning in non-cellular massive MIMO according to claim 4, characterized in that: The step 5 specifically includes the following steps: Step 5.

1. Uninstall Task V from User i Use the policy function π θ (a i |s i ) Randomly sample L subtasks Step 5.2: Gradient function The calculation is as follows: Where c is the loss coefficient, is the pruning function, is the value loss function, which is calculated as follows: Among them, E is the mean calculation function, τ is the subtask, P Vi is the user offload task distribution, ε is the truncation factor, l i (θ) is the strategy ratio, is the general advantage estimator (GAE), V π (s i ) is the value function, is the true value function, l i (θ) and The calculation methods are as follows: Among them, π θ is the policy function, is the policy function before parameter update, λ is the parameter for adjusting the deviation, and parameter δ i is defined as: δ i =r i (s i )+γV π (s i+1 )-V π (s i ), Where γ is the discount factor, γ∈[0,1]; Step 5.3: Use the calculated gradient function Update the model parameters of meta-reinforcement learning: Among them, α is the learning rate of the inner loop gradient update.

6. The method for computation offloading based on meta-reinforcement learning in non-cellular massive MIMO according to claim 5, characterized in that: The step 6 specifically includes the following steps: Step 6.1: After a training cycle, the node recalculates the gradient J(θ) based on the updated meta-reinforcement learning model parameters. The calculation method is as follows: Using the calculated gradients, update the model parameters of the meta-reinforcement learning twice: Where β is the learning rate of the outer loop gradient update.

Images