Multi-server edge computing task unloading method based on RSMA assistance

Abstract

The invention discloses a multi-server edge computing task unloading method based on RSMA assistance, and relates to the technical field of wireless communication, an edge computing environment mainly comprises M MEC servers which are randomly distributed, and N users are included in the coverage range of the M MEC servers; a discrete time model is used, and a time slot set is T; each user generates a calculation task Cn, and the calculation task comprises input data and calculation resources required for processing one data bit; due to limited energy, all users select to completely unload their tasks; a set of accessible MECs is used to execute different computing tasks; in consideration of the time-varying property of channel information, the method mainly adopts a reinforcement learning method, so that each user can autonomously select the optimal MEC server through interaction with the environment, and the delay experienced by the user is reduced to the minimum.

Description

Technical Field

[0001] The present invention relates to the field of wireless communication technologies, and in particular to a multi-server edge computing task offloading method based on RSMA assistance. Background Art

[0002] With the development of science and technology, in the upcoming 6G network, Rate Splitting Multiple Access (RSMA) has become the leader of multiple access schemes. Downlink RSMA requires dividing each user's data into public and private components. When public messages are sent to all users at the same time, private messages are sent only to designated recipients. Users decode public messages before decoding their own private messages, partially alleviating the interference caused by other users' transmissions. RSMA can improve the system's data transmission capacity on limited spectrum resources through rate segmentation, thereby improving spectrum utilization. By reasonably segmenting and decoding user signals, multi-user interference is reduced, thereby improving the overall performance of the system and improving interference management. The rate allocation can be dynamically adjusted according to the network environment and user needs to improve the flexibility and adaptability of the system.

[0003] Multi-access edge computing (MEC) is a technology that moves computing and data storage tasks from centralized data centers to the edge of the network, aiming to reduce latency, improve data processing efficiency, and save bandwidth resources. In scenarios such as the Internet of Things (IoT), smart cities, and autonomous driving, edge computing can provide real-time data processing and analysis capabilities.

[0004] Task offloading is a core concept in edge computing, which refers to offloading computing tasks from resource-constrained end devices (such as smartphones, sensors, etc.) to edge servers or the cloud to achieve more efficient computing and faster response speed. Task offloading strategies need to consider multiple factors such as computing resources, network bandwidth, latency, and energy consumption. Combined with RSMA's edge computing task offloading strategy, the performance and response speed of edge computing systems can be significantly improved by optimizing spectrum utilization, improving data transmission efficiency, and enhancing system robustness. This technology has important practical value in application scenarios such as smart transportation, industrial automation, and smart cities. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides a multi-server edge computing task offloading method based on RSMA assistance, comprising the following steps:

[0006] S1. Establish an edge computing model and initialize the edge computing environment, number of users, and number of MEC servers;

[0007] S2. According to the user and MEC server status, formulate the requirements for task offloading, and determine the optimization objectives and constraints;

[0008] S3. According to the optimization objectives, convert the optimization problem into a Markov decision process; construct a reinforcement learning framework, and determine the agent, state space, action space, and reward function;

[0009] S4. The agent continuously interacts with the environment according to the TD3 algorithm to update its own network;

[0010] S5. Obtain the optimal task offloading scheme.

[0011] The further limited technical solution of the present invention is:

[0012] Further, in step S1, the number of users is initialized as N, and the number of MEC servers is M. Assuming M < N, a discrete-time model is used, the time slot set is f, and each user generates a computing task C n ={I n , R n}, where I n represents the input data, and R n represents the computing resources required to process one data bit.

[0013] For a multi-server edge computing task offloading method based on RSMA assistance as described above, in step S2, using the rate splitting multiple access principle, each user is allowed to simultaneously offload each task to multiple MEC servers; the message W n，m contains the corresponding offloading tasks α n，m I n R n planned by the MEC server from the users, where α n，m represents the task allocation ratio from the user to the server, α n，m ∈ [0, 1], I n represents the input data, and R n represents the computing resources required to process one data bit;

[0014] The message is decomposed into a common part and a private part The common part of all the predetermined MEC servers starting from the user is merged and encoded into a common signal s n，0 and transmitted to the selected MEC server with power p n，0 ; each private part is encoded into a private signal s n，m and transmitted to the corresponding MEC server with power p n，m ; therefore, the transmitted signal of the user is:

[0015]

[0016] The signal received by the MEC server from the user is:

[0017]

[0018] Where M represents the number of MEC servers, z n，m Denote the variance as σ 2 Zero-mean additive Gaussian white noise, g n，m represents the channel gain from user n to MEC server m determined by the distance-dependent path loss formula, p n，j represents the transmission power of the private signal transmitted to the jth server, s n，j represents a private signal sent to server j;

[0019] MEC server decodes n，0 The achievable rate is:

[0020]

[0021] Among them, B n Indicates the average bandwidth allocated to users;

[0022] Assume that the channel gains offloaded from user n to MEC server m are arranged in ascending order, such that g n，1 ≤…≤g n，m ≤…≤g n，M , in order for each MEC server m to successfully decode s in user n n，0 , the allocation rate β n，m Must meet:

[0023]

[0024] The equality of the formula is the result of the ordered channel gain; given the ordered channel gain, in order to ensure the successful application of the continuous interference cancellation technology on the MEC server m, this condition must be met:

[0025]

[0026] Among them, g n，1 represents the first channel gain of user n, represents the continuous interference cancellation detection threshold;

[0027] Each MEC server decodes s at an achievable rate n，m :

[0028]

[0029] The total achievable rate from user n to MEC server m is Therefore, the upload delay of a task from user n to MEC server m is:

[0030]

[0031] As described above, in the RSMA-assisted multi-server edge computing task offloading method, in step S2, f n，m It is represented as the computing resources F of MEC server m m Evenly distributed to each user n, MEC server m handles the offloaded task α n，m I n R n The processing delay generated when

[0032]

[0033] The processing and unloading delay is So user n processes task C n The delay experienced is T n =max{T n，m}.

[0034] As described above, in a multi-server edge computing task offloading method based on RSMA assistance, in step S2, the optimization goal is determined to minimize the delay:

[0035]

[0036] in, represents the joint optimization task allocation ratio, Indicates the allocation of public rate, Indicates the transmission power.

[0037] As described above, in the RSMA-assisted multi-server edge computing task offloading method, in step S3,

[0038] Agent: Users are considered as agents, and each agent chooses to offload tasks to the MEC server;

[0039] State space: The state s(t) contains the task information assigned to the user, channel gain, and computing resources of the MEC server. Indicates that, C n represents the computational task generated by the user, f n，m Indicates that the computing resources F of MEC server m are m Evenly distributed to each user n, g n，m represents the channel gain from user n to MEC server m determined by the distance-dependent path loss formula;

[0040] Action space: The action a(t) is defined by the optimization variables as Among them, α n represents the joint optimization task allocation ratio, β n Indicates the allocation of public rate, P n Indicates the transmission power;

[0041] Reward function: The reward r(t) is defined as minus the system delay The agent maximizes the expected cumulative reward as Where N represents the number of users, t represents the time slot, T represents the total number of time slots, and γ represents the discount factor, γ∈[0,1].

[0042] As described above, in the TD3 algorithm of step S4, each agent includes a dual Q network, a policy network, and a target network; the dual Q network includes two Q networks for estimating the value of an action, and the two Q networks are respectively and is a parameter used to reduce the overestimation of Q value; the policy network uses θπ as a parameter to generate a deterministic strategy from state to action; the parameters of the target network are and θ π′ , used to stabilize the training process.

[0043] As described above, the multi-server edge computing task offloading method based on RSMA assistance, the TD3 algorithm of step S4 specifically includes the following sub-steps:

[0044] S4.1. Initialize the weights of the two Q networks and ;

[0045] S4.2. Initialize the weights θ of the policy network π π ;

[0046] S4.3. Initialize the weights of the two target networks and the target policy network and θ π′ ;

[0047] S4.4, initialize the experience pool;

[0048] S4.5, set the discount factor γ, learning rate α and soft update coefficient;

[0049] S4.6, entering the sampling phase, each agent continuously interacts with the environment and stores the experience samples in the interaction process into the experience pool until the experience pool stores the maximum capacity of samples;

[0050] S4.7, enter the learning phase and use the delayed strategy update mechanism to update the network;

[0051] S4.8. When the number of rounds is reached, the training ends.

[0052] As described above, in the RSMA-assisted multi-server edge computing task offloading method, the sampling phase of step S4.6 specifically includes the following sub-steps:

[0053] S4.6.1. The agent obtains the current state S from the environment t , generate continuous actions a through the policy network t ;

[0054] S4.6.2. Agent performs action a t Afterwards, you will receive a reward t And enter the next state S t+1 , collect experience groups {S t , a t , r t , S t+1};

[0055] S4.6.3. The experience sample {S t , a t , r t , S t+1}Stored in the experience pool until there are enough samples stored in the experience pool.

[0056] As described above, in a multi-server edge computing task offloading method based on RSMA assistance, the learning phase of step S4.7 specifically includes the following sub-steps:

[0057] S4.7.1. Randomly draw a batch of experience samples from the experience pool {S t , a t , r t , S t+1};

[0058] S4.7.2. Calculate the target Q value by the following formula:

[0059] y=r(t)+γ·min(Q′1(S′ t+1 , a′ t+1 ), Q′2(S′ t+1 , a′ t+1 ))

[0060] Among them, Q′1 and Q′2 represent the target Q network, S′ t+1 represents the state generated by the current policy network, a′ t+1 represents the action generated by the current policy network;

[0061] S4.7.3. Calculate the loss function of the Q network and optimize it using the following formula:

[0062]

[0063] Where O represents the batch size;

[0064] S4.7.4. Every certain number of steps, update the policy network according to the following formula and optimize the policy network to maximize the Q value:

[0065]

[0066] in, It is the gradient of the Q network to the action, indicating how the current action strategy affects the Q value in a given state; is the gradient of the policy network, which indicates how to adjust the parameters of the policy network to optimize the policy in the current state;

[0067] S4.7.5. Every certain number of steps, the weights of the Q network and the policy network are soft-updated to the target network:

[0068] θ Q′ ←τθ Q +(1-τ)θ Q′

[0069] θ π′ ←τθ π +(1-τ)θ π′

[0070] Where τ represents the coefficient of soft update.

[0071] The beneficial effects of the present invention are:

[0072] (1) In the present invention, for edge computing scenarios with time-varying properties, most traditional algorithms can no longer meet the high-demand task offloading solutions, while the deep reinforcement learning algorithm used in the present invention can effectively cope with the time-varying nature of information transmission, which is more practical;

[0073] (2) In the present invention, the reinforcement learning method has good scalability and can adapt to networks of different sizes and different numbers of users and MEC servers, and has strong versatility;

[0074] (3) In the present invention, the TD3 algorithm solves the problems of over-estimation of the value function and unstable strategy update by introducing a dual Q network and a delayed update mechanism, thereby improving the stability and reliability of training; compared with other reinforcement learning algorithms, although the implementation of TD3 is complex, these improvements make it perform well in dealing with continuous action space problems; TD3 reduces the over-estimation of the value function by using a dual Q network, and improves the stability and efficiency of training by delaying the update of the strategy and target network; at the same time, the present invention optimizes the TD3 algorithm by introducing a dual Q network and a delayed strategy update mechanism, thereby improving the stability and effect of the training process;

[0075] (4) In the present invention, the integration of RSMA and MEC provides a solution to solve the application delay. The user chooses to offload to the optimal MEC server. By optimizing spectrum utilization, improving data transmission efficiency and enhancing system robustness, the system delay can be significantly reduced, and more efficient computing and system response speed can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1 A schematic diagram of an edge computing task offloading model in an embodiment of the present invention;

[0077] Figure 2 It is a schematic diagram of the overall process of the present invention;

[0078] Figure 3 A schematic diagram of the network architecture of the TD3 algorithm in an embodiment of the present invention;

[0079] Figure 4 The training flow chart of a specific embodiment of the present invention. DETAILED DESCRIPTION

[0080] This embodiment provides a multi-server edge computing task offloading method based on RSMA assistance, such as Figure 1 As shown, the scenario considered in this embodiment is a multi-server RSMA edge computing network, which includes M MEC servers randomly distributed, including N users within its coverage area; a discrete time model is used, and the time slot set is f; each user generates a computing task C n , the computing task includes the computing resources required to input data and process one data bit; due to limited energy, all users choose to completely offload their tasks; a group of accessible MECs are used to perform different computing tasks; considering the time-varying nature of channel information, the present invention mainly adopts the reinforcement learning method, so that each user can autonomously select the optimal MEC server through interaction with the environment, so as to minimize the delay experienced by the user.

[0081] like Figure 2As shown in the figure, a multi-server edge computing task offloading method based on RSMA assistance includes the following steps;

[0082] S1. Establish an edge computing model and initialize the edge computing environment, the number of users, and the number of MEC servers; initialize the number of users as N and the number of MEC servers as M. Assume M < N. Use a discrete-time model with a time slot set T. Each user generates a computing task C n ={I n , R n}, where I n represents the input data and R n represents the computing resources required to process one data bit.

[0083] S2. According to the user and MEC server status, formulate the requirements for task offloading, determine the optimization goal and constraints; utilize the rate-splitting multiple access principle to allow each user to simultaneously offload each task to multiple MEC servers; the message W n，m contains the corresponding offloading task α n，m I n R n planned by the MEC server from the user, where α n，m represents the task allocation ratio from the user to the server, α n，m ∈ [0, 1], I n represents the input data, and R n represents the computing resources required to process one data bit.

[0084] The message is decomposed into a common part and a private part The common part for all the scheduled MEC servers starting from the user is merged and encoded into a common signal s n，0 and transmitted to the selected MEC server with power p n，0 ; each private part is encoded into a private signal s n，m and transmitted to the corresponding MEC server with power p n，m ; thus, the transmitted signal of the user is:

[0085]

[0086] The signal received by the MEC server from the user is:

[0087]

[0088] where M represents the number of MEC servers, z n，m represents zero-mean additive white Gaussian noise with variance σ 2 and gn，m represents the channel gain from user n to MEC server m determined by the distance-dependent path loss formula, P n，j represents the transmission power of the private signal transmitted to the jth server, s n，j Represents a private signal sent to server j.

[0089] MEC server decodes n，0 The achievable rate is:

[0090]

[0091] Among them, B n Indicates the average bandwidth allocated to users.

[0092] Assume that the channel gains offloaded from user n to MEC server m are arranged in ascending order, such that g n，1 ≤…≤g n，m ≤…≤g n，M , in order to ensure that each MEC server m can successfully decode s in user n n，0 , the allocation rate β n，m Must meet:

[0093]

[0094] The equality in the formula is the result of ordered channel gains. The equality in the formula is established because the channel gains are arranged in a certain order, which simplifies the specific channel selection problem. The channel gains in the formula have been sorted, so the min can be directly used. m∈M c n，m =c n，1 .

[0095] In order to ensure the successful application of the Successive Interference Cancellation (SIC) technology on the MEC server m under the condition of a given ordered channel gain, the following conditions must be met:

[0096]

[0097] Among them, g n，1 represents the first channel gain of user n, Represents the continuous interference cancellation detection threshold.

[0098] Each MEC server decodes s at an achievable rate n，m :

[0099]

[0100] The total achievable rate from user n to MEC server m is Therefore, the upload delay of a task from user n to MEC server m is:

[0101]

[0102] In order to simplify the allocation of computing resources, f n，m It is represented as the computing resources F of MEC server m m Evenly distributed to each user n, in addition, since the downlink transmission delay of the calculated output has a small impact on the overall delay, it can be ignored. MEC server m handles the offload task α n，m I n R n The processing delay generated when

[0103]

[0104] The delay for processing and unloading is So user n processes task C n The delay experienced is T n =max{T n，m}.

[0105] Determine the optimization goal to minimize latency:

[0106]

[0107] in, represents the joint optimization task allocation ratio, Indicates the allocation of public rate, Indicates the transmission power.

[0108] S3. According to the optimization goal, the optimization problem is converted into a Markov decision process (MDP); a reinforcement learning framework is constructed to determine the agent, state space, action space, and reward function.

[0109] Agent: Users are considered as agents, and each agent chooses to offload tasks to the MEC server.

[0110] State space: The state s(t) contains the task information assigned to the user, channel gain, and computing resources of the MEC server. Indicates that, C n represents the computational task generated by the user, f n，m Indicates that the computing resources F of MEC server m are m Evenly distributed to each user n, g n，mrepresents the channel gain from user n to MEC server m determined by the distance-dependent path loss formula.

[0111] Action space: The action a(t) is defined by the optimization variables as Among them, α n represents the joint optimization task allocation ratio, β n Indicates the allocation of public rate, P n Indicates the transmission power.

[0112] Reward function: The reward r(t) is defined as minus the system delay The agent is trained to maximize the expected cumulative reward as Where N represents the number of users, t represents the time slot, T represents the total number of time slots, and γ represents the discount factor, γ∈[0,1].

[0113] S4, the agent continuously interacts with the environment according to the TD3 algorithm to update its own network.

[0114] The deep reinforcement learning method is adopted to allow the intelligent agent to learn autonomously. In order to effectively process the Markov decision process model, this embodiment adopts the TD3 algorithm. This model-free deep reinforcement learning technology expands the deterministic policy gradient algorithm (Deep Deterministic Policy Gradient, DDPG), including two key improvements: dual Q network and delayed policy update.

[0115] The TD3 algorithm uses two deep neural networks as Q networks to estimate the value of actions, which can reduce over-estimation of Q values ​​and thus improve the stability of training. At the same time, it also includes a policy network (actor network) and a value network (critic network). The policy network generates a deterministic strategy that maps states to actions, while the value network is used to estimate the long-term expected reward for each action.

[0116] To further improve the stability of training, the TD3 algorithm introduces a delayed update mechanism, that is, regularly updating the policy network and target network to avoid overfitting of the training data. This mechanism ensures stable optimization of the policy. These improvements enable the well-trained policy network to be effectively used for online execution.

[0117] In the TD3 algorithm, each agent includes a dual Q network, a policy network (i.e., Actor network), and a target network. The dual Q network includes two Q networks for estimating the value of actions. and is a parameter used to reduce the overestimation of Q value; the policy network uses θπ as a parameter to generate a deterministic strategy from state to action; the parameters of the target network are and θπ′ , used to stabilize the training process.

[0118] The policy network in the TD3 algorithm directly generates continuous actions, while the Q network estimates the long-term expected reward of a given action; TD3 improves the stability of training through two main mechanisms, namely the dual Q network reduces the over-estimation problem of Q values, while the delayed update of the policy network and the target network ensures the stability and convergence of training; the learning process of the agent is mainly divided into two stages: the sampling stage and the learning stage.

[0119] In the TD3 algorithm, the sampling phase refers to each agent continuously interacting with the environment, and the agent will t Input to the policy network to generate continuous actions a t After the action is executed, the environment returns the immediate reward r t and the next state S t+1 , the agent stores these experience samples in the experience pool (Replay Buffer); this process continues until the experience pool stores a maximum of 2048 samples; the sampled experience samples are then used in the training process to achieve offline updates.

[0120] In the learning phase, when enough sample data has accumulated in the experience pool, the TD3 algorithm updates the network; the policy network and two Q networks (i.e., Q network 1 and Q network 2) are optimized using the sample data in the experience pool respectively. Through the dual Q networks, the TD3 algorithm reduces the over-estimation problem of the Q value; in the updating process, the TD3 algorithm uses a delayed policy update mechanism, that is, the policy network is updated every certain number of steps; for the training of the Q network, the target Q value is first calculated, and then the Q network is optimized through the mean square error (MSE) loss function.

[0121] Specifically, the target Q value is:

[0122] y=r(t)+γ·min(Q′1(S′ t+1 , a′ t+1 ), Q′2(S′ t+1 , a′ t+1 ))

[0123] Among them, Q′1 and Q′2 represent the target Q network, S′ t+1 represents the state generated by the current policy network, a′ t+1 represents the action generated by the current policy network.

[0124] The loss function is:

[0125]

[0126] Here, O represents the batch size.

[0127] The policy network is updated by maximizing the Q value, so that the policy tends to choose high-value actions; the policy network is updated every certain number of steps; the value estimated by the Q network is used to optimize the policy network:

[0128]

[0129] in, It is the gradient of the Q network to the action, indicating how the current action strategy affects the Q value in a given state; is the gradient of the policy network, which indicates how to adjust the parameters of the policy network to optimize the policy in the current state.

[0130] Then, every certain number of steps, the weights of the Q network and the policy network are soft-updated to the target network:

[0131] θ Q′ ←τθ Q +(1-τ)θ Q′

[0132] θ π′ ←τθ π +(1-τ)θ π′

[0133] Among them, τ represents the coefficient of soft update, which is usually very small.

[0134] The main improvements of the TD3 algorithm include dual Q networks (to reduce over-estimation of Q values) and delayed policy updates (to improve stability); the specific process includes acquiring and storing data from the environment in the sampling phase, updating the Q network and policy network in the learning phase, and performing stable soft updates through the target network; these steps help improve the learning efficiency and stability of the TD3 algorithm in complex environments.

[0135] The TD3 algorithm architecture is as follows Figure 3 The specific algorithm flow is as follows:

[0136] Initialize the weights of the two Q networks and

[0137] Initialize the weights θ of the policy network (Actor) π π

[0138] Initialize the weights of the two target networks and the target policy network and θ π′

[0139] Initialize experience pool

[0140] Set the discount factor γ, learning rate α and soft update coefficient

[0141] for iin Epi sode:

[0142] for j in Step:

[0143] Get the current state S from the environment t , generate continuous actions a through the policy network t

[0144] The agent performs action a t Afterwards, you will receive a reward t And enter the next state S t+1 , collect experience groups {S t , a t , r t , S t+1}

[0145] The sampled data {S t , a t , r t , S t+1}Store to experience pool

[0146] Randomly draw a batch of experience samples from the experience pool {S t , a t , r t , S t+1}

[0147] By y = r(t) + γ min(Q′1(S′ t+1 , a′ t+1 ), Q′2(S′ t+1 , a′ t+1 )) Calculate the target Q value

[0148] pass Calculate the loss function of the Q network and optimize it

[0149] Every certain number of steps, according to Update the policy network and optimize the policy network to maximize the Q value

[0150] At certain intervals, the weights of the Q network and the policy network are soft-updated until the target network reaches the number of rounds, and the training ends.

[0151] S5. Obtain the optimal task offloading solution.

[0152] In this embodiment, if Figure 4 As shown, the specific process includes the following:

[0153] Step 1: Initialize the edge computing environment and simulate the environment as 10 users randomly distributed and moving within a range of 200m×200m. Three MEC servers are randomly distributed in the area and the computing resources F m =10GHz.

[0154] Step 2: The device transmit power is 20 dBm, the minimum transmission rate is 1 Mbps, the SIC detection threshold is -94 dBm, the average bandwidth allocated to users is 1 MHz, the task size is randomly generated in the range of [0.8, 1.2] Mbits, and the CPU cycle is [800, 1000] CPU cycles / bit.

[0155] Step 3, initialize network parameters. The Q network and the policy network contain three fully connected layers. The number of neurons in the three hidden layers is 256, 128, and 64 respectively. Regarding hyperparameters, in this embodiment, the batch size is set to 32, the replay buffer capacity is 2048, the discount factor is 0.9, the soft update factor is 0.01, and the learning rate is set to 0.001.

[0156] Step 4: Set the number of agent training rounds T = 1000, set the number of training steps in each round to step = 100, use the optimizer to optimize the neural network parameters every 50 steps, record the rewards according to the agent's training process, and the agent continuously optimizes its own strategy according to the proposed algorithm to finally obtain the optimal task offloading solution.

[0157] In the prior art, although MEC scenarios have been explored in many studies, advanced multiple access schemes have not been fully explored. Therefore, this embodiment proposes to utilize the power of the downlink RSMA principle. This work can offload concurrent computing tasks from multiple tasks to multiple MEC servers, thereby minimizing the overall delay perceived by users. To achieve this goal, the method of this embodiment introduces a task offloading method based on reinforcement learning, which jointly optimizes the task allocation ratio, public rate and transmission power, and formulates a reasonable task offloading scheme to minimize the delay experienced by all users in the RSMA-assisted multi-MEC server environment.

[0158] In addition to the above embodiments, the present invention may also have other implementation modes. Any technical solution formed by equivalent replacement or equivalent transformation falls within the protection scope required by the present invention.

Claims

1. A multi-server edge computing task offloading method based on RSMA assistance, characterized in that: The steps include: S1. Establish an edge computing model and initialize the edge computing environment, number of users, and number of MEC servers; S2. According to the user and MEC server status, formulate the requirements for task offloading, determine the optimization goals and constraints; S3. According to the optimization goal, the optimization problem is converted into a Markov decision process; Build a reinforcement learning framework, determine the agent, state space, action space, and reward function; S4, the agent continuously interacts with the environment according to the TD3 algorithm to update its own network; S5. Obtain the optimal task offloading solution.

2. According to claim 1, a multi-server edge computing task offloading method based on RSMA assistance is characterized in that: In the step S1, the number of users is initialized as N, and the number of MEC servers is M. Assuming M < N, a discrete-time model is used, the time slot set is T, and each user generates a computing task C n ={I n , R n}, where I n represents the input data, and R n represents the computing resources required to process one data bit.

3. According to claim 1, a multi-server edge computing task offloading method based on RSMA assistance is characterized in that: In step S2, the rate division multiple access principle is used to allow each user to offload each task to multiple MEC servers at the same time; message W n，m Contains the corresponding offload tasks planned by the MEC server from users α n，m I n R n , where α n，m represents the task allocation ratio from user to server, α n，m ∈[0, 1], I n Represents input data, R n represents the computational resources required to process one bit of data; The message is broken down into common parts and private part The common part of all scheduled MEC servers started by the user are combined and encoded into a common signal s n，0 , with power p n，0 Transmitted to the selected MEC server; each private part is encoded into a private signal s n，m , with power P n，m Transmitted to the corresponding MEC server; therefore, the user's transmission signal is: The signal received by the MEC server from the user is: Where M represents the number of MEC servers, Z n，m Denote the variance as σ 2 Zero-mean additive Gaussian white noise, g n，m represents the channel gain from user n to MEC server m determined by the distance-dependent path loss formula, p n，j represents the transmission power of the private signal transmitted to the jth server, s n，j represents a private signal sent to server j; MEC server decodes n，0 The achievable rate is: Among them, B n Indicates the average bandwidth allocated to users; Assume that the channel gains offloaded from user n to MEC server m are arranged in ascending order, such that g n，1 ≤…≤g n，m ≤…≤g n，M , in order for each MEC server m to successfully decode s in user n n，0 , the allocation rate β n，m Must meet: The equality of the formula is the result of the ordered channel gain; given the ordered channel gain, in order to ensure the successful application of the continuous interference cancellation technology on the MEC server m, this condition must be met: Among them, g n，1 represents the first channel gain of user n, represents the continuous interference cancellation detection threshold; Each MEC server decodes s at an achievable rate n，m : The total achievable rate from user n to MEC server m is Therefore, the upload delay of a task from user n to MEC server m is:

4. According to claim 3, a multi-server edge computing task offloading method based on RSMA assistance is characterized in that: In step S2, f n，m It is represented as the computing resources F of MEC server m m Evenly distributed to each user n, MEC server m handles the offloaded task α n，m I n R n The processing delay generated is: The delay for processing and unloading is So user n processes task C n The delay experienced is T n =max{T n，m }.

5. The RSMA-assisted multi-server edge computing task offloading method according to claim 4 is characterized in that: In step S2, the optimization goal is determined to be minimizing the delay: in, represents the joint optimization task allocation ratio, Indicates the allocation of public rate, Indicates the transmission power.

6. The RSMA-assisted multi-server edge computing task offloading method according to claim 1 is characterized in that: In the step S3, Agent: Users are considered as agents, and each agent chooses to offload tasks to the MEC server; State space: The state s(t) contains the task information assigned to the user, channel gain, and computing resources of the MEC server. Indicates that, C n represents the computational task generated by the user, f n，m Indicates that the computing resources F of MEC server m are m Evenly distributed to each user n, g n，m represents the channel gain from user n to MEC server m determined by the distance-dependent path loss formula; Action space: The action a(t) is defined by the optimization variables as Among them, α n represents the joint optimization task allocation ratio, β n Indicates the allocation of public rate, P n Indicates the transmission power; Reward function: The reward r(t) is defined as minus the system delay The agent maximizes the expected cumulative reward as Where N represents the number of users, t represents the time slot, T represents the total number of time slots, and γ represents the discount factor, γ∈[0,1].

7. The RSMA-assisted multi-server edge computing task offloading method according to claim 6 is characterized in that: In the TD3 algorithm of step S4, each agent includes a dual Q network, a policy network and a target network; the dual Q network includes two Q networks for estimating the value of an action, and the two Q networks are respectively and is a parameter used to reduce the overestimation of Q value; the policy network uses θπ as a parameter to generate a deterministic strategy from state to action; the parameters of the target network are and θ π′ , used to stabilize the training process.

8. The RSMA-assisted multi-server edge computing task offloading method according to claim 7 is characterized in that: The TD3 algorithm of step S4 specifically includes the following sub-steps: S4.

1. Initialize the weights of the two Q networks and S4.

2. Initialize the weights θ of the policy network π π ; S4.

3. Initialize the weights of the two target networks and the target policy network and θ π′ ; S4.4, initialize the experience pool; S4.5, set the discount factor γ, learning rate α and soft update coefficient; S4.6, entering the sampling phase, each agent continuously interacts with the environment and stores the experience samples in the interaction process into the experience pool until the experience pool stores the maximum capacity of samples; S4.7, enter the learning phase and use the delayed strategy update mechanism to update the network; S4.

8. When the number of rounds is reached, the training ends.

9. A multi-server edge computing task offloading method based on RSMA assistance according to claim 8, characterized in that: The sampling phase of step S4.6 specifically includes the following sub-steps: S4.6.

1. The agent obtains the current state S from the environment t , generate continuous actions a through the policy network t ; S4.6.

2. Agent performs action a t Afterwards, you will receive a reward t And enter the next state S t+1 , collect experience groups {S t , a t , r t , S t+1 }; S4.6.

3. The experience sample {S t , a t , r t , S t+1 }Stored in the experience pool until enough samples are stored in the experience pool.

10. The RSMA-assisted multi-server edge computing task offloading method according to claim 8, characterized in that: The learning phase of step S4.7 specifically includes the following sub-steps: S4.7.

1. Randomly draw a batch of experience samples from the experience pool {S t , a t , r t , S t+1 }; S4.7.

2. Calculate the target Q value by the following formula: y=r(t)+γ·min(Q′1(S′ t+1 ,a′ t+1 ),Q′2(S′ t+1 ,a′ t+1 )) Among them, Q′1 and Q′2 represent the target Q network, S′ t+1 represents the state generated by the current policy network, a′ t+1 represents the action generated by the current policy network; S4.7.

3. Calculate the loss function of the Q network and optimize it using the following formula: Where O represents the batch size; S4.7.

4. Every certain number of steps, update the policy network according to the following formula and optimize the policy network to maximize the Q value: in, It is the gradient of the Q network to the action, indicating how the current action strategy affects the Q value in a given state; is the gradient of the policy network, which indicates how to adjust the parameters of the policy network to optimize the policy in the current state; S4.7.

5. Every certain number of steps, the weights of the Q network and the policy network are soft-updated to the target network: i Q′ ←tth Q +(1-τ)θ Q′ i π′ ←tth π +(1-τ)θ π′ Where τ represents the coefficient of soft update.

Images