Task unloading method for cloud edge-end collaborative environment

Abstract

The invention relates to the field of edge computing, in particular to a task unloading method for a cloud edge-end collaborative environment. Comprising the following steps: constructing a time delay energy consumption calculation model for executing an unloading task by the cloud edge-end cooperative system; building a problem model for an unloading task process based on a Markov decision process; a PER-DDPG-BiLSTM algorithm is adopted to solve the problem model; and selecting a server or an edge server or a terminal to execute task unloading according to a solving result. The invention solves the problem of task interruption caused by mobility of terminal equipment in the prior art and the technical defects of poor flexibility of an overall unloading strategy and low resource utilization rate. The cloud is introduced to serve as a secondary unloading node, dynamic migration and intelligent distribution of tasks among different levels of computing nodes are achieved, and therefore the task completion efficiency and the overall performance of the system are remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of edge computing, in particular to a task offloading method for a cloud-edge-end collaborative environment. BACKGROUND

[0002] With the continuous progress of information technology and the continuous improvement of equipment intelligence level, the computing tasks involved in the equipment operation process are increasingly complex and diversified. These tasks usually need to process massive data and put forward higher requirements on real-time performance, computing efficiency and system stability. However, in practical applications, terminal devices generally face problems such as limited local computing capacity and tight storage resources, and the traditional edge-end collaborative double-layer computing architecture has gradually exposed performance bottlenecks when facing high-load and complex computing tasks. In addition, terminal devices are often in a dynamic mobile state, their location changes frequently and is difficult to accurately predict, which can easily cause communication interruption between the terminal device and the edge server, thereby causing task execution failure or service quality degradation.

[0003] Currently, most researches on cloud-edge-end collaborative computing still mainly focus on the overall offloading strategy of tasks, that is, the entire task is uniformly offloaded to the edge node or the cloud for processing. Although this kind of method is relatively simple to implement, it has poor flexibility, high response delay and low resource utilization when facing real scenarios such as complex task structure, strong resource heterogeneity and frequent switching of service nodes by terminal devices. In contrast, although the partial offloading strategy of tasks can effectively improve the task execution efficiency and resource utilization efficiency, it involves complex problems such as sub-task division, dependency modeling and multi-node collaborative scheduling, and related researches are still in the exploratory stage. Especially in the environment where the terminal device has high frequency mobility, how to ensure task integrity while achieving efficient task splitting and scheduling is still a key problem to be solved.

[0004] In summary, the present application provides a cloud-edge-end collaborative task partial offloading method and system for mobile terminals, which solves the technical defects of task interruption caused by terminal device mobility and poor flexibility and low resource utilization of the overall offloading strategy in the prior art. By introducing the cloud as a secondary offloading node, dynamic migration and intelligent allocation of tasks between different levels of computing nodes are realized, thereby significantly improving the task completion efficiency and overall system performance. SUMMARY

[0005] The present application discloses a task offloading method for a cloud-edge-end collaborative environment, which comprises a cloud-edge-end collaborative system composed of a cloud server, an edge server and a terminal, and performs task offloading of the cloud-edge-end collaborative system. The specific method is as follows:

[0006] A time delay and energy consumption calculation model of the cloud-edge-end collaborative system for executing offloading tasks is constructed;

[0007] A problem model for the unloading task process is constructed based on Markov decision processes.

[0008] The PER-DDPG-BiLSTM algorithm is used to solve the problem model;

[0009] Based on the solution results, choose to perform the task uninstallation on the server, edge server, or terminal.

[0010] Furthermore, if terminal mobility causes connection interruption or edge server timeout, a secondary offloading to the cloud is triggered, with the specific triggering conditions as follows:

[0011] If the task is not completed in time slot t and the connection state is Conn m,n If (t) = 0, the connection is interrupted, triggering a secondary unloading;

[0012] If the edge server execution latency There is a risk of timeout, triggering a second uninstallation.

[0013] Furthermore, the latency and energy consumption calculation model includes: a terminal device local calculation sub-model, a communication transmission sub-model, an edge server offloaded calculation sub-model, and a cloud server offloaded calculation sub-model;

[0014] The terminal device local computing sub-model is used to calculate the local processing of the task. and energy consumption

[0015] The communication transmission sub-model is used to calculate the latency and energy consumption of data transmission from the task terminal device to the edge server and cloud server, thus obtaining the transmission latency. and transmission energy consumption

[0016] The edge server offloading computing sub-model addresses the latency issues associated with the transfer of computing tasks from terminal devices to edge servers and cloud servers. Energy consumption And computation time delay and energy consumption

[0017] The cloud server offloading computing sub-model is used to address the latency of transmitting computing tasks from terminal devices to the cloud server. and energy consumption And computation time delay and energy consumption

[0018] Furthermore, a problem model for the unloading task process is constructed based on Markov decision processes, and the specific method is as follows:

[0019] Define the state space s m (t) is represented as:

[0020]

[0021] Among them, L m (t) represents the size of the computation task. This refers to the time the task waits for computation on the terminal device. To calculate the time the task waits for transmission, K(t) represents the number of computing tasks waiting to be executed on the edge servers, and K(t) represents the historical load data of the edge servers.

[0022] Define action space a m (t) is represented as:

[0023]

[0024] in, Represents task Z m (t) The proportion processed locally, α m,n (t)∈[0,1] represents task Z m (t) The proportion of processing offloaded to edge server n. Represents task Z m (t) The proportion of data offloaded to cloud servers for processing, which satisfies the constraints. and

[0025] Mig m,n,C When (t) = 0, it means that the task will not be unloaded a second time and will continue to be executed on the edge server;

[0026] The reward function is defined as the negative of the system cost, i.e.:

[0027]

[0028] In the formula, This represents the total cost.

[0029] Furthermore, total cost The calculation method is as follows:

[0030] The total time delay during the computation task processing is represented by D. total (t) means that:

[0031]

[0032] The total energy consumption during the computation task processing is represented by E. total (t) means that:

[0033]

[0034] Total cost Defined as the weighted sum of processing latency and energy consumption required to complete a computational task, i.e.:

[0035]

[0036] Where λ∈[0,1] represents the latency weighting coefficient of the computation task, and the value of λ is dynamically adjusted according to the load of the edge server.

[0037] Furthermore, the PER-DDPG-BiLSTM algorithm is used to solve the problem model, and the specific method is as follows:

[0038] At the beginning of each time slot t, the agent first obtains the current state information s from the cloud-edge-device collaborative environment. m (t);

[0039] The historical load information K(t) of the edge server is used as the input of BiLSTM. BiLSTM processes the forward and backward timing information through forward and backward LSTM units respectively, and concatenates the hidden states in the two directions to form the final output. The formula used to represent the load trend characteristics of the system is as follows:

[0040]

[0041] Load trend characteristics after BiLSTM processing Input to the Actor network, output a deterministic unloading action a m (t); including the task offloading ratio of terminals, edge servers, and cloud servers, and the decision variable Mig for whether to perform secondary offloading. m,n,C (t);

[0042] Add behavioral noise N t Obtain decision action a m (t)=μ(s m (t))+N t ;

[0043] The agent then uses action a m (t) interacts with the environment and can observe the reward R and the next state s. m (t+1);

[0044] The experience value {s} generated during the interaction i ,a i ,R i ,s i+1 These are stored in the experience pool, and each experience value is assigned a priority.

[0045] At preset intervals, the AI ​​will preferentially sample from the experience pool according to the priority of the experience. The priority is determined based on the magnitude of the TD error, and its calculation formula is as follows:

[0046] δ i =R(t)+γQ target (s m (t+1),μ target (s m (t+1)))-Q(s m (t),a m (t))

[0047] Among them, Q target For the target Critic network, μ target For the target Actor network, γ is the discount factor;

[0048] Experiences with higher priority are more likely to be selected for replay. The priority p(i) of each experience sample can be defined as:

[0049] p(i) = |δ i |+∈

[0050] The sampled small batches of experience are input into the actor network and the critic network for training.

[0051] Furthermore, the sampled small-batch experiences are input into the actor network and critic network for training, as follows:

[0052] The actor's target network μ′ will move the action μ′(s) i+1 The output is fed into the critic's target network Q′ to calculate the target output value, using the following formula:

[0053] y i =r i +γQ′(s i+1 ,μ′(s i+1 |θ μ′ )|θ Q′ )

[0054] Where y i Refers to target value, r i γ refers to the immediate reward, γ refers to the discount factor, i.e., the decay coefficient of future rewards, and θ refers to the immediate reward. μ′ and θ Q′ Refers to the target network parameters for the actor, s i+1 The next state;

[0055] During training, the network parameters θ of the actor network and the critic network need to be updated in real time. μ and θ Q The parameters of the commentator network are updated based on the mean squared error, and its loss function formula is:

[0056]

[0057] Where N is the sample size, Q(s) i ,a i |θ Q ) refers to s i In the state, action a i With parameter θ Q The output of the commentator network;

[0058] The actor network aims to maximize rewards and uses policy gradients to update the current policy, with the following policy formula:

[0059]

[0060] Where N is the number of samples, μ(s|θ) μ () refers to the parameter θ in state s. μ The output of the actor network;

[0061] The agent performs soft updates to the actor target network μ′ and the critic target network Q′ using a constant τ, that is, iteratively updates the actor target network parameters θ in each round. μ′ and θ Q′ The specific formula is as follows:

[0062] θ μ′ ←τθ μ +(1-τ)θ μ′

[0063] θ Q′ ←τθ Q +(1-τ)θ Q′

[0064] Where θ μ and θ Q Refers to the parameters before the update, θ μ′ and θ Q′ Refers to the updated parameters;

[0065] Repeatedly interact with the environment, sample experience, and update network parameters until the preset number of training rounds is reached or the network performance tends to converge.

[0066] Due to the adoption of the above technical solution, this application has the following advantages:

[0067] 1. This invention addresses the task offloading decision optimization problem of terminal devices in highly mobile dynamic environments, focusing on task offloading strategies in cloud-edge-device collaborative scenarios.

[0068] 2. By constructing a cloud-edge-device collaborative system model with dynamic characteristics, an optimization algorithm based on PER-DDPG-BiLSTM is proposed. This algorithm can effectively address the challenges posed by the high mobility of terminal devices and intelligently offload computing tasks to appropriate nodes.

[0069] 3. In response to task execution failures caused by device mobility, the algorithm supports secondary unloading of failed tasks, thereby significantly reducing time delays and energy consumption during task unloading, and improving the overall system performance and user experience.

[0070] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0071] The accompanying drawings of this invention are described below.

[0072] Figure 1 This is a model diagram of a cloud-edge-device collaborative system.

[0073] Figure 2 This is a comparison chart of the task unloading of the four algorithms in Simulation Experiment 1.

[0074] Figure 3 This is a comparison chart of the task unloading of the four algorithms in Simulation Experiment 2. Detailed Implementation

[0075] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0076] A task offloading method for cloud-edge-device collaborative environments, relating to cloud-edge-device collaborative systems, such as... Figure 1 As shown, the cloud-edge-device collaborative system includes cloud servers, edge servers, and terminals. The specific method for unloading tasks within the cloud-edge-device collaborative system is as follows:

[0077] S1. Construct a latency and energy consumption calculation model for the cloud-edge-device collaborative system to execute unloading tasks.

[0078] In step S1, the latency and energy consumption calculation model includes: a terminal device local calculation sub-model, a communication transmission sub-model, an edge server offloaded calculation sub-model, and a cloud server offloaded calculation sub-model;

[0079] The terminal device local computing sub-model is used to calculate the local processing of the task. and energy consumption

[0080] The communication transmission sub-model is used to calculate the latency and energy consumption of data transmission from the task terminal device to the edge server and cloud server, thus obtaining the transmission latency. and transmission energy consumption

[0081] The edge server offloading computing sub-model addresses the latency issues associated with the transfer of computing tasks from terminal devices to edge servers and cloud servers. Energy consumption And computation time delay and energy consumption

[0082] The cloud server offloading computing sub-model is used to address the latency of transmitting computing tasks from terminal devices to the cloud server. and energy consumption And computation time delay and energy consumption

[0083] S2. A problem model for the unloading task process is constructed based on Markov decision processes. The specific method is as follows:

[0084] S21. Define the state space s m (t) is represented as:

[0085]

[0086] Among them, L m (t) represents the size of the computation task. This refers to the time the task waits for computation on the terminal device. To calculate the time the task waits for transmission, K(t) represents the number of computing tasks waiting to be executed on the edge servers, and K(t) represents the historical load data of the edge servers.

[0087] S22, Define the action space a m (t) is represented as:

[0088]

[0089] in, Represents task Z m (t) The proportion processed locally, α m,n (t)∈[0,1] represents task Z m (t) The proportion of processing offloaded to edge server n. Represents task Z m (t) The proportion of data offloaded to cloud servers for processing, which satisfies the constraints. and

[0090] Mig m,n,CWhen (t) = 0, it means that the task will not be unloaded a second time and will continue to be executed on the edge server;

[0091] S23. Define the reward function as the negative of the system cost, that is:

[0092]

[0093] In the formula, This represents the total cost.

[0094] In step S23, the total cost The calculation method is as follows:

[0095] The total time delay during the computation task processing is represented by D. total (t) means that:

[0096]

[0097] The total energy consumption during the computation task processing is represented by E. total (t) means that:

[0098]

[0099] Total cost Defined as the weighted sum of processing latency and energy consumption required to complete a computational task, i.e.:

[0100]

[0101] Where λ∈[0,1] represents the latency weighting coefficient of the computation task, and the value of λ is dynamically adjusted according to the load of the edge server.

[0102] S3. The PER-DDPG-BiLSTM algorithm is used to solve the problem model. The specific method is as follows:

[0103] At the beginning of each time slot t, the agent first obtains the current state information s from the cloud-edge-device collaborative environment. m (t);

[0104] The historical load information K(t) of the edge server is used as the input of BiLSTM. BiLSTM processes the forward and backward timing information through forward and backward LSTM units respectively, and concatenates the hidden states in the two directions to form the final output. The formula used to represent the load trend characteristics of the system is as follows:

[0105]

[0106] Load trend characteristics after BiLSTM processing Input to the Actor network, output a deterministic unloading action a m (t); including the task offloading ratio of terminals, edge servers, and cloud servers, and the decision variable Mig for whether to perform secondary offloading. m,n,C (t);

[0107] Add behavioral noise N t Obtain decision action a m (t)=μ(s m (t))+N t ;

[0108] The agent then uses action a m (t) interacts with the environment and can observe the reward R and the next state s. m (t+1);

[0109] The experience value {s} generated during the interaction i ,a i ,R i ,s i+1 These are stored in the experience pool, and each experience value is assigned a priority.

[0110] At preset intervals, the AI ​​will preferentially sample from the experience pool according to the priority of the experience. The priority is determined based on the magnitude of the TD error, and its calculation formula is as follows:

[0111] δ i =R(t)+γQ target (s m (t+1),μ target (s m (t+1)))-Q(s m (t),a m (t))

[0112] Among them, Q target For the target Critic network, μ target For the target Actor network, γ is the discount factor;

[0113] Experiences with higher priority are more likely to be selected for replay. The priority p(i) of each experience sample can be defined as:

[0114] p(i) = |δ i |+∈

[0115] The sampled small-batch experiences are input into the actor network and the critic network for training;

[0116] The actor's target network μ′ will move the action μ′(s) i+1 The output is fed into the critic's target network Q′ to calculate the target output value, using the following formula:

[0117] y i =r i +γQ′(s i+1 ,μ′(s i+1 |θ μ′ )|θ Q′ )

[0118] During training, the network parameters θ of the actor network and the critic network need to be updated in real time. μ and θ Q The parameters of the commentator network are updated based on the mean squared error, and its loss function formula is:

[0119]

[0120] The actor network aims to maximize rewards and uses policy gradients to update the current policy, with the following policy formula:

[0121]

[0122] The agent performs soft updates to the actor target network μ′ and the critic target network Q′ using a constant τ, that is, iteratively updates the actor target network parameters θ in each round. μ′ and θ Q′ The specific formula is as follows:

[0123] θ μ′ ←τθ μ +(1-τ)θ μ′

[0124] θ Q′ ←τθ Q +(1-τ)θ Q′

[0125] Repeatedly interact with the environment, sample experience, and update network parameters until the preset number of training rounds is reached or the network performance tends to converge.

[0126] S4. Based on the solution results, select to perform the task uninstallation on the server, edge server, or terminal.

[0127] S5. If terminal mobility causes connection interruption or edge server timeout, a secondary offload to the cloud will be triggered. The specific triggering conditions are as follows:

[0128] S51. If the task is not completed in time slot t and the connection state is Conn m,n If (t) = 0, the connection is interrupted, triggering a secondary unloading;

[0129] S52, If the edge server execution latency There is a risk of timeout, triggering a second uninstallation.

[0130] Simulation Experiment 1:

[0131] The algorithm's performance is evaluated by observing its behavior in response to changes in the number of terminal devices. An increase in the number of terminal devices significantly increases the system's scale and complexity, which burdens edge servers and the network. The proposed method is compared with three traditional algorithms.

[0132] The results are as follows Figure 2 As shown, with the number of devices gradually increasing, the method proposed in this invention outperforms the other three algorithms in key indicators such as total system cost, total latency, total energy consumption, and task drop rate, demonstrating its task scheduling and resource optimization capabilities in high-concurrency scenarios.

[0133] Simulation Experiment 2:

[0134] The task arrival rate directly reflects the system's workload. Increasing the task arrival rate allows us to test the algorithm's processing capacity under high load conditions, observe whether it can efficiently allocate computing resources, and ensure tasks are completed on time. The proposed method is compared with three traditional algorithms under varying task arrival rates.

[0135] The results are as follows Figure 3 As shown, with the task arrival rate gradually increasing, the method proposed in this invention outperforms the other three algorithms in key indicators such as total system cost, total latency, total energy consumption, and task drop rate, fully demonstrating its task scheduling and resource optimization capabilities under high load scenarios.

[0136] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A task offloading method for a cloud-edge-device collaborative environment, comprising a cloud-edge-device collaborative system consisting of a cloud server, an edge server, and a terminal, characterized in that, The specific method for unloading tasks from the cloud-edge-device collaborative system is as follows: Construct a latency and energy consumption calculation model for the execution of offloading tasks in a cloud-edge-device collaborative system; A problem model for the unloading task process is constructed based on Markov decision processes. The PER-DDPG-BiLSTM algorithm is used to solve the problem model; Based on the solution results, choose to perform the task uninstallation on the server, edge server, or terminal.

2. The task offloading method for a cloud-edge-device collaborative environment as described in claim 1, characterized in that, If connection interruption is caused by terminal mobility or edge server timeout, a secondary offload to the cloud will be triggered. The specific triggering conditions are as follows: If the task is not completed in time slot t and the connection state is Conn m,n If (t) = 0, the connection is interrupted, triggering a second unloading; If the edge server execution latency There is a risk of timeout, triggering a second uninstallation.

3. The task offloading method for a cloud-edge-device collaborative environment as described in claim 1, characterized in that, The latency and energy consumption calculation model includes: a terminal device local calculation sub-model, a communication transmission sub-model, an edge server offloaded calculation sub-model, and a cloud server offloaded calculation sub-model; The terminal device local computing sub-model is used to calculate the local processing of the task. and energy consumption The communication transmission sub-model is used to calculate the latency and energy consumption of data transmission from the task terminal device to the edge server and cloud server, thus obtaining the transmission latency. and transmission energy consumption The edge server offloading computing sub-model addresses the latency issues associated with the transfer of computing tasks from terminal devices to edge servers and cloud servers. Energy consumption And computation time delay and energy consumption The cloud server offloading computing sub-model addresses the latency of transmitting computing tasks from terminal devices to the cloud server. and energy consumption And computation time delay and energy consumption 4. The task offloading method for a cloud-edge-device collaborative environment as described in claim 3, characterized in that, A problem model for the unloading task process is constructed based on Markov decision processes, and the specific method is as follows: Define the state space s m (t) is represented as: Among them, L m (t) represents the size of the computation task. This refers to the time the task waits for computation on the terminal device. To calculate the time the task waits for transmission, K(t) represents the number of computing tasks waiting to be executed on the edge servers, and K(t) represents the historical load data of the edge servers. Define action space a m (t) is represented as: in, Represents task Z m (t) The proportion processed locally, α m,n (t)∈[0,1] represents task Z m (t) The proportion of processing offloaded to edge server n. Represents task Z m (t) The proportion of data offloaded to cloud servers for processing, which satisfies the constraints. and Mig m,n,C When (t) = 0, it means that the task will not be unloaded a second time and will continue to be executed on the edge server; The reward function is defined as the negative of the system cost, i.e.: In the formula, This represents the total cost.

5. The task offloading method for a cloud-edge-device collaborative environment as described in claim 4, characterized in that, Total cost The calculation method is as follows: The total time delay during the computation task processing is represented by D. total (t) means that: The total energy consumption during the computation task processing is represented by E. total (t) means that: Total cost Defined as the weighted sum of processing latency and energy consumption required to complete a computational task, i.e.: Where λ∈[0,1] represents the latency weighting coefficient of the computation task, and the value of λ is dynamically adjusted according to the load of the edge server.

6. The task offloading method for a cloud-edge-device collaborative environment as described in claim 4, characterized in that, The PER-DDPG-BiLSTM algorithm is used to solve the problem model. The specific method is as follows: At the beginning of each time slot t, the agent first obtains the current state information s from the cloud-edge-device collaborative environment. m (t); The historical load information K(t) of the edge server is used as the input of BiLSTM. BiLSTM processes the forward and backward timing information through forward and backward LSTM units respectively, and concatenates the hidden states in the two directions to form the final output. The formula used to represent the load trend characteristics of the system is as follows: Load trend characteristics after BiLSTM processing Input to the Actor network, output a deterministic unloading action a m (t); including the task offloading ratio of terminals, edge servers, and cloud servers, and the decision variable Mig for whether to perform secondary offloading. m,n,C (t); Add behavioral noise N t Obtain decision action a m (t)=μ(s m (t))+N t ; The agent then uses action a m (t) interacts with the environment and can observe the reward R and the next state s. m (t+1); The experience value generated during the interaction {s i ,a i ,R i ,s i+1 These are stored in the experience pool, and each experience value is assigned a priority. At preset intervals, the AI ​​will preferentially sample from the experience pool according to the priority of the experience. The priority is determined based on the magnitude of the TD error, and its calculation formula is as follows: δ i =R(t)+γQ target (s m (t+1),μ target (s m (t+1)))-Q(s m (t),a m (t)) Among them, Q target For the target Critic network, μ target For the target Actor network, γ is the discount factor; Experiences with higher priority are more likely to be selected for replay. The priority p(i) of each experience sample can be defined as: p(i)=|δ i |+∈ The sampled small batches of experience are input into the actor network and the critic network for training.

7. The task offloading method for a cloud-edge-device collaborative environment as described in claim 6, characterized in that, The sampled small-batch experience is input into the actor network and the critic network for training, as follows: The actor's target network μ′ will move the action μ′(s) i+1 The output is fed into the critic's target network Q′ to calculate the target output value, using the following formula: y i =r i +γQ′(s i+1 ,μ′(s i+1 |θ μ′ )|θ Q′ ) Where y i Refers to target value, r i γ refers to the immediate reward, γ refers to the discount factor, i.e., the decay coefficient of future rewards, and θ refers to the immediate reward. μ′ and θ Q′ Refers to the target network parameters for the actor, s i+1 The next state; During training, the network parameters θ of the actor network and the critic network need to be updated in real time. μ and θ Q The parameters of the commentator network are updated based on the mean squared error, and its loss function formula is: Where N is the sample size, Q(s) i ,a i |θ Q ) refers to s i In the state, action a i With parameter θ Q The output of the commentator network; The actor network aims to maximize rewards and uses policy gradients to update the current policy, with the following policy formula: Where N is the number of samples, μ(s|θ) μ () refers to the parameter θ in state s. μ The output of the actor network; The agent performs soft updates to the actor target network μ′ and the critic target network Q′ using a constant τ, that is, iteratively updates the actor target network parameters θ in each round. μ′ and θ Q′ The specific formula is as follows: i μ′ ←tth μ +(1-τ)θ μ′ i Q′ ←tth Q +(1-τ)θ Q′ Where θ μ and θ Q Refers to the parameters before the update, θ μ′ and θ Q′ Refers to the updated parameters; Repeatedly interact with the environment, sample experience, and update network parameters until the preset number of training rounds is reached or the network performance tends to converge.

Images