Cache-assisted satellite task unloading and resource allocation method

Abstract

The invention discloses a cache-assisted satellite task unloading and resource allocation method, and the key points of the technical scheme are that the method comprises the following steps: S1, building a system model which comprises the steps: building a user unloading model in a satellite-ground fusion network, building an energy consumption model, and building a cache utility model; s2, establishing an optimization problem, taking minimization of system cache utility as a target function, and defining constraint conditions; s3, converting the defined optimization problem into a Markov tuple, wherein the Markov tuple comprises a state space, an action space and a reward function; s4, a method based on deep reinforcement learning is provided for training, and a final unloading strategy is obtained; according to the method, computing resources in the low-orbit satellite network are effectively utilized by introducing the cache technology, the task unloading process is optimized, dependence on a satellite communication link can be relieved, data transmission delay is reduced, the response speed and stability of the system are improved, and the unloading efficiency of the computing edge system in the satellite-ground fusion network is improved.

Description

Technical Field

[0001] The present invention relates to the field of satellite-ground fusion network communication technology, and in particular to a cache-assisted satellite task unloading and resource allocation method. Background Art

[0002] In low-Earth orbit satellite networks, the application of mobile edge computing has a significant impact on network performance and user experience. Traditional cloud computing models often concentrate computing and data processing tasks in data centers far away from end-user devices, resulting in high data transmission latency and limitations on applications with strict real-time requirements. On the other hand, MEC extends computing power to the edge of the network, bringing data processing closer to end users, effectively reducing latency and enhancing real-time performance. In LEO satellite systems, deploying MEC enables satellites to flexibly perform data processing tasks, thereby improving the efficiency and performance of communication networks.

[0003] However, mobile edge computing also presents some challenges and disadvantages. First, the deployment of MEC needs to consider the resource constraints of edge devices, as these devices usually have limited computing and storage capacity. Second, due to the instability of satellite communication links, MEC in LEO satellite networks may be susceptible to fluctuations in communication quality, resulting in performance variations. In addition, for some large-scale data processing tasks, the computing power of edge devices may be insufficient and require more powerful central processing units to support them. To solve the problem of limited computing resources in the mobile edge computing system in the above-mentioned satellite-ground fusion network, we propose a cache-assisted satellite task offloading and resource allocation method. Summary of the invention

[0004] In view of the deficiencies of the prior art, the present invention provides a cache-assisted satellite task offloading and resource allocation method to solve the problems raised in the background technology.

[0005] The above technical objectives of the present invention are achieved through the following technical solutions:

[0006] A cache-assisted satellite task offloading and resource allocation method comprises the following steps:

[0007] S1. Establishing a system model, which includes establishing a user unloading model in a satellite-ground fusion network, establishing an energy consumption model, and establishing a cache utility model;

[0008] S2. Establish an optimization problem with minimizing the system cache utility as the objective function and define constraints;

[0009] S3, transforming the defined optimization problem into a Markov tuple, including state space, action space and reward function;

[0010] S4. A deep reinforcement learning-based training method is proposed to obtain the final unloading strategy.

[0011] Furthermore, in S1, the user unloading model in the satellite-ground fusion network is:

[0012] Consider 66 low-orbit satellites distributed in 6 orbits. Each satellite establishes satellite links with the 4 satellites around it, and is connected to the front and rear adjacent satellites in the same orbital plane, and is connected to the two satellites in the adjacent planes on both sides. Each satellite is deployed with a MEC server and has caching capabilities. The results of caching tasks can improve service efficiency. The data packets that need to be processed come from the ground and are processed by access satellites. It is chosen to be processed on the access satellite or offloaded to its adjacent satellites for processing. We consider a multi-user cache MEC system with N users, N = {1,...,N}, and the offloading decision vector of task n is α n Indicates that, where α n ∈{0,1},α n =1 means the user is offloaded to the access satellite, α n =0 means the task is offloaded to the adjacent satellite.

[0013] Furthermore, in S1, the energy consumption model is:

[0014] The computing resources and cache resources are divided into a resource pool to provide a unified resource granularity for the system. The cache capacity of each satellite is represented by C1, and the computing capacity is represented by C2. Assuming that the computing capacity of each user is limited, each user has an inseparable computing-intensive task to be processed. The task of user n and its characteristic parameters are represented by M n ={X i，n , X j，n , X r，n}, where X i，n Indicates the size of the task, X j，n represents the computing power required by the soft task, that is, the number of CPU cycles, X r，n Indicates the size of the calculation result in bits.

[0015] Among them, if task n is unloaded when accessing the satellite, the energy consumption generated is:

[0016]

[0017] Among them, R u and R d They represent the uplink and downlink transmission rates of the task, respectively, and p u and p d represents the uplink and downlink transmission power, ∈ m represents the energy cost, γ nrepresents the task processing density, E up , E down , E mec They represent uplink energy consumption, downlink energy consumption, and MEC computing energy consumption respectively;

[0018] If unloading is done on other satellites, the energy consumption is Among them, R ISL represents the intersatellite transmission rate of the mission, P ISL Indicates the satellite transmit power for the mission.

[0019] Furthermore, in S1, the cache utility model is:

[0020] The cache uses the user task cache factor Q = q1, q2, ..., q n express;

[0021] Among them, q n ∈{0, 1},q n =1 indicates that task n is cached, otherwise it indicates that n is not cached;

[0022] After the task is cached, the average utility it brings can be expressed as:

[0023]

[0024] For satellite access, use U acc represents the average utility of the cache when offloading to the access satellite, U acc It can be expressed as:

[0025]

[0026] For adjacent satellites, use U adj represents the average utility of the cache when offloading to adjacent satellites, U adj It can be expressed as:

[0027]

[0028] Furthermore, in S2, the optimization problem and constraint conditions are established as follows:

[0029]

[0030]

[0031] Among them, constraint C1 means that all tasks may be cached by the node, and the cumulative cache quantity of all tasks cannot exceed the basic cache space S1 of the satellite node. Constraint C2 means that the offloading decision of all tasks has two possibilities 0 and 1, and the tasks offloaded to each node cannot exceed the computing resource threshold S2 of the satellite node.

[0032] Furthermore, in S3, the Markov tuple is established as:

[0033] The offloading process of the LEO network is approximated as an MDP problem. A typical MDP can be expressed as a tuple (X, A, P, R). In the MDP problem, the state corresponding to the t-th time slot is denoted as x(t), and the corresponding action is denoted as a(t), where a(t)∈A n (t), where A n (p) is the set of allowed actions, R is the state x(t), the action is the reward function of a(t), we use y to represent the network state x in the next stage, and the state transition probability is P xy (a), the recursive equation can be written as:

[0034]

[0035] Ω t (x(t)) is the value evaluation function, and Q(x,u) is defined as the value function of state x and action u. Then Q(x,u) evaluated by Ω is:

[0036]

[0037] Let Q * (x, u) is the optimal Q value for state x and action u, u(t)∈U x (t), the best evaluation function Ω * (x) is:

[0038]

[0039] Let Q t (x, u) represents Q * The estimate of (x, u) at time slice t, according to (23), Ω * The estimate of (x) at time slice t is:

[0040]

[0041] Furthermore, in S4, the method based on deep reinforcement learning is:

[0042] Two neural networks are used in the DRL method, one is the main neural network representing the Q function, denoted as Q(x, u; θ), θ represents the weight of the main neural network, and the other is the target neural network, which generates the target Q value y′ j :

[0043]

[0044] θ′ is the parameter of the target network. In order to stabilize the training, y′ is calculated jDuring this period, θ′ is a fixed value and is updated after one training session. We minimize the loss function by continuously updating it and train the Q function according to the Q value of the target neural network. The loss function Loss(θ) is:

[0045]

[0046] The weight θ′ of the target neural network j According to the most recent θ j to update.

[0047] Furthermore, in S4, in order to eliminate the divergence or oscillation caused by the strong correlation between consecutive transitions in the approximation of the Q function, the DRL method uses experience playback to convert the empirical feature vector (x t ,u t , F t , x t+1 ) is stored in the playback experience pool D = {D1, D2, ..., D t Then, DRL-Agent randomly extracts small batches of data from the experience pool to train the parameters of the DNN, which makes the reuse of experience data improve the sample efficiency and accelerate the convergence of the algorithm.

[0048] Furthermore, in S4, the DRL-based algorithm is mainly divided into two stages: an offline training stage and an online deep Q-learning stage:

[0049] In the offline training phase, the DNN is constructed to obtain the correlation between each state-action pair (x, u) of the controlled system and its value function Q(x, u). Specifically, the computational mode selection and the resource allocation of the cache-assisted satellite network with a random strategy in a sufficiently long time are preprocessed first, and then the state transition profile and the corresponding estimated Q(x, u) are obtained and stored in a capacity C. D Finally, the DNN is preprocessed using the input state-action pair (x, u) and the output Q(x, u);

[0050] Among them, the online deep Q-learning learning stage, the schematic diagram of the online learning model based on the deep reinforcement learning method, uses the deep Q-learning method to update the selected action and Q value. Specifically, in each episode, the DRL agent first starts to observe the network state x1, and uses the ε-greedy strategy to select the execution action u k , randomly select actions when using ε probability, otherwise, by (x t ,u t ) input, get the maximum Q value estimate from the DNN network to select the action, and execute action uk When the reward value F is obtained from the neural network t and the next state x t+1 When the eigenvector (x t ,u t , F t , x t+1 ) will be stored in the experience pool D. After calculating the target Q value, DRL-Agent uses the stochastic gradient descent method to calculate the loss function Loss(θ) and updates the weight θ until convergence.

[0051] In summary, the present invention mainly has the following beneficial effects:

[0052] The present invention effectively utilizes the computing resources in the low-orbit satellite network by introducing caching technology, and optimizes the task offloading process. This optimization not only reduces the waste of computing resources, but also reduces system latency, thereby improving the overall offloading efficiency of the edge computing system in the satellite-ground fusion network. Therefore, it can alleviate the dependence on satellite communication links, reduce data transmission latency, improve the response speed and stability of the system, and increase the offloading efficiency of the computing edge system in the satellite-ground fusion network. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 is a schematic diagram of a cache auxiliary resource allocation model of the present invention;

[0054] Figure 2 It is a schematic diagram of the online learning model of the deep reinforcement learning based method of the present invention. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings of the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the described embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the scope of protection of the present invention.

[0056] The following examples are used to illustrate the present invention, but they cannot be used to limit the scope of protection of the present invention. The conditions in the examples can be further adjusted according to specific conditions. Simple improvements to the method of the present invention under the premise of the concept of the present invention belong to the scope of protection claimed in the present invention.

[0057] Example 1

[0058] refer to Figure 1-Figure 2 , this embodiment provides a cache-assisted satellite task offloading and resource allocation model, including the following steps:

[0059] S1. Establishing a system model, which includes establishing a user unloading model in a satellite-ground fusion network, establishing an energy consumption model, and establishing a cache utility model;

[0060] S2. Establish an optimization problem with minimizing the system cache utility as the objective function and define constraints.

[0061] The user unloading model in the S1 satellite-ground fusion network is:

[0062] Consider 66 low-orbit satellites, distributed in 6 orbits. Each satellite establishes satellite links with the 4 satellites around it, and is connected to the front and rear adjacent satellites in the same orbital plane, and is connected to the two satellites in the adjacent planes on both sides. Each satellite is deployed with a MEC server and has caching capabilities. The results of caching tasks can improve service efficiency. The data packets that need to be processed come from the ground and are processed by access satellites. It is chosen to be processed on the access satellite or offloaded to its adjacent satellites for processing. We consider a multi-user cache MEC system with N users, N = {1, ..., N}, and the offloading decision vector of task n is α n Indicates that, where α n ∈{0,1},α n =1 means the user is offloaded to the access satellite, α n =0 means the task is offloaded to the adjacent satellite.

[0063] The S1 energy consumption model is:

[0064] The computing resources and cache resources are divided into a resource pool to provide a unified resource granularity for the system. The cache capacity of each satellite is represented by C1, and the computing capacity is represented by C2. Assuming that the computing capacity of each user is limited, each user has an inseparable computing-intensive task to be processed. The task of user n and its characteristic parameters are represented by M n ={X i，n , X j，n , X r，n}, where X i，n Indicates the size of the task, X j，n represents the computing power required by the soft task, that is, the number of CPU cycles, X r，n Indicates the size of the calculation result in bits.

[0065] Among them, if task n is unloaded when accessing the satellite, the energy consumption generated is:

[0066]

[0067] Among them, R u and R d They represent the uplink and downlink transmission rates of the task, respectively, and p u and pd represents the uplink and downlink transmission power, ∈ m represents the energy cost, γ n represents the task processing density, E up , E down , E mec They represent uplink energy consumption, downlink energy consumption, and MEC computing energy consumption respectively;

[0068] If unloading is done on other satellites, the energy consumption is Among them, R ISL represents the intersatellite transmission rate of the mission, P ISL Indicates the satellite transmit power for the mission.

[0069] The S1 cache utility model is:

[0070] The cache uses the user task cache factor Q = q1, q2, ..., q n express;

[0071] Among them, q n ∈{0, 1},q n =1 indicates that task n is cached, otherwise it indicates that n is not cached;

[0072] After the task is cached, the average utility it brings can be expressed as:

[0073]

[0074] For satellite access, use U acc represents the average utility of the cache when offloading to the access satellite, U acc It can be expressed as:

[0075]

[0076] For adjacent satellites, use U adj represents the average utility of the cache when offloading to adjacent satellites, U adj It can be expressed as:

[0077]

[0078] The S2 optimization problem and constraints are established as follows:

[0079]

[0080]

[0081] Among them, constraint C1 means that all tasks may be cached by the node, and the cumulative cache quantity of all tasks cannot exceed the basic cache space S1 of the satellite node. Constraint C2 means that the offloading decision of all tasks has two possibilities 0 and 1, and the tasks offloaded to each node cannot exceed the computing resource threshold S2 of the satellite node.

[0082] Example 2

[0083] refer to Figure 1-Figure 2 This embodiment provides a cache-assisted satellite task offloading and resource allocation method, including:

[0084] S3, transforming the defined optimization problem into a Markov tuple, including state space, action space and reward function;

[0085] S4. A deep reinforcement learning-based training method is proposed to obtain the final unloading strategy.

[0086] The Markov tuple in S3 is established as:

[0087] The offloading process of the LEO network is approximated as an MDP problem. A typical MDP can be expressed as a tuple (X, A, P, R). In the MDP problem, the state corresponding to the t-th time slot is denoted as x(t), and the corresponding action is denoted as a(t), where a(t)∈A n (t), where A n (p) is the set of allowed actions, R is the state x(t), the action is the reward function of a(t), we use y to represent the network state x in the next stage, and the state transition probability is P xy (a), the recursive equation can be written as:

[0088]

[0089] Ω t (x(t)) is the value evaluation function, and Q(x,u) is defined as the value function of state x and action u. Then Q(x,u) evaluated by Ω is:

[0090]

[0091] Let Q * (x, u) is the optimal Q value for state x and action u, u(t)∈U x (t), the best evaluation function Ω * (x) is:

[0092]

[0093] Let Q t (x, u) represents Q *The estimate of (x, u) at time slice t, according to (23), Ω * The estimate of (x) at time slice t is:

[0094]

[0095] The method based on deep reinforcement learning in S4 is:

[0096] Two neural networks are used in the DRL method, one is the main neural network representing the Q function, denoted as Q(x, u; θ), θ represents the weight of the main neural network, and the other is the target neural network, which generates the target Q value y′ j :

[0097]

[0098] θ′ is the parameter of the target network. In order to stabilize the training, y′ is calculated j During this period, θ′ is a fixed value and is updated after one training session. We minimize the loss function by continuously updating it and train the Q function according to the Q value of the target neural network. The loss function Loss(θ) is:

[0099]

[0100] The weight θ′ of the target neural network j According to the most recent θ j to update.

[0101] In order to eliminate the divergence or oscillation caused by the strong correlation between consecutive transitions in the approximation of the Q function, the DRL method uses experience replay to convert the empirical feature vector (x t ,u t , F t , x t+1 ) is stored in the playback experience pool D = {D1, D2, ..., D t Then, DRL-Agent randomly extracts small batches of data from the experience pool to train the parameters of the DNN, which makes the reuse of experience data improve the sample efficiency and accelerate the convergence of the algorithm.

[0102] The DRL-based algorithm is mainly divided into two stages: offline training stage and online deep Q-learning stage:

[0103] In the offline training phase, the DNN is constructed to obtain the correlation between each state-action pair (x, u) of the controlled system and its value function Q(x, u). Specifically, the computational mode selection and the resource allocation of the cache-assisted satellite network with a random strategy in a sufficiently long time are preprocessed first, and then the state transition profile and the corresponding estimated Q(x, u) are obtained and stored in a capacity C. D Finally, the DNN is preprocessed using the input state-action pair (x, u) and the output Q(x, u);

[0104] Among them, the online deep Q-learning learning stage, the schematic diagram of the online learning model based on the deep reinforcement learning method, uses the deep Q-learning method to update the selected action and Q value. Specifically, in each episode, the DRL agent first starts to observe the network state x1, and uses the ε-greedy strategy to select the execution action u k , randomly select actions when using ε probability, otherwise, by (x t ,u t ) input, get the maximum Q value estimate from the DNN network to select the action, and execute action u k When the reward value F is obtained from the neural network t and the next state x t+1 When the eigenvector (x t ,u t , F t , x t+1 ) will be stored in the experience pool D. After calculating the target Q value, DRL-Agent uses the stochastic gradient descent method to calculate the loss function Loss(θ) and updates the weight θ until convergence.

[0105] Although the embodiments of the present invention have been shown and described, it is understandable to those skilled in the art that, unless otherwise defined, the technical terms or scientific terms used in the present invention shall have the usual meanings understood by persons with ordinary skills in the field to which the present invention belongs. The words "including" or "comprising" and the like used in the present invention mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. The words "connect" or "connected" and the like are not limited to physical or mechanical connections, but may also include electrical connections, whether direct or indirect. "Up", "down", "left", "right", etc. are only used to indicate relative position relationships. When the absolute position of the described object changes, the relative position relationship may also change accordingly.

[0106] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A cache-assisted satellite task offloading and resource allocation method, characterized in that: The following steps are involved: S1. Establishing a system model, which includes establishing a user unloading model in a satellite-ground fusion network, establishing an energy consumption model, and establishing a cache utility model; S2. Establish an optimization problem with minimizing the system cache utility as the objective function and define constraints; S3, transforming the defined optimization problem into a Markov tuple, including state space, action space and reward function; S4. A deep reinforcement learning-based training method is proposed to obtain the final unloading strategy.

2. A cache-assisted satellite task offloading and resource allocation method according to claim 1, characterized in that: In S1, the user unloading model in the satellite-ground fusion network is: Consider 66 low-orbit satellites distributed in 6 orbits. Each satellite establishes satellite links with the 4 satellites around it, and is connected to the front and rear adjacent satellites in the same orbital plane, and is connected to the two satellites in the adjacent planes on both sides. Each satellite is deployed with a MEC server and has caching capabilities. The results of caching tasks can improve service efficiency. The data packets that need to be processed come from the ground and are processed by access satellites. It is chosen to be processed on the access satellite or offloaded to its adjacent satellites for processing. We consider a multi-user cache MEC system with N users, N = {1,...,N}, and the offloading decision vector of task n is α n Indicates that, where α n ∈{0,1},α n =1 means the user is offloaded to the access satellite, α n =0 means the task is offloaded to the adjacent satellite.

3. A cache-assisted satellite task offloading and resource allocation method according to claim 1, characterized in that: In S1, the energy consumption model is: The computing resources and cache resources are divided into a resource pool to provide a unified resource granularity for the system. The cache capacity of each satellite is represented by C1, and the computing capacity is represented by C2. Assuming that the computing capacity of each user is limited, each user has an inseparable computing-intensive task to be processed. The task of user n and its characteristic parameters are represented by M n ={X i,n ,X j,n ,X r,n }, where X i,n Indicates the size of the task, X j,n represents the computing power required by the soft task, that is, the number of CPU cycles, X r,n Indicates the size of the calculation result in bits.

4. A cache-assisted satellite task offloading and resource allocation method according to claim 1, characterized in that: In S1, the cache utility model is: The cache uses the user task cache factor express; Among them, q n ∈{0,1},q n =1 indicates that task n is cached, otherwise it indicates that n is not cached; After the task is cached, the average utility it brings can be expressed as: For satellite access, use U acc represents the average utility of the cache when offloading to the access satellite, U acc It can be expressed as: For adjacent satellites, use U adj represents the average utility of the cache when offloading to adjacent satellites, U adj It can be expressed as:

5. The cache-assisted satellite task offloading and resource allocation method according to claim 1, characterized in that: In S2, the optimization problem and constraints are established as follows: P1: s.t.C1: C2: Among them, constraint C1 means that all tasks may be cached by the node, and the cumulative cache quantity of all tasks cannot exceed the basic cache space S1 of the satellite node. Constraint C2 means that the offloading decision of all tasks has two possibilities 0 and 1, and the tasks offloaded to each node cannot exceed the computing resource threshold S2 of the satellite node.

6. A cache-assisted satellite task offloading and resource allocation method according to claim 1, characterized in that: In S3, the Markov tuple is established as: The offloading process of the LEO network is approximated as an MDP problem. A typical MDP can be expressed as a tuple (X, A, P, R). In the MDP problem, the state corresponding to the t-th time slot is denoted as x(t), and the corresponding action is denoted as a(t), a(t)∈A n (t), where A n (p) is the set of allowed actions, R is the state x(t), the action is the reward function of a(t), we use y to represent the network state x in the next stage, and the state transition probability is P xy (a), the recursive equation can be written as: Ω t (x(t)) is the value evaluation function, and Q(x,u) is defined as the value function of state x and action u. Then Q(x,u) evaluated by Ω is: Let Q * (x,u) is the optimal Q value for state x and action u, u(t)∈U x (t), the best evaluation function Ω * (x) is: Let Q t (x,u) represents Q * The estimate of (x,u) at time slice t, according to (23), Ω * The estimate of (x) at time slice t is:

7. A cache-assisted satellite task offloading and resource allocation method according to claim 1, characterized in that: In S4, the method based on deep reinforcement learning is: Two neural networks are used in the DRL method, one is the main neural network representing the Q function, denoted as Q(x,u;θ), θ represents the weight of the main neural network, and the other is the target neural network, which generates the target Q value y′ j : θ′ is the parameter of the target network. In order to stabilize the training, y′ is calculated j During this period, θ′ is a fixed value and is updated after one training session. We minimize the loss function by continuously updating it and train the Q function according to the Q value of the target neural network. The loss function Loss(θ) is: The weight θ′ of the target neural network j According to the most recent θ j to update.

8. The cache-assisted satellite task offloading and resource allocation method according to claim 1, characterized in that: In S4, in order to eliminate the divergence or oscillation caused by the strong correlation between consecutive transitions in the approximation of the Q function, the DRL method uses experience playback to convert the empirical feature vector (x t ,u t ,F t ,x t+1 ) is stored in the playback experience pool D = {D1, D2, ..., D t Then, DRL-Agent randomly extracts small batches of data from the experience pool to train the parameters of the DNN, which makes the reuse of experience data improve the sample efficiency and accelerate the convergence of the algorithm.

9. The cache-assisted satellite task offloading and resource allocation method according to claim 1, characterized in that: In S4, the DRL-based algorithm is mainly divided into two stages: offline training stage and online deep Q-learning stage: In the offline training phase, the DNN is constructed to obtain the correlation between each state-action pair (x,u) of the controlled system and its value function Q(x,u). Specifically, the computational mode selection and the resource allocation of the cache-assisted satellite network with a random strategy for a sufficiently long time are preprocessed first, and then the state transition profile and the corresponding estimated Q(x,u) are obtained and stored in a capacity C D Finally, the DNN is preprocessed using the input state-action pair (x,u) and the output Q(x,u); Among them, the online deep Q-learning learning stage, the schematic diagram of the online learning model based on the deep reinforcement learning method, uses the deep Q-learning method to update the selected action and Q value. Specifically, in each episode, the DRL agent first starts to observe the network state x1, and uses the ε-greedy strategy to select the execution action u k , randomly select actions when using ε probability, otherwise, by (x t ,u t ) input, get the maximum Q value estimate from the DNN network to select the action, and execute action u k When the reward value F is obtained from the neural network t and the next state x t+1 When the eigenvector (x t ,u t ,F t ,x t+1 ) will be stored in the experience pool D. After calculating the target Q value, DRL-Agent uses the stochastic gradient descent method to calculate the loss function Loss(θ) and updates the weight θ until convergence.

Images