A DQN-based service offloading method and terminal

Abstract

The present invention discloses a service offloading method and terminal based on DQN, and establishes an optimization model for the average value of the peak value of information age for information communication between a sensor and an edge computing node; when it is detected that the sensor needs to perform service offloading, the optimization model is solved by DQN to obtain a solution result; according to the number of sub-channels to be allocated to the sensor and the number of computing resources in the solution result, a channel is configured for the sensor to perform service offloading; the present invention proposes an optimization problem of the minimum average peak value AoI, and introduces a deep Q network (DQN) to solve the problem, which can effectively reduce the average peak value AoI, thereby improving the service quality of the distribution network.

Description

Technical Field

[0001] The present invention relates to the field of information and communication technology, and in particular to a DQN-based service unloading method and terminal. Background Art

[0002] With the popularization of electric vehicles, dynamic loads will also be widely connected to the power grid, and the distribution network will be transformed from traditional one-way power transmission to two-way power flow. The topology of the active distribution network (ADN) changes with the access of distributed power sources and the connection of electric vehicle loads. The access of a large number of distributed generation (DG) and the connection of electric vehicle (EV) loads to the grid make the traditional one-way radial distribution system have "active" characteristics. However, DG and EV usually have the disadvantages of small capacity, randomness and high volatility. The access of a large number of DG and EV to the distribution system will bring great challenges to the operation and control of the system. Especially under extreme disasters with low probability and high loss, the actual power outage situation and the surrounding environment are highly uncertain and complex, which puts higher requirements on the operation optimization of ADN.

[0003] The distribution network equipment is large in scale, changes rapidly and develops unevenly. The multi-source, heterogeneous and scaled information that characterizes the state of the distribution network requires a lot of computing in the process of information feature extraction and fusion to complete the operation and maintenance of the distribution network equipment. Especially with the application of artificial intelligence technology in the operation and maintenance of distribution networks, the requirements for computing resources will become increasingly higher. Multi-Access Edge Computing (MEC) can sink storage data and computing resources to the edge of the network, thereby reducing transmission delays.

[0004] However, in the operation and maintenance of distribution network equipment, low latency and high throughput cannot guarantee the freshness of information acquisition, thereby reducing the accuracy and reliability of distribution network equipment status analysis; the characterization indicator of information freshness is the age of information (AoI), that is, the time when the receiving end receives the information (calculated from the time when the information is generated by the sending end), which is affected by communication throughput and transmission delay; at the same time, since the time-varying characteristics of the wireless channel need to be considered in the communication collection and computing network of distribution network equipment, it is necessary to optimize the communication resources in the random network, improve the computing efficiency of MEC, improve the accuracy of distribution network equipment status analysis, and effectively improve the stability of the distribution network. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a DQN-based service unloading method and terminal, which can effectively reduce the average peak AoI and improve the service quality of the distribution network.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0007] A service unloading method based on DQN, comprising the steps of:

[0008] S1. Establish an optimization model for the average value of the peak age of information for information communication between sensors and edge computing nodes;

[0009] S2. When it is detected that the sensor needs to offload services, the optimization model is solved by DQN to obtain a solution result;

[0010] S3. According to the number of sub-channels to be allocated to the sensor and the number of computing resources in the solution result, channels are configured for the sensor to perform service unloading.

[0011] In order to solve the above technical problems, another technical solution adopted by the present invention is:

[0012] A DQN-based service unloading terminal includes a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0013] S1. Establish an optimization model for the average value of the peak age of information for information communication between sensors and edge computing nodes;

[0014] S2. When it is detected that the sensor needs to offload services, the optimization model is solved by DQN to obtain a solution result;

[0015] S3. According to the number of sub-channels to be allocated to the sensor and the number of computing resources in the solution result, channels are configured for the sensor to perform service unloading.

[0016] The beneficial effects of the present invention are as follows: a DQN-based service offloading method and terminal of the present invention proposes an optimization problem of minimum average peak AoI, and introduces a deep Q network (DQN) to solve the problem, which can effectively reduce the average peak AoI, thereby improving the service quality of the distribution network. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A flow chart of a DQN-based service unloading method according to an embodiment of the present invention;

[0018] Figure 2 A structural diagram of a DQN-based service offloading terminal according to an embodiment of the present invention;

[0019] Figure 3 A schematic diagram of a state update and control system model for computing offloading of a DQN-based service offloading method according to an embodiment of the present invention;

[0020] Figure 4 A schematic diagram of AoI evolution of a DQN-based service offloading method according to an embodiment of the present invention;

[0021] Figure 5 A schematic diagram of average peak AoI of a DQN-based service offloading method according to an embodiment of the present invention and various existing service offloading methods when the computing resource is 5;

[0022] Figure 6 A schematic diagram of the number of sensors vs. average peak AoI of a DQN-based service offloading method according to an embodiment of the present invention and various existing service offloading methods;

[0023] Description of labels:

[0024] 1. A service offloading terminal based on DQN; 2. Processor; 3. Memory. DETAILED DESCRIPTION

[0025] In order to explain the technical content, achieved objectives and effects of the present invention in detail, the following is an explanation in combination with the implementation modes and the accompanying drawings.

[0026] Please refer to Figure 1 , a service unloading method based on DQN, comprising the steps of:

[0027] S1. Establish an optimization model for the average value of the peak age of information for information communication between sensors and edge computing nodes;

[0028] S2. When it is detected that the sensor needs to offload services, the optimization model is solved by DQN to obtain a solution result;

[0029] S3. According to the number of sub-channels to be allocated to the sensor and the number of computing resources in the solution result, channels are configured for the sensor to perform service unloading.

[0030] From the above description, it can be seen that the beneficial effects of the present invention are: a DQN-based service unloading method of the present invention proposes an optimization problem of the minimum information age peak average value, and introduces a deep Q network (DQN) to solve the problem, which can effectively reduce the information age peak average value, thereby improving the service quality of the distribution network.

[0031] Furthermore, the DQN includes an environment, a state, an action space, and a reward function;

[0032] Wherein, the environment is defined as:

[0033]

[0034] Where K represents the number of sensors, U represents the number of continuously updated data packets, represents the total communication resources of the edge computing node, Represents the total computing resources of the edge computing node;

[0035] The state is defined as:

[0036] s t (d t ,c t ,p t ,h t );

[0037] Among them, d t is the packet size of the service to be offloaded, c t The amount of computation required to execute the offloaded service, p t Represents the signal transmission power of the sensor, h t represents the channel gain between the sensor and the edge computing node;

[0038] The action space is defined as:

[0039] A=(0 / 1) c ×(0 / 1) b ;

[0040] The number of subchannels assigned to the sensor is represented by a one-hot vector (0 / 1) c , the number of computing resources allocated to the sensor is represented by a one-hot vector (0 / 1) b , the action space is the Cartesian product of the two, and α(x)∈A is used to represent an action performed at time x in the action space, which determines the channel gain h between the sensor and the edge computing node. t ;

[0041] The reward function is defined as:

[0042]

[0043] Among them, r(x) is defined as the immediate reward obtained by executing action α(x) in state st(x), t k Represents the service demand s generated for sensor k k The time it takes to offload a service according to action α(x), represents the service demand s generated by sensor k k Time spent executing locally on the sensor.

[0044] From the above description, it can be seen that when using deep reinforcement learning to offload the edge computing service of the distribution network, the intelligent agent uses state, action and reward to interact with the environment to learn. The environment consists of sensors, the number of tasks, the computing resources and communication resources of the edge computing nodes. The state reflects the status of the sensor, and the information collection of all sensors in a certain period of time, the network status, and the resource status of the edge computing nodes are used as the state of the system. The action space is defined as the number of sub-channels and computing resources allocated to the sensor. Since maximizing the time saved by offloading to the edge computing node compared to local execution is essentially the same as minimizing the average peak AoI (i.e., the peak average of the information age), the above formula is used as the reward function.

[0045] Furthermore, the step S2 of solving the optimization model by DQN to obtain a solution specifically includes the following steps:

[0046] S21, initialize the experience pool size, and use random weights to initialize the prediction network parameters and target network parameters;

[0047] S22, obtaining the current state of the system and inputting DQN;

[0048] S23, estimating the instantaneous reward of each action in the action space for each data packet transmission of each sensor, and obtaining an approximate Q value of each action in the action space in the current state;

[0049] S24, input all actions in the current state and the corresponding approximate Q values ​​into the decision network, and the decision network selects the optimal action;

[0050] S25, outputting the optimal action;

[0051] The step S3 is specifically as follows:

[0052] According to the optimal action, a channel is configured for each sensor to perform service offloading.

[0053] From the above description, it can be seen that in this application, the sensor status update requirements, network transmission power, wireless channel gain, and remaining sub-channels and computing resources are combined into the system status s μ Input into the DQN network and calculate the state s μ The approximate Q values ​​of all actions are input into the action selection to determine the optimal action output.

[0054] Furthermore, the optimal action selected by the decision network in step S24 is specifically:

[0055] The decision network is based on a greedy algorithm and selects the action with the largest Q value or randomly selects an action as the optimal action output according to a preset probability ratio.

[0056] From the above description, we can see that in the decision-making process of the reinforcement learning algorithm, if the action with the largest Q value is selected every time in the state, it is easy to cause the unloading strategy to terminate at the local optimum. Therefore, in order to avoid the strategy converging to the local optimum, the greedy algorithm (ε-greedy) is used in the service unloading DQN algorithm to perform a certain proportion of random exploration.

[0057] Furthermore, the optimization model is specifically:

[0058] P1:

[0059]

[0060] And satisfy:

[0061]

[0062]

[0063]

[0064] in, represents the average value of the peak value of information age, K represents the number of sensors, U represents the number of continuously updated data packets, μ represents the μth data packet, μ∈U, Indicates that the data packet needs to be guaranteed within T I (μ-1,μ) represents the mutual arrival time of packets μ and μ-1, that is, the time interval between the two packets starting to be unloaded:

[0065]

[0066] T A (0) = 0;

[0067] T(μ) is the sojourn time, which indicates the time it takes for the data packet μ to be received from the beginning of unloading to the edge computing node:

[0068]

[0069] T A (μ) represents the arrival time of the μth data packet, T S (μ) represents the service time of the μth data packet, T D (μ) represents the departure time of the μth packet, T S (ν,μ) represents the cumulative service time from packet ν to μ:

[0070]

[0071] T I Similarly, (ν,μ) is expressed as the cumulative mutual arrival time of packets ν to μ, and Represents the processing time of the μth data packet at the edge computing node:

[0072]

[0073] β is the number of cycles used by the edge computing node to process a data packet, f is the calculation frequency of the edge computing node, and d is the size of the data packet:

[0074] d = log2 M / n;

[0075] M is the number of bits of information contained in the data packet, n means that the data packet is encoded as a codeword with n channels used, R k represents the transmission rate of sensor k:

[0076]

[0077]

[0078] γ(μ)=P(μ)|h(μ)| 2 / σ 2 ;

[0079] γ represents the signal-to-noise ratio, Q -1 (·) represents the inverse function of Q(x), ε k Represents the error probability. In the case of short packets, the error probability is generally 10 -6 ~10 -9 , σ 2 is the noise power, P(μ) represents the signal transmission power of the sensor when transmitting data packet μ, and h(μ) represents the channel gain between the sensor and the edge computing node when transmitting data packet μ.

[0080] From the above description, we can see that the optimization model of the average value of the information age peak is as shown above. Due to the constraints The non-convexity of , therefore the entire optimization problem is reduced to a non-convex optimization problem with continuous randomness.

[0081] Please refer to Figure 2 , a DQN-based service unloading terminal, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0082] S1. Establish an optimization model for the average value of the peak age of information for information communication between sensors and edge computing nodes;

[0083] S2. When it is detected that the sensor needs to offload services, the optimization model is solved by DQN to obtain a solution result;

[0084] S3. According to the number of sub-channels to be allocated to the sensor and the number of computing resources in the solution result, channels are configured for the sensor to perform service unloading.

[0085] From the above description, it can be seen that the beneficial effects of the present invention are: a DQN-based service unloading method of the present invention proposes an optimization problem of the minimum information age peak average value, and introduces a deep Q network (DQN) to solve the problem, which can effectively reduce the information age peak average value, thereby improving the service quality of the distribution network.

[0086] Furthermore, the DQN includes an environment, a state, an action space, and a reward function;

[0087] Wherein, the environment is defined as:

[0088]

[0089] Where K represents the number of sensors, U represents the number of continuously updated data packets, represents the total communication resources of the edge computing node, Represents the total computing resources of the edge computing node;

[0090] The state is defined as:

[0091] s t (d t ,c t ,p t ,h t );

[0092] Among them, d t is the packet size of the service to be offloaded, c t The amount of computation required to execute the offloaded service, p t Represents the signal transmission power of the sensor, h t represents the channel gain between the sensor and the edge computing node;

[0093] The action space is defined as:

[0094] A=(0 / 1) c ×(0 / 1) b ;

[0095] The number of subchannels assigned to the sensor is represented by a one-hot vector (0 / 1) c , the number of computing resources allocated to the sensor is represented by a one-hot vector (0 / 1) b, the action space is the Cartesian product of the two, and α(x)∈A is used to represent an action performed at time x in the action space, which determines the channel gain h between the sensor and the edge computing node. t ;

[0096] The reward function is defined as:

[0097]

[0098] Among them, r(x) is defined as the immediate reward obtained by executing action α(x) in state st(x), t k Represents the service demand s generated for sensor k k The time it takes to offload a service according to action α(x), represents the service demand s generated by sensor k k Time spent executing locally on the sensor.

[0099] From the above description, it can be seen that when using deep reinforcement learning to offload the edge computing service of the distribution network, the intelligent agent uses state, action and reward to interact with the environment to learn. The environment consists of sensors, the number of tasks, the computing resources and communication resources of the edge computing nodes. The state reflects the status of the sensor, and the information collection of all sensors in a certain period of time, the network status, and the resource status of the edge computing nodes are used as the state of the system. The action space is defined as the number of sub-channels and computing resources allocated to the sensor. Since maximizing the time saved by offloading to the edge computing node compared to local execution is essentially the same as minimizing the average peak AoI (i.e., the peak average of the information age), the above formula is used as the reward function.

[0100] Furthermore, the step S2 of solving the optimization model by DQN to obtain a solution specifically includes the following steps:

[0101] S21, initialize the experience pool size, and use random weights to initialize the prediction network parameters and target network parameters;

[0102] S22, obtaining the current state of the system and inputting DQN;

[0103] S23, estimating the instantaneous reward of each action in the action space for each data packet transmission of each sensor, and obtaining an approximate Q value of each action in the action space in the current state;

[0104] S24, input all actions in the current state and the corresponding approximate Q values ​​into the decision network, and the decision network selects the optimal action;

[0105] S25, outputting the optimal action;

[0106] The step S3 is specifically as follows:

[0107] According to the optimal action, a channel is configured for each sensor to perform service offloading.

[0108] From the above description, it can be seen that in this application, the sensor status update requirements, network transmission power, wireless channel gain, and remaining sub-channels and computing resources are combined into the system status s μ Input into the DQN network and calculate the state s μ The approximate Q values ​​of all actions are input into the action selection to determine the optimal action output.

[0109] Furthermore, the optimal action selected by the decision network in step S24 is specifically:

[0110] The decision network is based on a greedy algorithm and selects the action with the largest Q value or randomly selects an action as the optimal action output according to a preset probability ratio.

[0111] From the above description, we can see that in the decision-making process of the reinforcement learning algorithm, if the action with the largest Q value is selected every time in the state, it is easy to cause the unloading strategy to terminate at the local optimum. Therefore, in order to avoid the strategy converging to the local optimum, the greedy algorithm (ε-greedy) is used in the service unloading DQN algorithm to perform a certain proportion of random exploration.

[0112] Furthermore, the optimization model is specifically:

[0113] P1:

[0114]

[0115] And satisfy:

[0116]

[0117]

[0118]

[0119] in, represents the average value of the peak value of information age, K represents the number of sensors, U represents the number of continuously updated data packets, μ represents the μth data packet, μ∈U, Indicates that the data packet needs to be guaranteed within T I (μ-1,μ) represents the mutual arrival time of packets μ and μ-1, that is, the time interval between the two packets starting to be unloaded:

[0120]

[0121] T A (0) = 0;

[0122] T(μ) is the sojourn time, which indicates the time it takes for the data packet μ to be received from the beginning of unloading to the edge computing node:

[0123]

[0124] T A (μ) represents the arrival time of the μth data packet, T S (μ) represents the service time of the μth data packet, T D (μ) represents the departure time of the μth packet, T S (ν,μ) represents the cumulative service time from packet ν to μ:

[0125]

[0126] T I Similarly, (ν,μ) is expressed as the cumulative mutual arrival time of packets ν to μ, and Represents the processing time of the μth data packet at the edge computing node:

[0127]

[0128] β is the number of cycles used by the edge computing node to process a data packet, f is the calculation frequency of the edge computing node, and d is the size of the data packet:

[0129] d = log2 M / n;

[0130] M is the number of bits of information contained in the data packet, n means that the data packet is encoded as a codeword with n channels used, R k represents the transmission rate of sensor k:

[0131]

[0132]

[0133] γ(μ)=P(μ)|h(μ)| 2 / σ 2 ;

[0134] γ represents the signal-to-noise ratio, Q -1 (·) represents the inverse function of Q(x), ε k Represents the error probability. In the case of short packets, the error probability is generally 10 -6 ~10 -9 , σ 2is the noise power, P(μ) represents the signal transmission power of the sensor when transmitting data packet μ, and h(μ) represents the channel gain between the sensor and the edge computing node when transmitting data packet μ.

[0135] From the above description, we can see that the optimization model of the average value of the information age peak is as shown above. Due to the constraints The non-convexity of , therefore the entire optimization problem is reduced to a non-convex optimization problem with continuous randomness.

[0136] The DQN-based service unloading method and terminal of the present invention are suitable for scenarios in which communication resources in a random network need to be optimized in a distribution network equipment communication collection and computing network to improve the stability of the distribution network.

[0137] Please refer to Figure 1 as well as Figures 3 to 6 , Embodiment 1 of the present invention is:

[0138] A service unloading method based on DQN, comprising the steps of:

[0139] S1. Establish an optimization model for the average value of the peak age of information for information communication between sensors and edge computing nodes;

[0140] The optimization model is specifically:

[0141] P1:

[0142]

[0143] And satisfy:

[0144]

[0145]

[0146]

[0147] Among them, P AoI represents the average value of the peak value of information age, K represents the number of sensors, U represents the number of continuously updated data packets, μ represents the μth data packet, μ∈U, Indicates that the data packet needs to be guaranteed within T I (μ-1,μ) represents the mutual arrival time of packets μ and μ-1, that is, the time interval between the two packets starting to be unloaded:

[0148]

[0149] T A (0) = 0;

[0150] T(μ) is the sojourn time, which indicates the time it takes for the data packet μ to be received from the beginning of unloading to the edge computing node:

[0151]

[0152] T A (μ) represents the arrival time of the μth data packet, T S (μ) represents the service time of the μth data packet, T D (μ) represents the departure time of the μth packet, T S (ν,μ) represents the cumulative service time from packet ν to μ:

[0153]

[0154] T I Similarly, (ν,μ) is expressed as the cumulative mutual arrival time of packets ν to μ, and Represents the processing time of the μth data packet at the edge computing node:

[0155]

[0156] β is the number of cycles used by the edge computing node to process a data packet, f is the calculation frequency of the edge computing node, and d is the size of the data packet:

[0157] d = log2 M / n;

[0158] M is the number of bits of information contained in the data packet, n means that the data packet is encoded as a codeword with n channels used, R k represents the transmission rate of sensor k:

[0159]

[0160]

[0161] γ(μ)=P(μ)|h(μ)| 2 / σ 2 ;

[0162] γ represents the signal-to-noise ratio, Q -1 (·) represents the inverse function of Q(x), ε k Represents the error probability. In the case of short packets, the error probability is generally 10 -6 ~10 -9 , σ 2 is the noise power, P(μ) represents the signal transmission power of the sensor when transmitting data packet μ, and h(μ) represents the channel gain between the sensor and the edge computing node when transmitting data packet μ.

[0163] In this embodiment, under the condition of finite block length, consider that K sensors in multiple devices of a distribution network have U continuous status update data packets, and the sensors send their data packets to the edge server MEC (i.e., edge computing node). Assume that the status update data packet μ (μ∈U) contains M bits of information. By using finite block length coding (FBC), the status update packet can be encoded into a codeword with n channels. According to its finite block length transmission theory, the transmission rate for sensor k can be expressed as:

[0164]

[0165] The channel dispersion formula is defined as

[0166] V(γ)=1-(1+γ) -2 ;

[0167] γ represents the signal-to-noise ratio;

[0168]

[0169] Q -1 (·) represents its inverse function, ε k Represents the error probability. Under short packets, the block error probability is generally 10 -6 ~10 -9 , γ(μ) is the signal-to-noise ratio (SNR) of sending the μth state update packet:

[0170] γ(μ)=P(μ)|h(μ)| 2 / σ 2 ;

[0171] P(μ) and h(μ) are the transmission power and channel impulse response of the state update packet μ, σ 2 is the noise power. To ensure that the transmitted data packet μ is within the time period After the internal unloading is completed, the maximum channel capacity of the communication link should not be less than the actual size of the update information, that is, it should satisfy the constraint:

[0172]

[0173] Wherein, d=log2M / n is the size of the update data packet processed by the server.

[0174] Assume that the sensors in the distribution network upload the collected information to the MEC for calculation and generate readable status information. After successfully receiving the status update, the device sends an acknowledgment (ACK) packet to the source through an error-free and delay-free control channel and transmits the next data, such as Figure 3 shown.

[0175] When the MEC server receives the update information, if the current server still has tasks to process, it will put the new tasks into the buffer and wait; if the server is idle, it will process the tasks immediately. Therefore, the calculation time of the update information on the MEC server can be expressed as:

[0176]

[0177] Among them, β is the number of cycles used by the MEC server to process an update data packet, d is the size of the data packet, and f is the calculation frequency of the server.

[0178] When the MEC server completes processing the update data packet, it will send its calculation results to the result transmission channel queue. Considering that the return confirmation signal is small and the MEC has sufficient energy resources, the result transmission delay can be ignored, and then the waiting delay of the transmission queue can also be ignored.

[0179] Age of Information (AoI) is often used to characterize the freshness of information. AoI is defined as the time elapsed since the last received status update was delivered, and the mathematical expression is:

[0180] Δ μ (t) = tg μ (t);

[0181] Among them, t is the current time, g μ (t) is the start time of generating the state update packet μ. AoI can represent the generation time and delay of the state update, which can better reflect the timeliness of the state update message of the real-time application. When the receiving control end successfully receives a state update packet, the instantaneous age of such update data will decrease, otherwise it will increase linearly with time, such as Figure 4 As shown. A (μ), T S (μ) and T D (μ)(μ∈U,k∈K) represents the arrival time, service time, and departure time of the μth finite block length state update packet. Without loss of generality, let T A (0)=0.

[0182] Since the user end always retains the latest updated data packet, the only data packet that can affect the information age is the data packet that has been successfully offloaded. This paper only focuses on the data packet that has been successfully offloaded, assuming that the time when the sensor generates the state and the time when the state is transmitted to the user are negligible. For URLLC services, especially considering the impact of extreme events on the distribution network, the AoI peak directly affects the execution error rate of the distribution network control signal. Therefore, this paper uses the AoI peak as the performance metric of the system, denoted by PAoI ,like Figure 2 As shown in , the AoI increases linearly until the next update is delivered to the destination. The peak AoI is expressed as:

[0183] P AoI (μ) = T I (μ-1,μ)+T(μ);

[0184] In the formula It is defined as the mutual arrival time between the state update data packet μ and the state update data packet μ-1, that is, the time interval between the state update data packet μ and the state update data packet μ-1 starting to be unloaded.

[0185]

[0186] T(μ) is the sojourn time, i.e., the time from the start of unloading the state update data packet μ to the receiving control end, where:

[0187]

[0188] T S (ν,μ) is defined as the cumulative service time of the state update packet ν to μ. Therefore, the average peak AoI can be expressed as:

[0189]

[0190] Substituting the calculation formula of AoI peak into the above formula and combining it with ergodicity, the average peak age of the distribution network system can be obtained as follows:

[0191]

[0192] It can be seen that the average peak age of the system is the expected total time that each state update packet spends in the system, which includes the unloading interval time, service time and waiting time. In order to minimize the peak age of the system, the waiting time should be eliminated as much as possible and the service time should be shortened. Therefore, the system scheduling should meet the following constraints:

[0193]

[0194] This constraint indicates that when the state update data packet μ arrives at the MEC server, the MEC server should have completed the previous state update task. Since the channel can only transmit one set of state update data packets at a time, the waiting time for unloading queue is eliminated. Therefore, the unloading interval should also satisfy:

[0195]

[0196] In order to ensure that the data of the equipment sensors in the distribution network can be effectively transmitted to the MEC server and the status can be effectively updated, considering the actual needs of distribution network equipment monitoring, the unloading management is optimized, including the unloading interval T I And uninstall time To minimize the average peak age of the system, the optimization problem is defined as:

[0197] P1:

[0198]

[0199] And satisfy:

[0200]

[0201]

[0202]

[0203] Since the constraints in this optimization problem are The non-convexity of leads to a non-convex optimization problem, and the constraint In summary, this problem is a non-convex optimization problem with continuous randomness, which is difficult to solve using traditional methods. This paper then uses the deep reinforcement learning algorithm DQN to solve this problem.

[0204] S2. When it is detected that the sensor needs to offload services, the optimization model is solved by DQN to obtain a solution result;

[0205] The DQN includes an environment, a state, an action space, and a reward function;

[0206] Wherein, the environment is defined as:

[0207]

[0208] Where K represents the number of sensors, U represents the number of continuously updated data packets, represents the total communication resources of the edge computing node, Represents the total computing resources of the edge computing node;

[0209] The state is defined as:

[0210] s t (d t ,c t ,p t ,h t );

[0211] Among them, d tis the packet size of the service to be offloaded, c t The amount of computation required to execute the offloaded service, p t Represents the signal transmission power of the sensor, h t represents the channel gain between the sensor and the edge computing node;

[0212] The action space is defined as:

[0213] A=(0 / 1) c ×(0 / 1) b ;

[0214] The number of subchannels assigned to the sensor is represented by a one-hot vector (0 / 1) c , the number of computing resources allocated to the sensor is represented by a one-hot vector (0 / 1) b , the action space is the Cartesian product of the two, and α(x)∈A is used to represent an action performed at time x in the action space, which determines the channel gain h between the sensor and the edge computing node. t ;

[0215] The reward function is defined as:

[0216]

[0217] Among them, r(x) is defined as the immediate reward obtained by executing action α(x) in state st(x), t k Represents the service demand s generated for sensor k k The time it takes to offload a service according to action α(x), represents the service demand s generated by sensor k k Time spent executing locally on the sensor.

[0218] The step S2 of solving the optimization model by DQN to obtain the solution result specifically includes the following steps:

[0219] S21, initialize the experience pool size, and use random weights to initialize the prediction network parameters and target network parameters;

[0220] S22, obtaining the current state of the system and inputting DQN;

[0221] S23, estimating the instantaneous reward of each action in the action space for each data packet transmission of each sensor, and obtaining an approximate Q value of each action in the action space in the current state;

[0222] S24, input all actions in the current state and the corresponding approximate Q values ​​into the decision network, and the decision network selects the optimal action;

[0223] S25, outputting the optimal action;

[0224] The step S3 is specifically as follows:

[0225] According to the optimal action, a channel is configured for each sensor to perform service offloading.

[0226] The optimal action selected by the decision network in step S24 is specifically:

[0227] The decision network is based on a greedy algorithm and selects the action with the largest Q value or randomly selects an action as the optimal action output according to a preset probability ratio.

[0228] In this embodiment, the optimization model is configured in an edge computing node (referred to as a decision maker) that is only used for decision-making, and the decision maker knows the resource usage of each edge server. In this system, time is discretized into different time periods. At the beginning of each time period, the sensors in the distribution network will collect information within the corresponding range (such as voltage, current vector, etc.) and send it to the decision maker. After the decision maker makes an unloading decision, the sensor needs to be unloaded to the corresponding edge computing node for processing according to the decision result. At the same time, the decision maker will send the service unloading results to the central cloud for aggregation and optimization of the unloading parameters. The central cloud then returns the optimized parameters to the decision maker, thereby realizing continuous tuning of the service unloading decision.

[0229] In this embodiment, the optimization problem of the average value of the information age peak value is solved by DQN. First, the four key elements of DQN in the distribution network are given:

[0230] (1) Environment

[0231] When using deep reinforcement learning to offload edge computing services in distribution networks, the agent uses states, actions, and rewards to interact with the environment to learn. The environment consists of sensors, the number of tasks, and the computing and communication resources of the edge server. Therefore, the environment E is defined as:

[0232]

[0233] in, represents the total communication resources of the edge server, Represents the total computing resources of the edge server. There are K sensors in multiple devices in a distribution network with U continuous status update data packets.

[0234] (2) Status

[0235] When using deep reinforcement learning to offload edge computing services in the distribution network, the state is used to reflect the status of the sensor. The information collected by all sensors, the network status, and the resource status of the edge server in a certain period of time are taken as the system status, which is represented as s t (d t ,c t ,p t ,h t ). Where d t is the input data size of the service to be offloaded, c t The amount of computation required to perform the service, p t Represents the signal transmission power of the sensor, h t Represents the channel gain between the sensor and the edge computing node.

[0236] (3) Action

[0237] The action space is defined as the number of subchannels and computing resources allocated to the sensor, using one-hot vectors (0 / 1) c and (0 / 1) b The action space is the Cartesian product of the two, defined as:

[0238] A=(0 / 1) c ×(0 / 1) b ;

[0239] Where × is the Cartesian product. In this paper, α(x)∈A is used to represent an action performed at time x in the action space, which determines the channel gain between the sensor and the edge computing node.

[0240] (4) Rewards

[0241] The purpose of service offloading is to minimize the average peak age of the system. Define the immediate reward obtained by executing action α(x) when the system state is st(x) as r(x): Assuming that st(x) determines the service demand offloading of sensor k, then the service demand s generated by sensor k k Compared with offloading s k The time that can be saved when offloading to the sensor for local execution is:

[0242]

[0243] Since maximizing the time savings compared to local execution and minimizing the average peak AoI are essentially the same, the above formula is used as the reward function.

[0244] In this embodiment, the decision maker is the agent of the DQN algorithm, and the agent interacts with the environment through state, action, and reward. First, when the decision maker receives a task request from a distribution network sensor, it searches for the optimal action based on the current environmental state, and then returns the optimal action to the sensor. The action can indicate that the task is executed on the sensor, MEC, or cloud server. After the action is executed, the decision maker will receive the reward for the action. Finally, the environment enters the next state.

[0245] DQN uses a neural network to approximate the Q value under different state-action combinations, turning the Q table update problem into a function fitting problem. In this embodiment, the decision maker combines the sensor state update requirements, network transmission power, wireless channel gain, and remaining sub-channels and computing resources into the system state s μ Input into the DQN network and calculate the state s μ The approximate Q value of all actions under the state, and all Q values ​​and state s μ The input is used to determine the optimal action in the action selection, which is the decision maker's offloading decision.

[0246] In this embodiment, the pseudo code of the algorithm steps is as follows:

[0247]

[0248]

[0249] In the decision-making process of the reinforcement learning algorithm, if the action with the largest Q value is selected every time in the state, it is easy to cause the unloading strategy to terminate at the local optimum. Therefore, in order to avoid the strategy converging to the local optimum, the greedy algorithm (ε-greedy) is used in the service unloading DQN algorithm to perform a certain proportion of random exploration. The action selection in the service unloading DQN algorithm is defined as:

[0250]

[0251] ε is the probability of the greedy algorithm randomly selecting an action, which is a positive number less than 1. In this embodiment, the value is 0.1. That is, there is a 10% probability α μ is to randomly select an action in the action space, otherwise it is the action with the largest Q value.

[0252] S3. According to the number of sub-channels to be allocated to the sensor and the number of computing resources in the solution result, channels are configured for the sensor to perform service unloading.

[0253] In this embodiment, the sensor performs service unloading according to the optimal action. After the service unloading is completed, as shown in the above pseudo code, the agent observes the next state of the environment and the reward r of the action t , and the experience eμ =(s μ ,a μ ,r μ ,s μ+1 ) is stored in the experience pool D. Then the network parameters θ pre and θ tar , experience pool D, and current state s μ Input into the network update to implement DQN network update.

[0254] There are currently three methods for service unloading in distribution networks:

[0255] (1) LO (Locally Offloading): All information collected by distribution network sensors is configured and executed locally.

[0256] (2) CO (Cloud Offloading): When the sensor generates a service request, the edge server uploads all requests to the cloud for corresponding service offloading operations.

[0257] (3) RO (Edge Computing Devices Offloading): The edge server randomly assigns sub-channels and computing resources to sensors to perform corresponding service offloading operations.

[0258] When the computing power of the edge server is weak (the number of allocable computing resources is 5), the performance of different service offloading methods in different network environments is as follows: Figure 5 As shown. The experimental results show that the service offloading DQN algorithm performs better than other algorithms in service offloading under various network environments. When the number of subchannels in the system is 20, 30 or 40, the average peak AoI of the sensor of the service offloading DQN is 1.3, 1.19 and 1.015 respectively. Compared with the worst performing LO and CO algorithms, the performance of the service offloading DQN algorithm is improved by 54.6% to 87.3%. Compared with the best performing RO algorithm, the service offloading DQN also reduces the average peak AoI by 11.6% to 14.1%. Therefore, the experimental results prove that the service offloading DQN algorithm can effectively reduce the average peak AoI of the system when the computing power of the edge server is weak.

[0259] The computing power of the edge server is set to 10, the number of channels is set to 20, the number of distribution network sensors is set to between 2 and 16, and the service offloading DQN algorithm is adopted, while other parameters remain unchanged. Figure 5 As shown in Figure 2, considering three cases where the original data size is 30MB, 50MB, and 70MB respectively, the system average peak AoI increases with the increase in the number of sensors. Figure 6As shown in Figure 1, the average peak AoI of the systems of LO, RO, CO, and DQN also increases with the increase of raw data. Since the computing power of the edge server is limited, when the size of the raw data is greater than or equal to 70MB, the average peak AoI will be higher than that of the CO algorithm. The increase in the average peak AoI by DQN is not significant. The time overhead generated in data transmission has an impact on DQN.

[0260] In summary, the DQN algorithm proposed in this paper can significantly reduce the system average peak AoI in the distribution network edge computing environment, and has a performance improvement of 2.5% to 74.8% compared with the other four comparison algorithms. This proves that the DQN algorithm can reduce the peak information age in the distribution network edge environment and improve the service quality of the distribution network.

[0261] Please refer to Figure 2 , Embodiment 2 of the present invention is:

[0262] A DQN-based service unloading terminal 1 includes a processor 2, a memory 3, and a computer program stored in the memory 3 and executable on the processor 2. When the processor executes the computer program, the steps in the above embodiment 1 are implemented.

[0263] The present invention discloses a DQN-based service unloading method and terminal, the main principle of which is to express the optimization of the average value of the information age peak value as a non-convex optimization problem with continuous randomness, and solve it through the deep reinforcement learning algorithm DQN.

[0264] In summary, the present invention provides a DQN-based service offloading method and terminal, and proposes a DQN-based service offloading method to reduce the information age of the system. Considering the impact of extreme events, the peak information age is used as a performance indicator. In the end-edge-cloud collaborative service offloading architecture, the optimization problem of the minimum average peak AoI is proposed, and the deep Q-network (DQN) is introduced to solve the optimization problem. Finally, a comparative experiment was carried out with three other existing algorithms. The experiment shows that the DQN-based service offloading method can effectively reduce the average peak AoI, verifying the effectiveness of the method in this paper.

[0265] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent transformations made using the contents of the present invention's specification and drawings, or directly or indirectly applied in related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A service offloading method based on DQN, characterized in that: Includes steps: S1. Establish an optimization model for the average value of the peak age of information for information communication between sensors and edge computing nodes; S2. When it is detected that the sensor needs to offload services, the optimization model is solved by DQN to obtain a solution result; The DQN includes an environment, a state, an action space, and a reward function; Wherein, the environment is defined as: Where K represents the number of sensors, U represents the number of continuously updated data packets, represents the total communication resources of the edge computing node, Represents the total computing resources of the edge computing node; The state is defined as: s t (d t ,c t ,p t ,h t ); Among them, d t is the packet size of the service to be offloaded, c t The amount of computation required to execute the offloaded service, p t Represents the signal transmission power of the sensor, h t represents the channel gain between the sensor and the edge computing node; The action space is defined as: A＝(0 / 1) c ×(0 / 1) b ； The number of subchannels assigned to the sensor is represented by a one-hot vector (0 / 1) c , the number of computing resources allocated to the sensor is represented by a one-hot vector (0 / 1) b , the action space is the Cartesian product of the two, and α(x)∈A is used to represent an action performed at time x in the action space, which determines the channel gain h between the sensor and the edge computing node. t ; The reward function is defined as: Among them, r(x) is defined as the immediate reward obtained by executing action α(x) in state st(x), t k Represents the service demand s generated for sensor k k The time it takes to perform service offloading according to action α(x), represents the service demand s generated by sensor k k The time it takes to execute locally on the sensor; S3. According to the number of sub-channels to be allocated to the sensor in the solution result and the number of computing resources, channels are configured for the sensor to perform service unloading.

2. A DQN-based service offloading method according to claim 1, characterized in that: Solving the optimization model by DQN in step S2 to obtain a solution specifically includes the following steps: S21, initialize the experience pool size, and use random weights to initialize the prediction network parameters and target network parameters; S22, obtaining the current state of the system and inputting DQN; S23, estimating the instantaneous reward of each action in the action space for each data packet transmission of each sensor, and obtaining an approximate Q value of each action in the action space in the current state; S24, input all actions in the current state and the corresponding approximate Q values ​​into the decision network, and the decision network selects the optimal action; S25, outputting the optimal action; The step S3 is specifically as follows: According to the optimal action, a channel is configured for each sensor to perform service offloading.

3. A DQN-based service offloading method according to claim 2, characterized in that: The optimal action selected by the decision network in step S24 is specifically: The decision network is based on a greedy algorithm and selects the action with the largest Q value or randomly selects an action as the optimal action output according to a preset probability ratio.

4. A DQN-based service offloading method according to claim 1, characterized in that: The optimization model is specifically: And satisfy: in, represents the average value of the peak value of information age, K represents the number of sensors, U represents the number of continuously updated data packets, μ represents the μth data packet, μ∈U, Indicates that the data packet needs to be guaranteed within T I (μ-1,μ) represents the mutual arrival time of packets μ and μ-1, that is, the time interval between the two packets starting to be unloaded: T A (0)＝0； T(μ) is the sojourn time, which indicates the time it takes for the data packet μ to be received from the beginning of unloading to the edge computing node: T A (μ) represents the arrival time of the μth data packet, T S (μ) represents the service time of the μth data packet, T D (μ) represents the departure time of the μth packet, T S (ν,μ) represents the cumulative service time from packet ν to μ: T I Similarly, (ν,μ) is expressed as the cumulative mutual arrival time of packets ν to μ, and Represents the processing time of the μth data packet at the edge computing node: β is the number of cycles used by the edge computing node to process a data packet, f is the calculation frequency of the edge computing node, and d is the size of the data packet: d = log2M / n; M is the number of bits of information contained in the data packet, n means that the data packet is encoded as a codeword with n channels used, R k represents the transmission rate of sensor k: γ(μ)=P(μ)|h(μ)| 2 / s 2 ; γ represents the signal-to-noise ratio, Q -1 (·) represents the inverse function of Q(x), ε k Represents the error probability. In the case of short packets, the error probability is generally 10 -6 ~10 -9 , σ 2 is the noise power, P(μ) represents the signal transmission power of the sensor when transmitting data packet μ, and h(μ) represents the channel gain between the sensor and the edge computing node when transmitting data packet μ.

5. A DQN-based service offloading terminal, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the following steps are implemented: S1. Establish an optimization model for the average value of the peak age of information for information communication between sensors and edge computing nodes; S2. When it is detected that the sensor needs to offload services, the optimization model is solved by DQN to obtain a solution result; The DQN includes an environment, a state, an action space, and a reward function; Wherein, the environment is defined as: Where K represents the number of sensors, U represents the number of continuously updated data packets, represents the total communication resources of the edge computing node, Represents the total computing resources of the edge computing node; The state is defined as: s t (d t ,c t ,p t ,h t ); Among them, d t is the packet size of the service to be offloaded, c t The amount of computation required to execute the offloaded service, p t Represents the signal transmission power of the sensor, h t represents the channel gain between the sensor and the edge computing node; The action space is defined as: A＝(0 / 1) c ×(0 / 1) b ； The number of subchannels assigned to the sensor is represented by a one-hot vector (0 / 1) c , the number of computing resources allocated to the sensor is represented by a one-hot vector (0 / 1) b , the action space is the Cartesian product of the two, and α(x)∈A is used to represent an action performed at time x in the action space, which determines the channel gain h between the sensor and the edge computing node. t ; The reward function is defined as: Among them, r(x) is defined as the immediate reward obtained by executing action α(x) in state st(x), t k Represents the service demand s generated for sensor k k The time it takes to perform service offloading according to action α(x), represents the service demand s generated by sensor k k The time it takes to execute locally on the sensor; S3. According to the number of sub-channels to be allocated to the sensor in the solution result and the number of computing resources, channels are configured for the sensor to perform service unloading.

6. A DQN-based service offloading terminal according to claim 5, characterized in that: Solving the optimization model by DQN in step S2 to obtain a solution specifically includes the following steps: S21, initialize the experience pool size, and use random weights to initialize the prediction network parameters and target network parameters; S22, obtaining the current state of the system and inputting DQN; S23, estimating the instantaneous reward of each action in the action space for each data packet transmission of each sensor, and obtaining an approximate Q value of each action in the action space in the current state; S24, input all actions in the current state and the corresponding approximate Q values ​​into the decision network, and the decision network selects the optimal action; S25, outputting the optimal action; The step S3 is specifically as follows: According to the optimal action, a channel is configured for each sensor to perform service offloading.

7. A DQN-based service offloading terminal according to claim 6, characterized in that: The optimal action selected by the decision network in step S24 is specifically: The decision network is based on a greedy algorithm and selects the action with the largest Q value or randomly selects an action as the optimal action output according to a preset probability ratio.

8. A DQN-based service offloading terminal according to claim 5, characterized in that: The optimization model is specifically: And satisfy: in, represents the average value of the peak value of information age, K represents the number of sensors, U represents the number of continuously updated data packets, μ represents the μth data packet, μ∈U, Indicates that the data packet needs to be guaranteed within T I (μ-1,μ) represents the mutual arrival time of packets μ and μ-1, that is, the time interval between the two packets starting to be unloaded: T A (0)＝0； T(μ) is the sojourn time, which indicates the time it takes for the data packet μ to be received from the beginning of unloading to the edge computing node: T A (μ) represents the arrival time of the μth data packet, T S (μ) represents the service time of the μth data packet, T D (μ) represents the departure time of the μth packet, T S (ν,μ) represents the cumulative service time from packet ν to μ: T I Similarly, (ν,μ) is expressed as the cumulative mutual arrival time of packets ν to μ, and Represents the processing time of the μth data packet at the edge computing node: β is the number of cycles used by the edge computing node to process a data packet, f is the calculation frequency of the edge computing node, and d is the size of the data packet: d = log2M / n; M is the number of bits of information contained in the data packet, n means that the data packet is encoded as a codeword with n channels used, R k represents the transmission rate of sensor k: γ(μ)=P(μ)|h(μ)| 2 / s 2 ; γ represents the signal-to-noise ratio, Q -1 (·) represents the inverse function of Q(x), ε k Represents the error probability. In the case of short packets, the error probability is generally 10 -6 ~10 -9 , σ 2 is the noise power, P(μ) represents the signal transmission power of the sensor when transmitting data packet μ, and h(μ) represents the channel gain between the sensor and the edge computing node when transmitting data packet μ.

Images