Terminal task offloading method, device and equipment based on information freshness guarantee

Abstract

The embodiments of this specification relate to the field of smart substation technology, and in particular to a method, device and equipment for terminal task unloading based on information freshness guarantee. Based on reinforcement learning technology, the size of task data in the current time slot, the access service quality demand deficit, the terminal task unloading queuing delay deficit and the access service quality constraint are used as the state variables of the current time slot, and the state variables of the current time slot are input into the deep Q network for calculation to obtain the action variables of the current time slot. According to the action variables of the current time slot, the corresponding smart substation terminal is authorized to unload the task data generated in the current time slot, thereby minimizing the task age of all terminals under the long-term constraints of access service quality and queuing delay, so as to ensure the timeliness of task transmission and improve the freshness of task unloading information.

Description

Technical Field

[0001] The embodiments of this specification relate to the technical field of smart substations, and in particular to a method, device and equipment for terminal task offloading based on information freshness assurance. Background Art

[0002] The rapid construction of smart grids, the transformation of the global energy structure, and the development of renewable energy have provided a broad space for the development of smart substations. Smart substations will achieve efficient access to new energy, optimize scheduling and efficient management through the application of intelligent technologies and equipment, and promote the green transformation and sustainable development of the energy structure. Advanced information and communication technologies are key factors in improving the perception capability and precise control level of smart substations and realizing the automation and coordination of operation control. Wireless communication equipment in smart substations plays an important role in data transmission, remote monitoring, fault location and troubleshooting, and emergency communication. It has the characteristics of many nodes, wide coverage, and great differences in business communication requirements. The terminal has higher requirements for access service quality, communication delay, reliability, and speed. The existing information and communication technology cannot fully meet the economic and effectiveness requirements of diversified business transmission. Therefore, reliable, efficient, and flexible communication technology support is urgently needed. Information freshness is an important indicator for measuring the timeliness and effectiveness of terminal task offloading. However, due to the limitations of wireless resources and computing resources, the concurrent access of large-scale terminals in smart substations greatly increases the carrying pressure of the access network, resulting in network congestion, overload, and other problems. The effectiveness of task offloading is difficult to meet, which seriously threatens the safe and stable operation of the power grid.

[0003] How to improve the effectiveness of task offloading and the freshness of task offloading information is a technical problem that needs to be solved urgently. Summary of the invention

[0004] In order to solve the problem that the effectiveness of task offloading in the prior art is difficult to meet the requirements, the embodiments of this specification provide a terminal task offloading method, device and equipment based on information freshness guarantee, which is applied to the orderly access control and task offloading of smart substation terminals. The method consists of three steps: building a system model, long-term constraints, optimization goals, problem transformation and algorithm design. The method is based on reinforcement learning technology, dynamically optimizes the base station uplink transmission authorization strategy, and minimizes the task age of all terminals under the long-term constraints of access service quality and queuing delay to ensure the timeliness of task transmission and improve the freshness of task offloading information.

[0005] The specific technical solutions of the embodiments of this specification are as follows:

[0006] On the one hand, an embodiment of this specification provides a method for offloading terminal tasks based on information freshness guarantee, including:

[0007] Obtain the size of task data generated by each smart substation terminal in the current time slot;

[0008] For each smart substation terminal, the access service quality demand deficit of the current time slot is calculated according to the access service quality constraint set by the smart substation terminal, the access service quality demand deficit of the previous time slot and the authorization indicator variable of the previous time slot, wherein the access service quality constraint indicates the time slot ratio of the authorized task data unloading required by the smart substation terminal, and the authorization indicator variable indicates whether the task unloading is authorized;

[0009] For each smart substation terminal, the terminal task unloading queuing delay deficit of the current time slot is calculated according to the queuing delay constraint set by the smart substation terminal, the terminal task unloading queuing delay deficit of the previous time slot, the queue backlog in the local task cache area of ​​the previous time slot, and the task data size generated in the previous time slot. The queuing delay constraint represents the maximum number of consecutive time slots required by the smart substation terminal for unauthorized task data unloading;

[0010] The size of the task data of the current time slot of each smart substation terminal, the access service quality demand deficit, the terminal task unloading queuing delay deficit and the access service quality constraint are used as the state variables of the current time slot, and the state variables of the current time slot are input into the deep Q network for calculation to obtain the action variables of the current time slot, and the action variables include the authorization indicator variables of the current time slot of each smart substation terminal;

[0011] According to the action variable of the current time slot, the corresponding smart substation terminal is authorized to unload the task data generated in the current time slot.

[0012] Furthermore, the access service quality requirement deficit of each smart substation terminal in the initial time slot and the terminal task offloading queuing delay deficit are both predetermined values.

[0013] Furthermore, the formula for calculating the access service quality demand deficit of the current time slot according to the access service quality constraint set by the smart substation terminal, the access service quality demand deficit of the previous time slot, and the authorization indicator variable of the previous time slot is:

[0014] f(i,t+1)=max{f(i,t)-x(i,t)+κ(i),0};

[0015] Among them, f(i,t+1) represents the access service quality demand deficit of the i-th smart substation terminal in the t+1 time slot, f(i,t) represents the access service quality demand deficit of the i-th smart substation terminal in the t time slot, and x(i,t) represents the authorization indicator variable of the i-th smart substation terminal in the t time slot.

[0016] Furthermore, according to the queuing delay constraint set by the smart substation terminal, the terminal task unloading queuing delay deficit of the previous time slot, the queue backlog in the local task buffer area of ​​the previous time slot, and the task data size generated in the previous time slot, the formula for calculating the terminal task unloading queuing delay deficit of the current time slot is:

[0017]

[0018] Among them, h(i,t+1) represents the terminal task unloading queuing delay deficit of the i-th smart substation terminal in the t+1 time slot, h(i,t) represents the terminal task unloading queuing delay deficit of the i-th smart substation terminal in the t time slot, d(i) represents the queuing delay constraint set by the i-th smart substation terminal, and n(i,t) represents the size of the task data of the i-th smart substation terminal in the t time slot.

[0019] Furthermore, the state variable of the current time slot is input into the deep Q network for calculation, and the action variable of the current time slot is obtained, which further includes:

[0020] The state variables of the current time slot and the parameters of the deep Q network corresponding to the current time slot are calculated to obtain the action variables of the current time slot.

[0021] Furthermore, after obtaining the action variable of the current time slot, the method further includes:

[0022] For each smart substation terminal, the access service quality requirement deficit of the next time slot is calculated according to the service quality constraint of the smart substation terminal, the authorization indicator variable corresponding to the smart substation terminal in the action variable of the current time slot, and the access service quality requirement deficit of the smart substation terminal in the current time slot;

[0023] For each smart substation terminal, the terminal task unloading queuing delay deficit of the next time slot is calculated according to the queuing delay constraint set by the smart substation terminal, the terminal task unloading queuing delay deficit of the current time slot, and the size of the task data generated by the queue in the local task buffer area of ​​the current time slot;

[0024] Obtain the size of task data generated by each smart substation terminal in the next time slot;

[0025] The size of the task data of the next time slot of each smart substation terminal, the access service quality demand deficit, the terminal task unloading queuing delay deficit and the access service quality constraint are used as the state variables of the next time slot, and the loss value is calculated according to the state variables of the next time slot, the state variables of the current time slot and the action variables of the current time slot;

[0026] The parameters of the deep Q network corresponding to the current time slot are updated according to the loss value to obtain the parameters of the deep Q network for the next time slot.

[0027] Furthermore, the formula for calculating the loss value according to the state variable of the next time slot, the state variable of the current time slot, and the action variable of the current time slot is:

[0028]

[0029] in, represents the loss value, β represents the discount factor, Q(S(t+1), A(t), ω(t)) represents the Q value calculated according to the state variable S(t+1) of the t+1 time slot, the action space A(t) of the t time slot, and the parameter ω(t) of the deep Q network corresponding to the t time slot, Q(S(t), A(t), ω(t)) represents the Q value calculated according to the state variable S(t) of the t time slot, the action space A(t) of the t time slot, and the parameter ω(t) of the deep Q network corresponding to the t time slot;

[0030] represents the cost corresponding to the current time slot, and its formula is:

[0031]

[0032] Where I represents the total number of smart substation terminals, f(i,t) represents the access service quality demand deficit of the ith smart substation terminal in time slot t, x(i,t) represents the authorization indicator variable of the ith smart substation terminal in time slot t, κ(i) represents the access service quality constraint of the ith smart substation terminal, h(i,t) represents the terminal task unloading queuing delay deficit of the ith smart substation terminal in time slot t, d(i) represents the queuing delay constraint set by the ith smart substation terminal, q(i,t) represents the queue backlog in the local task cache area of ​​the ith smart substation terminal in time slot t, n(i,t) represents the size of the task data of the ith smart substation terminal in time slot t, V represents the non-negative weight parameter, ρ(i,t′,t) represents the age of the task generated by the ith smart substation terminal in time slot t′ (t′<t), C1 and C2 represent that at most M smart substation terminals can be scheduled simultaneously to unload task data, u i represents the ith smart substation terminal, U = {u1,u2,…,u i ,…,u I} is the smart substation terminal set, and T = {1,2,…,t,…,T} is the total time slot set.

[0033] Furthermore, the parameters of the deep Q network corresponding to the current time slot are updated according to the loss value, and the formula for obtaining the parameters of the deep Q network for the next time slot is:

[0034]

[0035] Where ω(t+1) represents the parameters of the deep Q network in the t+1 time slot, η is the learning rate of the deep Q network, ▽ ω(t) Represents the gradient.

[0036] On the other hand, the embodiment of this specification also provides a terminal task offloading device based on information freshness guarantee, the device comprising:

[0037] A task data size acquisition unit, used to acquire the size of task data generated by each smart substation terminal in the current time slot;

[0038] an access service quality demand deficit calculation unit, configured to calculate, for each smart substation terminal, an access service quality demand deficit of a current time slot according to an access service quality constraint set by the smart substation terminal, an access service quality demand deficit of a previous time slot, and an authorization indicator variable of a previous time slot, wherein the access service quality constraint indicates a time slot ratio of authorized task data unloading required by the smart substation terminal, and the authorization indicator variable indicates whether task unloading is authorized;

[0039] The terminal task unloading queuing delay deficit calculation unit is used to calculate the terminal task unloading queuing delay deficit of the current time slot for each smart substation terminal according to the queuing delay constraint set by the smart substation terminal, the terminal task unloading queuing delay deficit of the previous time slot, the queue backlog in the local task cache area of ​​the previous time slot, and the task data size generated in the previous time slot, wherein the queuing delay constraint represents the maximum number of consecutive time slots required by the smart substation terminal for unauthorized task data unloading;

[0040] An action variable calculation unit is used to take the size of the task data of the current time slot of each smart substation terminal, the access service quality requirement deficit, the terminal task unloading queuing delay deficit and the access service quality constraint as the state variable of the current time slot, input the state variable of the current time slot into the deep Q network for calculation, and obtain the action variable of the current time slot, wherein the action variable includes the authorization indicator variable of the current time slot of each smart substation terminal;

[0041] The task data unloading control unit is used to authorize the corresponding smart substation terminal to unload the task data generated in the current time slot according to the action variable of the current time slot.

[0042] On the other hand, an embodiment of the present specification further provides a computer device, including a memory, a processor, and a computer program stored in the memory, and the processor implements the above method when executing the computer program.

[0043] Using the embodiments of this specification, based on reinforcement learning technology, the size of the task data of the current time slot, the access service quality demand deficit, the terminal task unloading queuing delay deficit and the access service quality constraint are used as the state variables of the current time slot, and the state variables of the current time slot are input into the deep Q network for calculation to obtain the action variables of the current time slot. According to the action variables of the current time slot, the corresponding smart substation terminal is authorized to unload the task data generated in the current time slot, thereby minimizing the task age of all terminals under the long-term constraints of access service quality and queuing delay, so as to ensure the timeliness of task transmission and improve the freshness of task unloading information. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0045] Figure 1 It is a schematic diagram of a system for implementing a method for offloading terminal tasks based on information freshness guarantee in an embodiment of this specification;

[0046] Figure 2 The figure is a flow chart of a method for offloading terminal tasks based on information freshness guarantee according to an embodiment of the present specification;

[0047] Figure 3 It is a schematic diagram of the structure of a terminal task offloading device based on information freshness guarantee in an embodiment of this specification;

[0048] Figure 4 The figure is a schematic diagram of the structure of a computer device in an embodiment of the present specification.

[0049]

Description of the accompanying drawings

[0050] u1~u9, intelligent substation terminal;

[0051] 301. Task data size acquisition unit;

[0052] 302. Access service quality requirement deficit calculation unit;

[0053] 303, terminal task offloading queue delay deficit calculation unit;

[0054] 304, action variable calculation unit;

[0055] 305, task data unloading control unit;

[0056] 402. Computer equipment;

[0057] 404, processor;

[0058] 406. Memory;

[0059] 408, driving mechanism;

[0060] 410, input / output module;

[0061] 412. Input devices;

[0062] 414. Output device;

[0063] 416. Presentation equipment;

[0064] 418. Graphical user interface;

[0065] 420, network interface;

[0066] 422. Communication link;

[0067] 424. Communication bus. DETAILED DESCRIPTION

[0068] The following will be combined with the drawings in the embodiments of this specification to clearly and completely describe the technical solutions in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of them. Based on the embodiments in the embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the embodiments of this specification.

[0069] It should be noted that the terms "first", "second", etc. in the description and claims of the embodiments of this specification and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the embodiments of this specification described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, device, product or equipment that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0070] It should be noted that the acquisition, storage, use, and processing of data in the technical solutions of the embodiments of this specification comply with the relevant provisions of national laws and regulations.

[0071] It should be noted that in the embodiments of this specification, certain software, components, models and other existing solutions in the industry may be mentioned, and they should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of this application, but it does not mean that the applicant has or will necessarily use the solution.

[0072] like Figure 1 The figure shows a schematic diagram of the implementation system of a terminal task unloading method based on information freshness assurance in an embodiment of the present specification, including smart substation terminals u1~u9, base stations and edge servers, wherein the base stations provide wireless access services for the smart substation terminals, the edge servers and the base stations can be located at the same location, the edge servers are used to provide computing services, and the methods of the embodiments of the present specification can be applied on the edge servers.

[0073] It should be noted that Figure 1 The smart substation terminals u1 to u9 in the figure are only exemplary. In actual scenarios, multiple smart substation terminals may be included. For example, the smart substation terminal set is represented as U = {u1, u2, ..., u i ,…,u I}, where u i represents the i-th smart substation terminal, and I represents the total number of smart substation terminals.

[0074] The method of the embodiment of this specification adopts a time slot model, and the total time period is divided into T equal-length time slots. The length of each time slot can be set according to actual conditions, such as 1 hour, etc. This specification embodiment does not impose any restrictions, and the total time slot set is expressed as T = {1, 2, ..., t, ..., T}.

[0075] In addition, it should be noted that Figure 1 What is shown is only one application environment provided by the embodiment of this specification. In actual application, other application environments may also be included, such as application in other types of terminals, and this specification does not limit this.

[0076] In response to the problems existing in the prior art, an embodiment of this specification provides a terminal task unloading method based on information freshness guarantee, which is applied to the orderly access control and task unloading of smart substation terminals. The method consists of three steps: building a system model, long-term constraints, optimization goals, problem transformation and algorithm design. The method is based on reinforcement learning technology, dynamically optimizes the base station uplink transmission authorization strategy, and minimizes the task age of all terminals under the long-term constraints of access service quality and queuing delay to ensure the timeliness of task transmission and improve the freshness of task unloading information. Figure 2The figure shows a flow chart of the terminal task unloading method based on information freshness guarantee in the embodiment of this specification. This figure describes the process of controlling each smart substation to unload the generated task data. The sequence of steps listed in the embodiment is only one way of executing the sequence of many steps and does not represent the only execution sequence. When the system or device product is executed in practice, it can be executed in the order of the method shown in the embodiment or the accompanying drawings or in parallel.

[0077] Specific as Figure 2 As shown, the method may include:

[0078] Step 201: Obtain the size of task data generated by each smart substation terminal in the current time slot;

[0079] Step 202: for each smart substation terminal, the access service quality demand deficit of the current time slot is calculated according to the access service quality constraint set by the smart substation terminal, the access service quality demand deficit of the previous time slot, and the authorization indicator variable of the previous time slot, wherein the access service quality constraint indicates the time slot ratio of the authorized task data unloading required by the smart substation terminal, and the authorization indicator variable indicates whether the task unloading is authorized;

[0080] Step 203: for each smart substation terminal, the terminal task unloading queuing delay deficit of the current time slot is calculated according to the queuing delay constraint set by the smart substation terminal, the terminal task unloading queuing delay deficit of the previous time slot, the queue backlog in the local task buffer area of ​​the previous time slot, and the task data size generated in the previous time slot. The queuing delay constraint represents the maximum number of consecutive time slots required by the smart substation terminal for unauthorized task data unloading;

[0081] Step 204: The size of the task data of the current time slot of each smart substation terminal, the access service quality requirement deficit, the terminal task unloading queuing delay deficit and the access service quality constraint are used as the state variables of the current time slot, and the state variables of the current time slot are input into the deep Q network for calculation to obtain the action variables of the current time slot, and the action variables include the authorization indicator variables of the current time slot of each smart substation terminal;

[0082] Step 205: Authorize the corresponding smart substation terminal to unload the task data generated in the current time slot according to the action variable of the current time slot.

[0083] Using the embodiments of this specification, based on reinforcement learning technology, the size of the task data of the current time slot, the access service quality demand deficit, the terminal task unloading queuing delay deficit and the access service quality constraint are used as the state variables of the current time slot, and the state variables of the current time slot are input into the deep Q network for calculation to obtain the action variables of the current time slot. According to the action variables of the current time slot, the corresponding smart substation terminal is authorized to unload the task data generated in the current time slot, thereby minimizing the task age of all terminals under the long-term constraints of access service quality and queuing delay, so as to ensure the timeliness of task transmission and improve the freshness of task unloading information.

[0084] In the embodiment of this specification, a task division model is adopted, considering that in time slot t, terminal u i Generate task data of size n(i,t), and the generated tasks are first stored in u i In the local task cache area. i When establishing a connection with the base station, the task is offloaded to the edge server for calculation. Considering the timeliness of the task, ρ(i,t′,t) is defined as the time slot t when the terminal u i The age of the task generated in the t'th time slot (t'<t). If the task data is delayed and unloaded to the edge server, the age of the task data will increase. That is, in the t+1 time slot, the age of the task data is updated to:

[0085] ρ(i,t′,t+1)={ρ(i,t′,t)+1}·τ·[1-a(i,t',t)] (1)

[0086] Where a(i,t',t) represents the unloading indicator variable, and a(i,t',t) = 1 means that at time slot t, terminal u i The tasks generated in time slot t' are offloaded to the edge server. If not offloaded, a(i, t', t) = 0. τ represents the length of the time slot.

[0087] It should be noted that, in actual applications, because the processing services of each smart substation terminal are relatively fixed, the size of the task data generated in each time slot may be the same. In actual applications, the size of the task data of each time slot of each smart substation terminal can be a fixed value n(i). Therefore, through the method of the embodiment of this specification, the access service quality demand deficit of the initial time slot and the terminal task unloading queuing delay deficit can be used to directly predict the authorization indicator variable corresponding to each future time slot.

[0088] Information freshness is inversely proportional to task age. The older the terminal task is, the lower the information freshness is.

[0089] Define q(i,t) to represent the time slot t, terminal u iThe queue backlog in the local task buffer is updated as follows:

[0090] q(i,t+1)=max{q(i,t)+n(i,t)-z(i,t),0} (2)

[0091] Among them, q(i,t+1) means that in time slot t+1, terminal u i The queue backlog in the local task buffer, z(i,t) is the terminal u i The throughput at time slot t, considering the uplink task offloading, is defined as p(i,t) at time slot t+1, when terminal u i The transmission power of terminal u is g(i,t) at time slot t+1. i The channel gain of b(i,t) is the channel gain of terminal u at time slot t+1. i The transmission bandwidth, N0 is the bandwidth of the terminal u in time slot t+1. i The noise power spectral density is . Therefore, the signal-to-noise ratio of uplink data transmission is:

[0092]

[0093] The task offloading transfer rate can be expressed as:

[0094]

[0095] Where x(i,t) is defined as the terminal u of the i-th smart substation i The authorization indicator variable at time slot t, x(i,t) = 1 means that at time slot t, the smart substation terminal u i Obtain permission to allow task offloading, otherwise x(i,t)=0.

[0096] Considering that the task is successfully transmitted when it is completely transmitted to the edge server, the smart substation terminal u i The throughput at time slot t is z(i,t):

[0097]

[0098] Considering the first-in-first-out principle, the oldest task is transmitted first, so the formula of the unloading indicator variable is:

[0099]

[0100] in:

[0101]

[0102] For example, if l(i,t)=4, This means that when t' is between 4 and 7, a(i,t',t) = 1, that is, at terminal ui The data packets generated in the 4th, 5th, 6th, and 7th time slots are successfully unloaded in the tth time slot.

[0103] According to the importance of the business-generating tasks and the service quality requirements, smart substation terminals have different access priorities. Terminals with high access priorities need to complete more task offloading to ensure that important tasks are processed in a timely manner. Therefore, the long-term access priority constraint is defined as:

[0104]

[0105] Among them, κ(i)∈(0,1], κ(i) represents the access service quality constraint of the i-th smart substation terminal, and the access service quality constraint represents the time slot ratio of the authorized task data unloading required by the smart substation terminal. For example, there are 10 time slots in total, and the i-th smart substation terminal requires 5 time slots for authorized task data unloading, then κ(i)=0.5. The access service quality constraint of the embodiment of this specification is set by the smart substation terminal itself. Taking into account the differentiated access service quality constraints of the smart substation terminal, the differentiated unloading requirements of massive terminals can be met by reasonably setting the κ(i) parameter value.

[0106] Considering that the terminal queue cannot be infinitely backlogged, a queue delay constraint is defined to ensure the timeliness and effectiveness of task unloading. Based on Little's Law, the queue delay is proportional to the ratio of the average queue backlog to the average task arrival rate. Therefore, the queue delay of terminal task unloading can be constrained as:

[0107]

[0108] Among them, d(i) represents the queuing delay constraint set by the i-th smart substation terminal. The queuing delay constraint represents the maximum number of consecutive time slots required by the smart substation terminal for unauthorized task data unloading, that is, the queue backlog set by the smart substation terminal itself cannot exceed a certain number of time slots. For example, no more than 3 consecutive time slots are allowed and the smart substation terminal is still not authorized to unload task data.

[0109] Define the terminal u in time slot t i The information age θ(i,t) of the task queue in the local cache is:

[0110]

[0111] The optimization goal of the embodiment of this specification is to minimize the task age of all smart substation terminals under the long-term constraints of access service quality and queuing delay by optimizing the base station authorization strategy, so as to ensure the timeliness of task transmission and improve the freshness of task offloading information. The optimization goal is expressed as:

[0112]

[0113] Among them, x={x(i,t)|u i ∈U,t∈T} is the authorization indication vector. C1 and C2 indicate that the base station can simultaneously dispatch up to M smart substation terminals for task offloading. C3 and C4 are the long-term constraints of access service quality and the extension period constraints of smart substation terminal task offloading queue respectively.

[0114] Considering the coupling between the short-term authorization strategy of the base station and the long-term optimization objectives and constraints, the optimization objective P1 is difficult to solve directly. Therefore, the concept of virtual queue in Lyapunov optimization is applied to transform the access service quality requirements and the extension period constraints C3 and C4 of the intelligent substation terminal task unloading queue into queue stability constraints. The virtual queue f(i,t) of the access service quality requirement deficit is defined as follows:

[0115] f(i,t+1)=max{f(i,t)-x(i,t)+κ(i),0} (12)

[0116] Among them, f(i,t+1) represents the access service quality demand deficit of the i-th smart substation terminal in the t+1 time slot, f(i,t) represents the access service quality demand deficit of the i-th smart substation terminal in the t time slot, and x(i,t) represents the authorization indicator variable of the i-th smart substation terminal in the t time slot. If the access service quality demand deficit is equal to 0, it means that there is no deficit, that is, the constraint is met. If the access service quality demand deficit is greater than 0, it means that the constraint is not met.

[0117] Define the terminal task offloading queuing delay deficit virtual queue h(i,t), which is expressed as follows:

[0118]

[0119] Among them, h(i,t+1) represents the terminal task unloading queuing delay deficit of the i-th smart substation terminal in the t+1 time slot, h(i,t) represents the terminal task unloading queuing delay deficit of the i-th smart substation terminal in the t time slot, d(i) represents the queuing delay constraint set by the i-th smart substation terminal, and n(i,t) represents the size of the task data of the i-th smart substation terminal in the t time slot. If the terminal task unloading queuing delay deficit is equal to 0, it means that there is no deficit, that is, the constraint is met. If the terminal task unloading queuing delay deficit is greater than 0, it means that the constraint is not met.

[0120] f(i,t) and h(i,t) represent the terminal u in the tth time slot. i The deviation between the actual access service quality and queuing delay and the specified requirements.

[0121] When the virtual queues f(i, t) and h(i, t) are stable at the average rate, the long-term constraints C3 and C4 are automatically satisfied.

[0122] According to Lyapunov theory, the vector Θ(t) = [f(i, t), h(i, t)] is defined, so the Lyapunov function L(Θ(t)) can be expressed as:

[0123]

[0124] Lyapunov drift is defined as the expected value of the change of L(Θ(t)) in two consecutive time slots, expressed as:

[0125] ΔL(Θ(t))=E[L(Θ(t+1))-L(Θ(t))|Θ(t)] (15)

[0126] Under the long-term constraint of access service quality requirements, a weighted function drift plus penalty is defined to balance minimizing drift or minimizing penalty, that is, minimizing the access service quality requirement deficit or minimizing the mission age of all terminals, expressed as:

[0127] Δ V L(Θ(t))=ΔL(Θ(t))+V·E[Γ(t)|Θ(t)] (16)

[0128] Among them, ΔL(Θ(t)) represents the Lyapunov drift, E[Γ(t)|Θ(t)] represents the penalty, Δ V L(Θ(t)) is the weighted sum of drift and penalty, i.e. drift plus penalty, V is a non-negative weight parameter used to balance the performance of drift ΔL(Θ(t)) and penalty E[Γ(t)|Θ(t)]. The larger its value is, the more it tends to minimize the task age of all terminals, and vice versa, the more it tends to minimize the access service quality requirement deficit.

[0129] Substituting formula (14) and (15) into formula (16) and simplifying it, we can deduce that the upper bound of drift plus penalty is:

[0130]

[0131] Among them, C is a constant and does not affect the Lyapunov optimization. Therefore, the optimization problem P1 is transformed into minimizing the upper bound of drift plus penalty, expressed as:

[0132]

[0133] where x(t)={x(i,t)|u i ∈U}.

[0134] Based on this, the embodiment of this specification adopts reinforcement learning technology to predict the authorization indicator variable of each smart substation terminal in each time slot, that is, whether the smart substation terminal is authorized to perform task offloading in each time slot.

[0135] First, the base station authorization strategy optimization problem can be modeled as a Markov decision process (MDP) model, whose key elements include state space, action space and cost function.

[0136] The size of the task data of each smart substation terminal in time slot t, the access service quality demand deficit, the terminal task offloading queuing delay deficit and the access service quality constraint are used as the state variables of the current time slot:

[0137] S(t)={F(t),H(t),K,Ν(t)} (19)

[0138] Among them, F(t)={f(1,t),…,f(i,t),…,f(I,t)}, H(t)={h(1,t),…,h(i,t),…,h( I,t)}, K={κ(1),…,κ(i),…,κ(I)}, N(t)={n(1,t),…,n(i,t),…,n(I,t)}.

[0139] The action space is defined as the set of terminal authorization indicator variables x(i,t), namely:

[0140] A(t)={x(1,t),…,x(i,t),…,x(I,t)} (20)

[0141] Cost function. The cost function is defined as the optimization objective of P2, which is shown in formula (18)

[0142] Consider using reinforcement learning technology - deep Q network to solve the base station authorization strategy optimization problem. The deep Q network can use deep neural networks to extract features from historical data and learn the optimal strategy based on the estimated Q value Q(S(t), A(t), ω(t)), where ω(t) is the parameter of the deep Q network corresponding to the t time slot.

[0143] First, the access service quality demand deficit and the terminal task unloading queuing delay deficit of each smart substation terminal in the initial time slot are initialized to predetermined values, for example, they can be initialized to 0. At the beginning of each time slot, the edge server estimates the action Q value Q(S(t), A(t), ω(t)), calculates the state variables of the t time slot and the parameters of the deep Q network corresponding to the t time slot, and obtains the action variable of the t time slot. The action variable includes the authorization indicator variable of the current time slot of each smart substation terminal, thereby obtaining the authorization strategy π(t). The authorization strategy π(t) includes the smart substation terminal that is allowed to obtain permission in the t time slot and allow task unloading.

[0144] Secondly, the authorized smart substation terminal u i ∈π(t) establishes a wireless connection with the base station and performs task offloading. The throughput z(i,t) is calculated by formula (5), and the offloading indicator variable a(i,t',t) is updated by formula (6). On this basis, f(i,t+1) and h(i,t+1) of the next time slot (t+1) are calculated according to formulas (13) and (13), and the size of the task data of the next time slot is obtained, so that the size of the task data of the next time slot of each smart substation terminal, the access service quality demand deficit, the terminal task offloading queuing delay deficit and the access service quality constraint are used as the state variable S(t+1) of the next time slot.

[0145] The loss value is then calculated based on the state variable S(t+1) of the next time slot, the state variable S(t) of the current time slot (t), and the action space A(t) of the current time slot t; the action space refers to the set of terminal authorization indication variables x(i, t), including all possible actions.

[0146] According to the loss value, the parameter ω(t) of the deep Q network corresponding to the current time slot t is updated to obtain the parameter ω(t+1) of the deep Q network at the next time slot t+1.

[0147] Thus, the action variable of the next time slot t+1 is calculated according to the state variable S(t+1) of the next time slot t+1 and the parameter ω(t+1) of the deep Q network of the next time slot t+1, thereby obtaining the authorization strategy π(t+1) of the next time slot t+1.

[0148] By iterating and calculating according to the above steps, the authorization strategy for each time slot can be obtained, and the corresponding smart substation terminal can be authorized to unload the task data generated in the corresponding time slot according to the authorization strategy of each time slot.

[0149] The formula for calculating the loss value based on the state variables of the next time slot, the state variables of the current time slot, and the action space of the current time slot is:

[0150]

[0151] in, represents the loss value, β represents the discount factor, Q(S(t+1), A(t), ω(t)) represents the Q value of the action in the current action space estimated by the deep Q network based on the network parameter ω(t) in state S(t+1);

[0152] represents the cost corresponding to the current time slot, which is calculated using formula (18).

[0153] The parameters of the deep Q network corresponding to the current time slot are updated according to the loss value, and the formula for obtaining the parameters of the deep Q network for the next time slot is:

[0154]

[0155] Where ω(t+1) represents the parameters of the deep Q network in the t+1 time slot, η is the learning rate of the deep Q network, ▽ ω(t) Represents the gradient.

[0156] Based on the same inventive concept, the embodiment of this specification also provides a terminal task unloading device based on information freshness guarantee, such as Figure 3 As shown, the device comprises:

[0157] The task data size acquisition unit 301 is used to acquire the size of the task data generated by each smart substation terminal in the current time slot;

[0158] The access service quality requirement deficit calculation unit 302 is used to calculate the access service quality requirement deficit of the current time slot for each smart substation terminal according to the access service quality constraint set by the smart substation terminal, the access service quality requirement deficit of the previous time slot and the authorization indicator variable of the previous time slot, wherein the access service quality constraint indicates the time slot ratio of the authorized task data unloading required by the smart substation terminal, and the authorization indicator variable indicates whether the task unloading is authorized;

[0159] The terminal task unloading queuing delay deficit calculation unit 303 is used to calculate the terminal task unloading queuing delay deficit of the current time slot for each smart substation terminal according to the queuing delay constraint set by the smart substation terminal, the terminal task unloading queuing delay deficit of the previous time slot, the queue backlog in the local task cache area of ​​the previous time slot, and the task data size generated in the previous time slot. The queuing delay constraint represents the maximum number of consecutive time slots required by the smart substation terminal for unauthorized task data unloading;

[0160] The action variable calculation unit 304 is used to take the size of the task data of the current time slot of each smart substation terminal, the access service quality requirement deficit, the terminal task unloading queuing delay deficit and the access service quality constraint as the state variable of the current time slot, input the state variable of the current time slot into the deep Q network for calculation, and obtain the action variable of the current time slot, wherein the action variable includes the authorization indicator variable of the current time slot of each smart substation terminal;

[0161] The task data unloading control unit 305 is used to authorize the corresponding smart substation terminal to unload the task data generated in the current time slot according to the action variable of the current time slot.

[0162] Since the principle of solving the problem by the above device is similar to that of the above method, the implementation of the above system can refer to the implementation of the above method, and the repeated parts will not be repeated.

[0163] like Figure 4 The figure is a schematic diagram of the structure of a computer device according to an embodiment of the present specification. The computer device according to the embodiment of the present specification can run the method according to the embodiment of the present specification.

[0164] The computer device 402 may include one or more processors 404, such as one or more central processing units (CPUs), each of which may implement one or more hardware threads. The computer device 402 may also include any memory 406 for storing any type of information such as code, settings, data, etc. Without limitation, for example, the memory 406 may include any one or more combinations of the following: any type of RAM, any type of ROM, a flash memory device, a hard disk, an optical disk, etc. More generally, any memory may use any technology to store information.

[0165] Further, any memory may provide volatile or non-volatile retention of information.

[0166] Further, any memory may represent a fixed or removable component of the computer device 402. In one case, when the processor 404 executes the associated instructions stored in any memory or combination of memories, the computer device 402 can perform any operation of the associated instructions. The computer device 402 also includes one or more drive mechanisms 408 for interacting with any memory, such as a hard disk drive mechanism, an optical disk drive mechanism, etc.

[0167] The computer device 402 may also include an input / output module 410 (I / O) for receiving various inputs (via input devices 412) and for providing various outputs (via output devices 414). A specific output mechanism may include a presentation device 416 and an associated graphical user interface (GUI) 418. In other embodiments, the input / output module 410 (I / O), input device 412, and output device 414 may not be included, and the computer device 402 may be used as a computer device in a network. The computer device 402 may also include one or more network interfaces 420 for exchanging data with other devices via one or more communication links 422. One or more communication buses 424 couple the components described above together.

[0168] The communication link 422 may be implemented in any manner, for example, through a local area network, a wide area network (e.g., the Internet), a point-to-point connection, etc., or any combination thereof. The communication link 422 may include any combination of hardwired links, wireless links, routers, gateway functions, name servers, etc., governed by any protocol or combination of protocols.

[0169] The embodiments of the present specification also provide a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program implements the above method when executed by a processor.

[0170] The embodiments of the present specification also provide a computer-readable instruction, wherein when a processor executes the instruction, the program therein causes the processor to execute the above method.

[0171] It should be understood that in the various embodiments of the present specification, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present specification.

[0172] It should also be understood that in the embodiments of this specification, the term "and / or" is only a description of the association relationship of the associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the embodiments of this specification generally indicates that the associated objects before and after are in an "or" relationship.

[0173] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the embodiments of this specification can be implemented with electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the embodiments of this specification.

[0174] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0175] In the several embodiments provided in the embodiments of this specification, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, or it can be an electrical, mechanical or other form of connection.

[0176] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of this specification.

[0177] In addition, each functional unit in each embodiment of the present specification can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of software functional units.

[0178] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of this specification is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the embodiment of this specification. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0179] The embodiments of this specification use specific embodiments to illustrate the principles and implementation methods of the embodiments of this specification. The description of the above embodiments is only used to help understand the methods and core ideas of the embodiments of this specification. At the same time, for those skilled in the art, according to the ideas of the embodiments of this specification, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the embodiments of this specification.

Claims

1. A terminal task offloading method based on information freshness guarantee, characterized in that: The method comprises: Obtain the size of task data generated by each smart substation terminal in the current time slot; For each smart substation terminal, the access service quality demand deficit of the current time slot is calculated according to the access service quality constraint set by the smart substation terminal, the access service quality demand deficit of the previous time slot and the authorization indicator variable of the previous time slot, wherein the access service quality constraint indicates the time slot ratio of the authorized task data unloading required by the smart substation terminal, and the authorization indicator variable indicates whether the task unloading is authorized; For each smart substation terminal, the terminal task unloading queuing delay deficit of the current time slot is calculated according to the queuing delay constraint set by the smart substation terminal, the terminal task unloading queuing delay deficit of the previous time slot, the queue backlog in the local task cache area of ​​the previous time slot, and the task data size generated in the previous time slot. The queuing delay constraint represents the maximum number of consecutive time slots required by the smart substation terminal for unauthorized task data unloading; The size of the task data of the current time slot of each smart substation terminal, the access service quality demand deficit, the terminal task unloading queuing delay deficit and the access service quality constraint are used as the state variables of the current time slot, and the state variables of the current time slot are input into the deep Q network for calculation to obtain the action variables of the current time slot, and the action variables include the authorization indicator variables of the current time slot of each smart substation terminal; Authorize the corresponding smart substation terminal to unload the task data generated in the current time slot according to the action variable of the current time slot; The formula for calculating the access service quality demand deficit of the current time slot according to the access service quality constraint set by the smart substation terminal, the access service quality demand deficit of the previous time slot and the authorization indicator variable of the previous time slot is: f(i,t+1)=max{f(i,t)-x(i,t)+κ(i),0}; Wherein, f(i,t+1) represents the access service quality demand deficit of the i-th smart substation terminal in the t+1 time slot, f(i,t) represents the access service quality demand deficit of the i-th smart substation terminal in the t time slot, x(i,t) represents the authorization indicator variable of the i-th smart substation terminal in the t time slot, κ(i) represents the access service quality constraint of the i-th smart substation terminal; The formula for calculating the terminal task unloading queuing delay deficit of the current time slot is as follows based on the queuing delay constraint set by the smart substation terminal, the terminal task unloading queuing delay deficit of the previous time slot, the queue backlog in the local task buffer area of ​​the previous time slot, and the task data size generated in the previous time slot: Among them, h(i,t+1) represents the terminal task unloading queuing delay deficit of the i-th smart substation terminal in the t+1 time slot, h(i,t) represents the terminal task unloading queuing delay deficit of the i-th smart substation terminal in the t time slot, d(i) represents the queuing delay constraint set by the i-th smart substation terminal, n(i,t) represents the size of the task data of the i-th smart substation terminal in the t time slot, and q(i,t) represents the queue backlog in the local task cache area of ​​the i-th smart substation terminal in the t time slot.

2. The method according to claim 1, characterized in that The state variables of the current time slot are input into the deep Q network for calculation, and the action variables of the current time slot are further obtained including: The state variables of the current time slot and the parameters of the deep Q network corresponding to the current time slot are calculated to obtain the action variables of the current time slot.

3. The method according to claim 2, characterized in that After obtaining the action variable of the current time slot, the method further includes: For each smart substation terminal, the access service quality requirement deficit of the next time slot is calculated according to the service quality constraint of the smart substation terminal, the authorization indicator variable corresponding to the smart substation terminal in the action variable of the current time slot, and the access service quality requirement deficit of the smart substation terminal in the current time slot; For each smart substation terminal, the terminal task unloading queuing delay deficit of the next time slot is calculated according to the queuing delay constraint set by the smart substation terminal, the terminal task unloading queuing delay deficit of the current time slot, and the size of the task data generated by the queue in the local task buffer area of ​​the current time slot; Obtain the size of task data generated by each smart substation terminal in the next time slot; The size of the task data of the next time slot of each smart substation terminal, the access service quality demand deficit, the terminal task unloading queuing delay deficit and the access service quality constraint are used as the state variables of the next time slot, and the loss value is calculated according to the state variables of the next time slot, the state variables of the current time slot and the action space of the current time slot; The parameters of the deep Q network corresponding to the current time slot are updated according to the loss value to obtain the parameters of the deep Q network for the next time slot.

4. The method according to claim 3, characterized in that The formula for calculating the loss value based on the state variables of the next time slot, the state variables of the current time slot, and the action space of the current time slot is: in, represents the loss value, β represents the discount factor, Q(S(t+1), A(t), ω(t)) represents the Q value calculated according to the state variable S(t+1) of the t+1 time slot, the action space A(t) of the t time slot, and the parameter ω(t) of the deep Q network corresponding to the t time slot, Q(S(t), A(t), ω(t)) represents the Q value calculated according to the state variable S(t) of the t time slot, the action space A(t) of the t time slot, and the parameter ω(t) of the deep Q network corresponding to the t time slot; θ(t) represents the cost corresponding to the current time slot, and its formula is: Where I represents the total number of smart substation terminals, f(i,t) represents the access service quality demand deficit of the ith smart substation terminal in time slot t, x(i,t) represents the authorization indicator variable of the ith smart substation terminal in time slot t, κ(i) represents the access service quality constraint of the ith smart substation terminal, h(i,t) represents the terminal task unloading queuing delay deficit of the ith smart substation terminal in time slot t, d(i) represents the queuing delay constraint set by the ith smart substation terminal, q(i,t) represents the queue backlog in the local task cache area of ​​the ith smart substation terminal in time slot t, n(i,t) represents the size of the task data of the ith smart substation terminal in time slot t, V represents the non-negative weight parameter, ρ(i,t′,t) represents the age of the task generated by the ith smart substation terminal in time slot t′, where t′<t, C1 and C2 represent that at most M smart substation terminals can be scheduled simultaneously to unload task data, u i represents the ith smart substation terminal, U = {u1,u2,…,u i ,…,u I } is the terminal set of smart substation, and T is the total time slot set.

5. The method according to claim 4, characterized in that The parameters of the deep Q network corresponding to the current time slot are updated according to the loss value, and the formula for obtaining the parameters of the deep Q network for the next time slot is: Where ω(t+1) represents the parameters of the deep Q network in the t+1 time slot, η is the learning rate of the deep Q network, Represents the gradient.

6. A terminal task unloading device based on information freshness guarantee, characterized in that: The device comprises: A task data size acquisition unit, used to acquire the size of task data generated by each smart substation terminal in the current time slot; an access service quality demand deficit calculation unit, configured to calculate, for each smart substation terminal, an access service quality demand deficit of a current time slot according to an access service quality constraint set by the smart substation terminal, an access service quality demand deficit of a previous time slot, and an authorization indicator variable of a previous time slot, wherein the access service quality constraint indicates a time slot ratio of authorized task data unloading required by the smart substation terminal, and the authorization indicator variable indicates whether task unloading is authorized; The terminal task unloading queuing delay deficit calculation unit is used to calculate the terminal task unloading queuing delay deficit of the current time slot for each smart substation terminal according to the queuing delay constraint set by the smart substation terminal, the terminal task unloading queuing delay deficit of the previous time slot, the queue backlog in the local task cache area of ​​the previous time slot, and the task data size generated in the previous time slot, wherein the queuing delay constraint represents the maximum number of consecutive time slots required by the smart substation terminal for unauthorized task data unloading; An action variable calculation unit is used to take the size of the task data of the current time slot of each smart substation terminal, the access service quality requirement deficit, the terminal task unloading queuing delay deficit and the access service quality constraint as the state variable of the current time slot, input the state variable of the current time slot into the deep Q network for calculation, and obtain the action variable of the current time slot, wherein the action variable includes the authorization indicator variable of the current time slot of each smart substation terminal; A task data unloading control unit is used to authorize the corresponding smart substation terminal to unload the task data generated in the current time slot according to the action variable of the current time slot; The formula for calculating the access service quality demand deficit of the current time slot according to the access service quality constraint set by the smart substation terminal, the access service quality demand deficit of the previous time slot and the authorization indicator variable of the previous time slot is: f(i,t+1)=max{f(i,t)-x(i,t)+κ(i),0}; Wherein, f(i,t+1) represents the access service quality demand deficit of the i-th smart substation terminal in the t+1 time slot, f(i,t) represents the access service quality demand deficit of the i-th smart substation terminal in the t time slot, x(i,t) represents the authorization indicator variable of the i-th smart substation terminal in the t time slot, κ(i) represents the access service quality constraint of the i-th smart substation terminal; The formula for calculating the terminal task unloading queuing delay deficit of the current time slot is as follows based on the queuing delay constraint set by the smart substation terminal, the terminal task unloading queuing delay deficit of the previous time slot, the queue backlog in the local task buffer area of ​​the previous time slot, and the task data size generated in the previous time slot: Among them, h(i,t+1) represents the terminal task unloading queuing delay deficit of the i-th smart substation terminal in the t+1 time slot, h(i,t) represents the terminal task unloading queuing delay deficit of the i-th smart substation terminal in the t time slot, d(i) represents the queuing delay constraint set by the i-th smart substation terminal, n(i,t) represents the size of the task data of the i-th smart substation terminal in the t time slot, and q(i,t) represents the queue backlog in the local task cache area of ​​the i-th smart substation terminal in the t time slot.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 5 is implemented.

Images