A method and device for optimizing transmission delay of energy harvesting relay-assisted Internet of Things

Abstract

The present invention relates to the technical field of Internet of Things, and in particular to a transmission delay optimization method and device for energy harvesting relay-assisted Internet of Things. The method comprises: obtaining wireless channel parameters of a device and a relay node in each time slot and calculating the transmission rate of the device and the relay node; taking data balance and energy balance at the relay node as constraints, taking the transmission ratio of the device, the transmission power of the device and the relay node, and the sub-time slot fraction as optimization variables, and establishing an optimization problem P with the optimization goal of minimizing the transmission delay when each device transmits certain data; decomposing the optimization problem P into two sub-problems P‑1 and P‑2, and solving the two sub-problems using a neural network combined with a deep deterministic policy gradient algorithm and a binary search algorithm, thereby obtaining the optimal solution to the optimization problem P. It is achieved that the transmission delay when each device transmits data is minimized under the premise of ensuring data balance and energy balance at the relay node.

Description

Technical Field

[0001] The present invention relates to the technical field of Internet of Things, and in particular to a transmission delay optimization method for an energy harvesting relay-assisted Internet of Things. Background Art

[0002] Energy Harvesting Relay-Assisted Internet of Things (EH-RIoT) is an IoT communication method based on wireless communication and energy harvesting technology. In EH-RIoT, by using energy harvesting technology, terminal devices can obtain energy from the environment and convert the energy in the environment (such as solar energy, wind energy, mechanical energy, etc.) into electrical energy for powering terminal devices without using traditional battery power supply methods, achieving long-term wireless communication and extending the service life of the device. In EH-RIoT, the signals sent by the terminal device are forwarded to other nodes or the cloud by using relay nodes, thereby realizing data transmission and sharing. At the same time, EH-RIoT also uses relay nodes to expand the coverage of wireless networks and improve the reliability and capacity of the network. EH-RIoT has a wide range of applications, such as smart homes, smart cities, smart transportation and other fields. In these applications, EH-RIoT can help achieve long-term operation of devices and reliable transmission of data, thereby improving the efficiency and reliability of intelligent systems.

[0003] However, in the energy harvesting relay-assisted Internet of Things, due to the instability of energy supply and data collection, as well as interference between IoT devices, the energy harvesting relay-assisted Internet of Things has a long data transmission delay and cannot meet the technical problem of the minimum delay required for industrial IoT devices to upload certain data. Therefore, how to optimize the data transmission process and minimize the transmission delay while ensuring data balance and energy balance has become an important issue that needs to be solved in the energy harvesting relay-assisted Internet of Things technology. Summary of the invention

[0004] In response to the above technical problems, the present invention proposes a transmission delay optimization method and device for an energy harvesting relay-assisted Internet of Things, aiming to optimize the data transmission process of the energy harvesting relay-assisted Internet of Things.

[0005] In order to solve the above technical problems, the present invention adopts the following technical solution: a transmission delay optimization method for energy harvesting relay-assisted Internet of Things, comprising the following steps:

[0006] Step S1, placing a relay node in the base station, placing I IoT devices i within the communication range of the base station, represented by the set i = {1, 2, ..., I}, the relay node is used to receive data transmitted to it by the IoT device i, dividing the transmission delay K required for the device i to transmit data to the relay node into k time slots, and obtaining the wireless channel parameters of the device i and the relay node in each time slot;

[0007] Step S2, calculating the transmission rate of device i and the relay node according to the wireless channel parameters of device i and the relay node in each time slot;

[0008] Step S3, taking the data balance and energy balance at the relay node as constraints, taking the transmission ratio of device i, the transmission power of device i and the relay node, and the sub-slot fraction as optimization variables, and taking minimizing the transmission delay when each device i transmits a certain amount of data as the optimization goal to establish an optimization problem P;

[0009] Step S4, decompose the optimization problem P into two sub-problems P-1 and sub-problem P-2, use a neural network combined with a deep deterministic policy gradient algorithm to solve sub-problem P-1, and according to the solution result of sub-problem P-1, use a binary search algorithm to solve sub-problem P-2, so as to obtain the optimal solution of the optimization problem P.

[0010] Among them, delay refers to the time required for a signal to travel from the transmitter to the receiver, and time slot refers to the basic unit of communication in a communication system at fixed time intervals within a period of time.

[0011] The technical concept of the present invention is as follows: First, in the energy harvesting relay assisted Internet of Things, based on the consideration of the constraints of energy balance and data balance at the relay node, an optimization problem P of the transmission delay of the energy harvesting relay assisted Internet of Things is constructed. Subsequently, by analyzing the characteristics of the problem, the optimization problem P is divided into sub-problems P-1 and sub-problems P-2. Combined with the analysis of sub-problem P-1, a deep deterministic policy gradient algorithm for solving sub-problem P-1 and a binary search algorithm for solving sub-problem P-2 are proposed. Then, the solution to the original optimization problem P is obtained, thereby ensuring that the transmission delay is minimized when all the uploaded data is completed by the devices in the energy harvesting relay assisted Internet of Things.

[0012] By constructing the optimization problem P of the transmission delay of the energy harvesting relay-assisted Internet of Things, and decomposing the optimization problem P into sub-problems P-1 and sub-problems P-2, a deep deterministic policy gradient algorithm and a binary search algorithm are proposed to solve the sub-problems, thereby obtaining the optimal solution to the original optimization problem P. It achieves the premise of ensuring data balance and energy balance at the relay node, minimizing the transmission delay of each device when transmitting data, and improving the transmission rate of the energy harvesting relay-assisted Internet of Things, thereby saving network resources and increasing the system benefits of the entire network.

[0013] Preferably, in step S3, the expression of the optimization problem P is:

[0014] P:minK

[0015] The above formula is the objective function of the optimization problem P, which represents minimizing the transmission delay when each device i transmits a certain amount of data, where K represents the transmission delay required for device i to transmit data to the relay node;

[0016] The optimization variables include: i,k}, {p i,k}、{τ k}, K,

[0017] Among them, α i,k represents the transmission ratio of device i in the kth time slot, r represents the relay node, and p rk represents the transmission power of relay node r in the kth time slot, p i,k represents the transmission power of device i in the kth time slot, τ k represents the subslot fraction within the kth time slot;

[0018]

[0019] The above formula is the constraint condition 1 of the optimization problem P, which represents the data balance at the relay node, where T represents the length of a single time slot, represents the transmission rate of relay node r, Q i,k represents the amount of data transmitted by device i in the kth time slot, R i,k Indicates the transmission rate of device i;

[0020]

[0021]

[0022] The above formula is the constraint 2 and constraint 3 of the optimization problem P, which represents the energy balance at the relay node, where E max represents the maximum energy consumption at the relay node, E k represents the energy of the relay node r itself in the kth time slot, E Ak represents the energy obtained by the relay node r from the outside world in the kth time slot;

[0023] D i (k)≥L, i∈I,

[0024] The above formula is the constraint condition 4 of the optimization problem P, where D i(k) represents the amount of data transmitted by device i in the first k time slots, L represents the minimum amount of data transmission required by the Internet of Things, and the above formula indicates that the amount of data transmitted by device i in k time slots must exceed the minimum amount of data transmission required by the Internet of Things;

[0025] 0≤τ k ≤1, k∈K,

[0026]

[0027]

[0028]

[0029]

[0030] The above formula is the constraint condition 5-9 of the optimization problem P, which represents the value range of the optimization variable, where: represents the maximum transmission power of device i, represents the maximum transmission power of relay node r, represents the minimum transmission delay required for device i to transmit data to the relay node, K Indicates the maximum transmission delay required for device i to transmit data to the relay node.

[0031] Preferably, in step S4, the expression of sub-problem P-1 is:

[0032]

[0033] The optimization variables include: i,k}, {p i,k}、{τ k},

[0034] The above formula is the objective function of subproblem P-1, which means that by jointly optimizing {α i,k}, {p i,k} and {τ k}These four optimization variables maximize the amount of data transmitted by device i within a fixed time slot;

[0035]

[0036] The above formula is constraint 1 of subproblem P-1, which indicates the data balance at the relay node;

[0037]

[0038]

[0039] The above formula is the constraint 2 and constraint 3 of subproblem P-1, which represents the energy balance at the relay node;

[0040] 0≤τ k ≤1, k∈K,

[0041]

[0042]

[0043]

[0044]

[0045] The above formula is the constraint conditions 4-8 of subproblem P-1, which represents the value range of the optimization variable.

[0046] Preferably, in step S4, the method of using a neural network combined with a deep deterministic policy gradient algorithm to solve the subproblem P-1 is:

[0047] Step S401, determine the action space information and state space information, the action space information includes the transmission ratio α of device i i,k , the transmission power of the relay node p rk , the transmission power p of device i i,k and the subslot fraction τ k The state space information includes the amount of data Q transmitted by device i in the current time slot i,k And the energy E of the relay node itself in the kth time slot k ;

[0048] Step S402, using the actor-critic algorithm framework to construct a neural network model, the neural network model includes an actor network, a critic network and a reward function, the input of the actor network is state space information, and the output is an action space value, the input of the critic network is state space information and action space value, and the output is a state value function, and the reward function is used to describe the reward fed back from the environment after the neural network model takes each action in each state;

[0049] Step S403, initializing the parameters of the neural network model and defining the loss functions of the actor network and the critic network;

[0050] Step S404, using an optimizer to optimize the loss functions of the actor network and the critic network;

[0051] Step S405, randomly extracting data from the experience pool using the experience replay method to train the neural network model, repeatedly updating the actor network and the critic network until the optimal neural network model is obtained;

[0052] Step S406, input the current state space information into the trained actor network, output the optimal action space value and the maximum value of the transmission data volume of device i in each time slot, and use the optimal action space value as the optimal solution to subproblem P-1.

[0053] Preferably, after "step S404, using an optimizer to optimize the loss functions of the actor network and the critic network", the following steps are also included:

[0054] Input state space information s k To the actor network, output action space value a k ;

[0055] Set the action space value a k Applied to the environment, the environment returns the state space information s for the next time slot k+1 and reward r k ;

[0056] Will s k 、a k 、r k and k+1 It is recorded as a four-tuple data and put into the experience pool.

[0057] Preferably, the actor network includes an actor online network and an actor target network, and the critic network includes a critic online network and a critic target network. Then the sub-steps of step S405 include:

[0058] Extract quadruple data from the experience pool (s k ,a k ,r k ,s k+1 ) as a sample, and take s in the sample k and a k Input into the critic online network, and get the transmission data volume Q of the current time slot = (s k ,a k );

[0059] The s in the sample k+1 Input into the actor-target network to obtain the target action space value a′ k+1 , and s k+1 and a′ k+1 Input them into the critic target network together, and get the target transmission data volume Q′=(s k+1 ,a′ k+1 );

[0060] Update the actor online network, and update the critic online network according to the action space value output by the actor online network, so that the Q value output by the critic online network increases;

[0061] Repeat the above steps until the Q value output by the critic online network is equal to the target transmission data volume Q′ or the Q value is within the preset error range of the target transmission data volume Q′, and the neural network model at this time is taken as the optimal neural network model.

[0062] Preferably, in step S4, the expression of sub-problem P-2 is:

[0063] P-2: mink, k∈K

[0064] The optimization variables are: k,

[0065] The above formula is the objective function of subproblem P-2, which means minimizing the number of time slots under the amount of transmission data solved in subproblem P-1, thereby obtaining the minimized transmission delay;

[0066] D i (k)≥L, i∈I,

[0067] The above formula is constraint 1 of subproblem P-2, which means that the amount of data transmitted by device i in k time slots must exceed the minimum required amount of data transmission in the Internet of Things;

[0068]

[0069] The above formula is constraint 2 of subproblem P-2, which represents the value range of the transmission delay K.

[0070] Preferably, in step S4, the method of using the binary search algorithm to solve the sub-problem P-2 is:

[0071] Step S411, determine the maximum number of time slots k upp And the minimum number of time slots k low , where k upp >k low , and obtain the minimum transmission data volume L required by the Internet of Things;

[0072] Step S412, calculate the current time slot number k mid =(k low +k upp ) / 2;

[0073] Step S413, if D i (k mid )≥L, then execute step S414, otherwise execute step S415;

[0074] Step S414, let k upp =k mid ;

[0075] Step S415, let k low =kmid ;

[0076] Step S416, repeat steps S412-S415 until k upp ≤k low , output the current time slot number k mid As the optimal solution to subproblem P-2, the optimal solution to the optimization problem P is obtained.

[0077] Preferably, the wireless channel parameters of the device i and the relay node in each time slot include the channel bandwidth, channel gain and background noise power from the device to the relay node r, the transmission power of the device, the channel bandwidth, channel gain and background noise power from the relay node to the base station and the transmission power of the relay node. Then in step S2, according to the wireless channel parameters of the device i and the relay node in each time slot, the method for calculating the transmission rate of the device i and the relay node includes:

[0078]

[0079]

[0080] Where i and j both represent devices, r represents a relay node, k represents the number of time slots, and R i,k represents the transmission rate of device i in the kth time slot, B 1 represents the channel bandwidth from the device to the relay node r, p i,k represents the transmission power of device i in the kth time slot, g i,k represents the channel gain from device i to the relay node in the kth time slot, p j,k represents the transmission power of device j in the kth time slot, g j,k represents the channel gain from device j to the relay node in the kth time slot, N 1 Represents the background noise power from the device to the relay node, R rk represents the transmission rate of relay node r in the kth time slot, B 2 represents the channel bandwidth from the relay node to the base station, p rk represents the transmission power of relay node r in the kth time slot, g rk represents the channel gain from relay node r to the base station in the kth time slot, N 2 represents the background noise power from relay node r to the base station.

[0081] A transmission delay optimization device for an energy harvesting relay-assisted Internet of Things, comprising:

[0082] A parameter acquisition module, used to divide the transmission delay K required for device i to transmit data to the relay node into k time slots, and obtain the wireless channel parameters of device i and the relay node in each time slot;

[0083] A transmission rate calculation module, used to calculate the transmission rate of device i and the relay node according to the wireless channel parameters of device i and the relay node in each time slot;

[0084] An optimization problem establishment module is used to establish an optimization problem P with data balance and energy balance at the relay node as constraints, the transmission ratio of device i, the transmission power of device i and the relay node, and the sub-time slot fraction as optimization variables, and minimizing the transmission delay when each device i transmits a certain amount of data as the optimization goal;

[0085] The optimization problem solving module is used to decompose the optimization problem P into two sub-problems P-1 and P-2, and use a neural network combined with a deep deterministic policy gradient algorithm to solve sub-problem P-1. Based on the solution result of sub-problem P-1, a binary search algorithm is used to solve sub-problem P-2, thereby obtaining the optimal solution to the optimization problem P.

[0086] The beneficial technical effects of the present invention include: adopting a transmission delay optimization method and device for energy harvesting relay-assisted Internet of Things, constructing an optimization problem P for the transmission delay of energy harvesting relay-assisted Internet of Things, and decomposing the optimization problem P into sub-problems P-1 and sub-problems P-2, and proposing a deep deterministic policy gradient algorithm to obtain the transmission ratio α of the optimal device i of sub-problem P-1 i,k , the transmission power of the relay node p rk , the transmission power p of device i i,k , sub-slot fraction τ k And the maximum value of the amount of data transmitted by device i in each time slot, and then use the binary search algorithm to solve subproblem P-2 to get the optimal transmission time slot of the optimization problem, so as to find the optimal solution to the original optimization problem P, and achieve the premise of ensuring data balance and energy balance at the relay node. Minimize the transmission delay when each device transmits data, thereby improving the transmission rate of the energy harvesting relay-assisted Internet of Things, saving network resources, and increasing the system benefits of the entire network.

[0087] Other features and advantages of the present invention will be disclosed in detail in the following specific embodiments and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0088] The present invention will be further described below in conjunction with the accompanying drawings:

[0089] Figure 1 This is a flow chart of a method for optimizing transmission delay of an energy harvesting relay-assisted Internet of Things according to an embodiment of the present invention.

[0090] Figure 2 This is a schematic diagram of simulation experimental results of the relationship between the amount of data transmitted and the delay when the NOMA technology and the FDMA technology are used in an embodiment of the present invention when the number of devices I is 3, 5, and 7 respectively.

[0091] Figure 3 The following is a flow chart of a method for solving sub-problem P-1 according to an embodiment of the present invention.

[0092] Figure 4 This is a flowchart of the sub-steps of step S405 of an embodiment of the present invention.

[0093] Figure 5 Schematic diagram of the principle of the deep deterministic policy gradient algorithm of an embodiment of the present invention.

[0094] Figure 6 This is a flow chart of a method for solving subproblem P-2 using a binary search algorithm in an embodiment of the present invention.

[0095] Figure 7 This is a schematic diagram of simulation experiment results of the relationship between the amount of data transmitted and the delay under fixed transmission power and optimal power when the number of devices I is 3, 5, and 7, respectively, in an embodiment of the present invention.

[0096] Figure 8 This is a structural schematic diagram of a transmission delay optimization device for an energy harvesting relay-assisted Internet of Things according to an embodiment of the present invention.

[0097] Among them: 1. Parameter acquisition module, 2. Transmission rate calculation module, 3. Optimization problem establishment module, 4. Optimization problem solving module. DETAILED DESCRIPTION

[0098] The technical solutions of the embodiments of the present invention are explained and described below in conjunction with the drawings of the embodiments of the present invention, but the following embodiments are only preferred embodiments of the present invention, not all. Based on the embodiments in the implementation mode, other embodiments obtained by those skilled in the art without creative work are all within the protection scope of the present invention.

[0099] In the following description, terms such as "inside", "outside", "up", "down", "left", "right", etc. that indicate directions or positional relationships are only used to facilitate the description of the embodiments and simplify the description, and do not indicate or imply that the referred device or element must have a specific direction, be constructed and operate in a specific direction. Therefore, it should not be understood as a limitation of the present invention.

[0100] Embodiment 1:

[0101] The present application embodiment provides a method for optimizing transmission delay of an energy harvesting relay-assisted Internet of Things. Figure 1 , including the following steps:

[0102] Step S1, place a relay node in the base station, place I IoT devices i within the communication range of the base station, represented by the set i = {1, 2, ..., I}, the relay node is used to receive data transmitted to it by the IoT device i, divide the transmission delay K required for the device i to transmit data to the relay node into k time slots, and obtain the wireless channel parameters of the device i and the relay node in each time slot.

[0103] Among them, delay refers to the time required for a signal to travel from the transmitter to the receiver, and time slot refers to the basic unit of communication in a communication system at fixed time intervals within a period of time.

[0104] Since the energy harvesting relay-assisted Internet of Things uses relay nodes to expand network coverage, the data transmission will increase the transmission delay to a certain extent when it is forwarded through multiple relay nodes, that is, the number and deployment location of the relay nodes will affect the transmission delay, which requires reasonable planning and design. For this reason, the embodiment of the present application preferably uses one relay node.

[0105] Step S2, calculating the transmission rates of device i and the relay node according to the wireless channel parameters of device i and the relay node in each time slot.

[0106] Step S3, taking the data balance and energy balance at the relay node as constraints, the transmission ratio of device i, the transmission power of device i and the relay node, and the sub-slot fraction as optimization variables, and minimizing the transmission delay when each device i transmits a certain amount of data as the optimization goal to establish an optimization problem P.

[0107] The transmission ratio of device i refers to the ratio of the number of sub-time slots occupied by device i in the kth time slot to the total number of sub-time slots in the kth time slot. Usually, the transmission ratio is a decimal or percentage value, which is used to describe the transmission time ratio of device i in the kth time slot, that is, the transmission time of device i in the kth time slot occupies what proportion of the total time.

[0108] For example, if there are 100 sub-time slots in the kth time slot, and device i occupies 20 sub-time slots for data transmission, the transmission ratio of device i is 20 / 100=0.2 or 20%. This means that in the kth time slot, the data transmission of device i occupies 20% of the total time.

[0109] Among them, a sub-time slot is a component of a time slot. A time slot can be divided into multiple sub-time slots. The sub-time slot fraction is used to divide the time period within the time slot and can be set according to specific communication requirements.

[0110] Step S4, decompose the optimization problem P into two sub-problems P-1 and sub-problem P-2, use a neural network combined with a deep deterministic policy gradient algorithm to solve sub-problem P-1, and according to the solution result of sub-problem P-1, use a binary search algorithm to solve sub-problem P-2, so as to obtain the optimal solution of the optimization problem P.

[0111] Most existing technologies use frequency division multiple access (FDMA) technology to send data to relay nodes. Although FDMA technology can simply divide the frequency band into multiple sub-channels and has good interference suppression capability and stability, it has low frequency band utilization and is not flexible enough. For this reason, the embodiment of the present application preferably sends data to the relay node through non-orthogonal multiple access (NOMA) technology. NOMA technology can support more user access under the same bandwidth, frequency and power conditions, thereby improving spectrum utilization and the capacity and efficiency of the communication system and reducing transmission delay. When other parameters are the same, the number I of IoT devices is 3, 5, and 7 respectively. The simulation experimental results of the relationship between the amount of data transmitted and the delay when using NOMA technology and FDMA technology are shown in the attached figure. Figure 2 As shown, the simulation results verify the beneficial technical effect of this embodiment in reducing transmission delay.

[0112] The technical concept of the present invention is as follows: First, in the energy harvesting relay assisted Internet of Things, based on the consideration of the constraints of energy balance and data balance at the relay node, an optimization problem P of the transmission delay of the energy harvesting relay assisted Internet of Things is constructed. Subsequently, by analyzing the characteristics of the problem, the optimization problem P is divided into sub-problems P-1 and sub-problems P-2. Combined with the analysis of sub-problem P-1, a deep deterministic policy gradient algorithm for solving sub-problem P-1 and a binary search algorithm for solving sub-problem P-2 are proposed. Then, the solution to the original optimization problem P is obtained, thereby ensuring that the transmission delay is minimized when all the uploaded data is completed by the devices in the energy harvesting relay assisted Internet of Things.

[0113] By constructing the optimization problem P of the transmission delay of the energy harvesting relay-assisted Internet of Things, and decomposing the optimization problem P into sub-problems P-1 and sub-problems P-2, a deep deterministic policy gradient algorithm and a binary search algorithm are proposed to solve the sub-problems, thereby obtaining the optimal solution to the original optimization problem P. It achieves the premise of ensuring data balance and energy balance at the relay node, minimizing the transmission delay of each device when transmitting data, and improving the transmission rate of the energy harvesting relay-assisted Internet of Things, thereby saving network resources and increasing the system benefits of the entire network.

[0114] On the other hand, in this embodiment, in step S3, the optimization problem P is a non-convex optimization problem, and the expression of the optimization problem P is:

[0115] P: min K

[0116] The above formula is the objective function of the optimization problem P, which represents minimizing the transmission delay when each device i transmits a certain amount of data, where K represents the transmission delay required for device i to transmit data to the relay node;

[0117] The optimization variables include: i,k}, {p i,k}、{τ k}, K,

[0118] Among them, α i,k represents the transmission ratio of device i in the kth time slot, r represents the relay node, and p rk represents the transmission power of relay node r in the kth time slot, p i,k represents the transmission power of device i in the kth time slot, τ k represents the subslot fraction within the kth time slot;

[0119]

[0120] The above formula is constraint condition 1 of the optimization problem P, which represents the data balance at the relay node. Because in the k+1th time slot, the amount of data transmitted by device i at the relay node is the remaining amount of data transmitted at the previous time (k) plus the amount of data transmitted from device i to relay node r at time k+1 minus the amount of data transmitted from relay node r to the base station, that is, the formula Among them, Q i,k+1 represents the amount of data transmitted by device i in the k+1th time slot, T represents the length of a single time slot, represents the transmission rate of relay node r, Q i,k represents the amount of data transmitted by device i in the kth time slot, R i,k Indicates the transmission rate of device i;

[0121]

[0122]

[0123] The above formula is the constraint 2 and constraint 3 of the optimization problem P, which represents the energy balance at the relay node, because the energy consumed by the relay node in the kth time slot cannot exceed the maximum energy stored in the relay node, and in the k+1th time slot, the energy at the relay node is the remaining energy at the previous time (k) plus the energy obtained by the relay from the outside world at the k+1 time minus the energy consumed by the relay to the base station, that is, the formula Among them, E k+1 represents the energy of the relay node itself in the k+1th time slot, E max represents the maximum energy consumption at the relay node, Ek represents the energy of the relay node r itself in the kth time slot, E Ak represents the energy obtained by the relay node r from the outside world in the kth time slot;

[0124] D i (k)≥L, i∈I,

[0125] The above formula is the constraint condition 4 of the optimization problem P, where D i (k) represents the amount of data transmitted by device i in the first k time slots, L represents the minimum amount of data transmission required by the Internet of Things, and the above formula indicates that the amount of data transmitted by device i in k time slots must exceed the minimum amount of data transmission required by the Internet of Things;

[0126] 0≤τ k ≤1, k∈K,

[0127]

[0128]

[0129]

[0130]

[0131] The above formula is the constraint condition 5-9 of the optimization problem P, which represents the value range of the optimization variable, where: represents the maximum transmission power of device i, represents the maximum transmission power of relay node r, represents the minimum transmission delay required for device i to transmit data to the relay node, K Indicates the maximum transmission delay required for device i to transmit data to the relay node.

[0132] In this way, an optimization problem P is established with data balance and energy balance at the relay node as constraints, the transmission ratio of device i, the transmission power of device i and the relay node, and the sub-slot fraction as optimization variables, and minimizing the transmission delay when each device i transmits a certain amount of data as the optimization goal.

[0133] On the other hand, in this embodiment, since the optimization problem P is a mixed problem of discrete and continuous, the optimization problem P can be decomposed into sub-problems P-1 and P-2. In step S4, the expression of sub-problem P-1 is:

[0134]

[0135] The optimization variables include: i,k}, {p i,k}、{τ k},

[0136] The above formula is the objective function of subproblem P-1, which means that by jointly optimizing {α i,k}, {p i,k} and {τ k}These four optimization variables maximize the amount of data transmitted by device i within a fixed time slot;

[0137]

[0138] The above formula is constraint 1 of subproblem P-1, which indicates the data balance at the relay node;

[0139]

[0140]

[0141] The above formula is the constraint 2 and constraint 3 of subproblem P-1, which represents the energy balance at the relay node;

[0142] 0≤τ k ≤1, k∈K,

[0143]

[0144]

[0145]

[0146]

[0147] The above formula is the constraint conditions 4-8 of subproblem P-1, which represents the value range of the optimization variable.

[0148] On the other hand, in this embodiment, please refer to the attached Figure 3 In step S4, the method of using a neural network combined with a deep deterministic policy gradient algorithm to solve subproblem P-1 is:

[0149] Step S401, determine the action space information and state space information, the action space information includes the transmission ratio α of device i i,k , the transmission power of the relay node p rk , the transmission power p of device i i,k and the subslot fraction τ k , the state space information includes the amount of data Q transmitted by device i in the current time slot i,k And the energy E of the relay node itself in the kth time slot k .

[0150] Step S402, using the actor-critic algorithm framework to build a neural network model, the neural network model includes an actor network, a critic network and a reward function, the input of the actor network is state space information, and the output is an action space value, the input of the critic network is state space information and action space value, and the output is a state value function, and the reward function is used to describe the reward fed back from the environment after the neural network model takes each action in each state.

[0151] According to the actor-critic algorithm framework, the goal of subproblem P-1 is transformed into finding a strategy π(s,a) to maximize the sum of future reward values. Then the optimal strategy π * The expression is: Among them, E[] represents the expectation, γ i represents the discount factor, r i represents the reward function, s i represents the state space information, a i Represents action space information.

[0152] Step S403, initializing the parameters of the neural network model and defining the loss functions of the actor network and the critic network.

[0153] Among them, the parameters of the neural network model can be initialized using some common initialization methods, such as Xavier initialization, He initialization, etc.

[0154] Step S404: Use an optimizer to optimize the loss functions of the actor network and the critic network.

[0155] The operation of using an optimizer to optimize the loss functions of the actor network and the critic network in this embodiment is similar to the operation of optimizing the loss function of the deep reinforcement learning network by an optimizer in the prior art. Commonly used optimizers or optimization methods in the prior art include stochastic gradient descent (SGD), adaptive moment estimation (Adam), root mean square propagation (RMSprop) and temporal difference (TD) learning method, etc., which will not be repeated in the embodiments of the present application.

[0156] Step S405, using the experience replay method to randomly extract data from the experience pool to train the neural network model, repeatedly updating the actor network and the critic network until the optimal neural network model is obtained.

[0157] Step S406, input the current state space information into the trained actor network, output the optimal action space value and the maximum value of the transmission data volume of device i in each time slot, and use the optimal action space value as the optimal solution to subproblem P-1.

[0158] Among them, the optimal action space value is the optimal transmission ratio α of device i i,k , the transmission power of the relay node prk , the transmission power p of device i i,k and the subslot fraction τ k .

[0159] On the other hand, in this embodiment, after “step S404, using an optimizer to optimize the loss functions of the actor network and the critic network”, the following steps are also included:

[0160] Input state space information s k To the actor network, output action space value a k ;

[0161] Set the action space value a k Applied to the environment, the environment returns the state space information s for the next time slot k+1 and reward r k ;

[0162] Will s k 、a k 、r k and k+1 It is recorded as a four-tuple data and put into the experience pool.

[0163] On the other hand, in this embodiment, the actor network includes an actor online network θ μ and the actor-target network θ μ ′, the critic network includes the critic online network θ Q and the critic target network θ Q ′, the parameters of the online network are updated by the gradient descent method, and the parameters of the target network are updated at each time step. Figure 4 , then the sub-steps of step S405 include:

[0164] Step A01) Extract four-tuple data (s) from the experience pool k ,a k ,r k ,s k+1 ) as a sample, and take s in the sample k and a k Input into the critic online network, and get the transmission data volume Q of the current time slot = (s k ,a k );

[0165] Step A02) convert the s in the sample k+1 Input into the actor-target network to obtain the target action space value a′ k+1 , and s k+1 and a′ k+1 Input them into the critic target network together, and get the target transmission data volume Q′=(s k+1 ,a′ k+1);

[0166] Step A03) updating the actor online network, and updating the critic online network according to the action space value output by the actor online network, so that the Q value output by the critic online network increases;

[0167] Step A04) Repeat the above steps A01-A03 until the Q value output by the critic online network is equal to the target transmission data volume Q′ or the Q value is within the preset error range of the target transmission data volume Q′, and the neural network model at this time is taken as the optimal neural network model.

[0168] Preferably, the preset error range in this embodiment is 0.1.

[0169] Furthermore, in this embodiment, the loss function J of the actor online network is defined in step S403 i,k for:

[0170]

[0171] Among them, N is the total number of four-tuple data extracted from the experience pool, Q(s k ,a k θ Q ) is the amount of data transmitted by the critics’ online network output;

[0172] Define the loss function L of the critic online network i,k for:

[0173]

[0174] Where γ represents the discount factor, r k represents the reward function of the kth time slot, Q′(s k+1 ,a k+1 ′;θ Q′ ) represents the target transmission data volume output by the critic target network;

[0175] Update the weights θ of the actor online network using the following formula μ ′ and the weights θ of the critic online network Q ′:

[0176]

[0177] Wherein, β is a preset parameter, and in this embodiment, the preferred parameter β is 0.001.

[0178] Please refer to the attached diagram for the principle diagram of the deep deterministic policy gradient algorithm. Figure 5 , where Actor is the actor network and Critic is the critic network.

[0179] On the other hand, in this embodiment, in step S4, the expression of sub-problem P-2 is:

[0180] P-2: min k, k∈K

[0181] The optimization variables are: k,

[0182] The above formula is the objective function of subproblem P-2, which means minimizing the number of time slots under the amount of transmission data solved in subproblem P-1, thereby obtaining the minimized transmission delay;

[0183] D i (k)≥L, i∈I,

[0184] The above formula is constraint 1 of subproblem P-2, which means that the amount of data transmitted by device i in k time slots must exceed the minimum required amount of data transmission in the Internet of Things;

[0185]

[0186] The above formula is constraint 2 of subproblem P-2, which represents the value range of the transmission delay K.

[0187] On the other hand, in this embodiment, please refer to the attached Figure 6 In step S4, the method of using the binary search algorithm to solve subproblem P-2 is:

[0188] Step S411, determine the maximum number of time slots k upp And the minimum number of time slots k low , where k upp >k low , and obtain the minimum transmission data volume L required by the Internet of Things;

[0189] Step S412, calculate the current time slot number k mid =(k low +k upp ) / 2;

[0190] Step S413, if D i (k mid )≥L, then execute step S414, otherwise execute step S415;

[0191] Step S414, let k upp =k mid ;

[0192] Step S415, let k low =k mid ;

[0193] Step S416, repeat steps S412-S415 until k upp ≤k low, output the current time slot number k mid As the optimal solution to subproblem P-2, the optimal solution to the optimization problem P is obtained.

[0194] On the other hand, in this embodiment, the wireless channel parameters of the device i and the relay node in each time slot include the channel bandwidth, channel gain and background noise power from the device to the relay node r, the transmission power of the device, the channel bandwidth, channel gain and background noise power from the relay node to the base station and the transmission power of the relay node. Due to the use of a non-orthogonal multiple access transmission mode, when transmitting data uplink, the transmission rate of the device i and the relay node is expressed as follows:

[0195]

[0196]

[0197] Where i and j both represent devices, r represents a relay node, k represents the number of time slots, and R i,k represents the transmission rate of device i in the kth time slot, B 1 represents the channel bandwidth from the device to the relay node r, p i,k represents the transmission power of device i in the kth time slot, g i,k represents the channel gain from device i to the relay node in the kth time slot, p j,k represents the transmission power of device j in the kth time slot, g j,k represents the channel gain from device j to the relay node in the kth time slot, N 1 Represents the background noise power from the device to the relay node, R rk represents the transmission rate of relay node r in the kth time slot, B 2 represents the channel bandwidth from the relay node to the base station, p rk represents the transmission power of relay node r in the kth time slot, g rk represents the channel gain from relay node r to the base station in the kth time slot, N 2 represents the background noise power from relay node r to the base station.

[0198] The energy harvesting relay assisted Internet of Things transmission delay optimization method proposed in this embodiment is used to obtain the optimal transmission delay K when each device i transmits a certain amount of data. When the number I of Internet of Things devices is 3, 5, and 7, respectively, simulation experiments are performed on the relationship between the amount of data transmitted and the delay at a fixed transmission power and the relationship between the amount of data transmitted and the delay at the optimal power corresponding to the optimal transmission delay K obtained in this embodiment. The simulation experiment results are shown in the attached figure. Figure 7 As shown in the figure, “-*-” indicates the relationship between the amount of transmitted data and the delay under a fixed transmission power. It represents the relationship between the amount of data transmitted and the delay under the optimal power corresponding to the optimal transmission delay K obtained in this embodiment. The simulation results verify that the transmission delay optimization method for the energy harvesting relay assisted Internet of Things proposed in this embodiment has produced substantial beneficial technical effects in reducing transmission delays.

[0199] In summary, under the constraints of data balance and energy balance, the embodiment of the present application decomposes the optimization problem of the transmission delay of the energy harvesting relay-assisted Internet of Things, considers the wireless channel parameters such as the limited data buffer of the relay, the limited energy storage of the relay, the channel gain, the channel bandwidth, and the noise power, and adopts the deep deterministic policy gradient algorithm to obtain the optimal transmission ratio α of device i. i,k , the transmission power of the relay node p rk , the transmission power p of device i i,k , sub-slot fraction τ k And the maximum value of the transmission data amount of device i in each time slot, and then use the binary search algorithm to obtain the optimal transmission time slot of the optimization problem, so as to minimize the transmission delay of the energy harvesting relay assisted Internet of Things, thereby improving the transmission rate of the energy harvesting relay assisted Internet of Things, saving network resources and increasing the system benefits of the entire network.

[0200] Embodiment 2:

[0201] The embodiment of the present application also provides a transmission delay optimization device for energy harvesting relay-assisted Internet of Things, please refer to the attached Figure 8 ,include:

[0202] Parameter acquisition module 1, used to divide the transmission delay K required for device i to transmit data to the relay node into k time slots, and obtain the wireless channel parameters of device i and the relay node in each time slot;

[0203] The transmission rate calculation module 2 is used to calculate the transmission rate of the device i and the relay node according to the wireless channel parameters of the device i and the relay node in each time slot;

[0204] The optimization problem establishment module 3 is used to establish the optimization problem P with the data balance and energy balance at the relay node as constraints, the transmission ratio of device i, the transmission power of device i and the relay node, and the sub-time slot fraction as optimization variables, and the optimization goal of minimizing the transmission delay when each device i transmits a certain amount of data;

[0205] The optimization problem solving module 4 is used to decompose the optimization problem P into two sub-problems P-1 and P-2, and use a neural network combined with a deep deterministic policy gradient algorithm to solve the sub-problem P-1. According to the solution result of the sub-problem P-1, a binary search algorithm is used to solve the sub-problem P-2, thereby obtaining the optimal solution of the optimization problem P.

[0206] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Those skilled in the art should understand that the present invention includes but is not limited to the contents described in the drawings and the above specific embodiments. Any modification that does not deviate from the functional and structural principles of the present invention will be included in the scope of the claims.

Claims

1. A transmission delay optimization method for energy harvesting relay-assisted Internet of Things, characterized in that: The following steps are involved: Step S1, placing a relay node in the base station, placing I IoT devices i within the communication range of the base station, represented by the set i = {1, 2, ..., I}, the relay node is used to receive data transmitted to it by the IoT device i, dividing the transmission delay K required for the device i to transmit data to the relay node into k time slots, and obtaining the wireless channel parameters of the device i and the relay node in each time slot; Step S2, calculating the transmission rate of device i and the relay node according to the wireless channel parameters of device i and the relay node in each time slot; Step S3, taking the data balance and energy balance at the relay node as constraints, taking the transmission ratio of device i, the transmission power of device i and the relay node, and the sub-slot fraction as optimization variables, and taking minimizing the transmission delay when each device i transmits a certain amount of data as the optimization goal, an optimization problem P is established, wherein the expression of the optimization problem P is: P: min K The above formula is the objective function of the optimization problem P, which represents minimizing the transmission delay when each device i transmits a certain amount of data, where K represents the transmission delay required for device i to transmit data to the relay node; The optimization variables include: i,k }、 {p i,k }、{τ k }, K, Among them, α i,k represents the transmission ratio of device i in the kth time slot, r represents the relay node, and p rk represents the transmission power of relay node r in the kth time slot, p i,k represents the transmission power of device i in the kth time slot, τ k represents the subslot fraction within the kth time slot; The above formula is the constraint condition 1 of the optimization problem P, which represents the data balance at the relay node, where T represents the length of a single time slot, represents the transmission rate of relay node r, Q i,k represents the amount of data transmitted by device i in the kth time slot, R i,k Indicates the transmission rate of device i; The above formula is the constraint 2 and constraint 3 of the optimization problem P, which represents the energy balance at the relay node, where E max represents the maximum energy consumption at the relay node, E k represents the energy of the relay node r itself in the kth time slot, E Ak represents the energy obtained by the relay node r from the outside world in the kth time slot; D i (k)≥L,i∈I, The above formula is the constraint condition 4 of the optimization problem P, where D i (k) represents the amount of data transmitted by device i in the first k time slots, L represents the minimum amount of data transmission required by the Internet of Things, and the above formula indicates that the amount of data transmitted by device i in k time slots must exceed the minimum amount of data transmission required by the Internet of Things; 0≤τ k ≤1,k∈K, The above formula is the constraint condition 5-9 of the optimization problem P, which represents the value range of the optimization variable, where: represents the maximum transmission power of device i, represents the maximum transmission power of relay node r, represents the minimum transmission delay required for device i to transmit data to the relay node, K represents the maximum transmission delay required for device i to transmit data to the relay node; Step S4, decomposing the optimization problem P into two sub-problems P-1 and P-2, using a neural network combined with a deep deterministic policy gradient algorithm to solve sub-problem P-1, and according to the solution result of sub-problem P-1, using a binary search algorithm to solve sub-problem P-2, so as to obtain the optimal solution of the optimization problem P; Among them, in step S4, the method of using a neural network combined with a deep deterministic policy gradient algorithm to solve subproblem P-1 is: Step S401, determining action space information and state space information, wherein the action space information includes the transmission ratio α of device i i,k , the transmission power of the relay node p rk , the transmission power p of device i i,k and the subslot fraction τ k The state space information includes the amount of data Q transmitted by device i in the current time slot i,k And the energy E of the relay node itself in the kth time slot k ; Step S402, using the actor-critic algorithm framework to construct a neural network model, the neural network model includes an actor network, a critic network and a reward function, the input of the actor network is state space information, and the output is an action space value, the input of the critic network is state space information and action space value, and the output is a state value function, and the reward function is used to describe the reward fed back from the environment after the neural network model takes each action in each state; Step S403, initializing the parameters of the neural network model and defining the loss functions of the actor network and the critic network; Step S404, using an optimizer to optimize the loss functions of the actor network and the critic network; Step S405, randomly extracting data from the experience pool using the experience replay method to train the neural network model, repeatedly updating the actor network and the critic network until the optimal neural network model is obtained; Step S406, input the current state space information into the trained actor network, output the optimal action space value and the maximum value of the transmission data volume of device i in each time slot, and use the optimal action space value as the optimal solution to subproblem P-1.

2. The method for optimizing transmission delay of an energy harvesting relay-assisted Internet of Things according to claim 1, characterized in that: In step S4, the expression of sub-problem P-1 is: The optimization variables include: i,k }、 {p i,k }、{τ k }, The above formula is the objective function of subproblem P-1, which means that by jointly optimizing {α i,k }、 {p i,k } and {τ k }These four optimization variables maximize the amount of data transmitted by device i within a fixed time slot; The above formula is constraint 1 of subproblem P-1, which indicates the data balance at the relay node; The above equations are constraints 2 and 3 of subproblem P-1, which represent the energy balance at the relay node; The above formula is the constraint conditions 4-8 of subproblem P-1, which represents the value range of the optimization variable.

3. The method for optimizing transmission delay of an energy harvesting relay-assisted Internet of Things according to claim 1, characterized in that: After "step S404, using an optimizer to optimize the loss functions of the actor network and the critic network", the following steps are also included: Input state space information s k To the actor network, output action space value a k ; Set the action space value a k Applied to the environment, the environment returns the state space information s for the next time slot k+1 and reward r k ; Will s k 、a k 、r k and k+1 It is recorded as a four-tuple data and put into the experience pool.

4. The method for optimizing transmission delay of an energy harvesting relay-assisted Internet of Things according to claim 3, characterized in that: The actor network includes an actor online network and an actor target network, and the critic network includes a critic online network and a critic target network. The sub-steps of step S405 include: Step A01) Extract four-tuple data (s) from the experience pool k ,a k ,r k ,s k+1 ) as a sample, and take s in the sample k and a k Input into the critic online network, and get the transmission data volume Q of the current time slot = (s k ,a k ); Step A02) convert the s in the sample k+1 Input into the actor-target network to obtain the target action space value a ′ k+1 , and s k+1 with a ′ k+1 Input them into the critic target network together to get the target transmission data volume Q ′ =(s k+1 ,a ′ k+1 ); Step A03) updating the actor online network, and updating the critic online network according to the action space value output by the actor online network, so that the Q value output by the critic online network increases; Step A04) Repeat the above steps A01)-A03) until the Q value output by the critic online network is equal to the target transmission data volume Q ′ Or the Q value is the target transmission data volume Q ′ Within the preset error range, the neural network model at this time is taken as the optimal neural network model.

5. The method for optimizing transmission delay of an energy harvesting relay-assisted Internet of Things according to claim 1, characterized in that: In step S4, the expression of sub-problem P-2 is: P-2: mink, k∈K The optimization variables are: k, The above formula is the objective function of subproblem P-2, which means minimizing the number of time slots under the amount of transmission data solved in subproblem P-1, thereby obtaining the minimized transmission delay; D i (k)≥L,i∈I, The above formula is constraint 1 of subproblem P-2, which means that the amount of data transmitted by device i in k time slots must exceed the minimum required amount of data transmission of the Internet of Things; The above formula is constraint 2 of subproblem P-2, which represents the value range of the transmission delay K.

6. The method for optimizing transmission delay of an energy harvesting relay-assisted Internet of Things according to claim 5, characterized in that: In step S4, the method of using the binary search algorithm to solve subproblem P-2 is: Step S411, determine the maximum number of time slots k upp And the minimum number of time slots k low , where k upp >k low , and obtain the minimum transmission data volume L required by the Internet of Things; Step S412, calculate the current time slot number k mid =(k low +k upp ) / 2; Step S413, if D i (k mid )≥L, then execute step S414, otherwise execute step S415; Step S414, let k upp =k mid ; Step S415, let k low =k mid ; Step S416, repeat steps S412-S415 until k upp ≤k low , output the current time slot number k mid As the optimal solution to subproblem P-2, the optimal solution to the optimization problem P is obtained.

7. A method for optimizing transmission delay of an energy harvesting relay-assisted Internet of Things according to any one of claims 1 to 6, characterized in that: The wireless channel parameters of the device i and the relay node in each time slot include the channel bandwidth, channel gain and background noise power from the device to the relay node r, the transmission power of the device, the channel bandwidth, channel gain and background noise power from the relay node to the base station and the transmission power of the relay node. Then, in step S2, according to the wireless channel parameters of the device i and the relay node in each time slot, the method for calculating the transmission rate of the device i and the relay node includes: Where i and j both represent devices, r represents a relay node, k represents the number of time slots, and R i,k represents the transmission rate of device i in the kth time slot, B1 represents the channel bandwidth from the device to the relay node r, and p i,k represents the transmission power of device i in the kth time slot, g i,k represents the channel gain from device i to the relay node in the kth time slot, p j,k represents the transmission power of device j in the kth time slot, g j,k represents the channel gain from device j to the relay node in the kth time slot, N1 represents the background noise power from the device to the relay node, and R rk represents the transmission rate of the relay node r in the kth time slot, B2 represents the channel bandwidth from the relay node to the base station, and p rk represents the transmission power of relay node r in the kth time slot, g rk represents the channel gain from the relay node r to the base station in the kth time slot, and N2 represents the background noise power from the relay node r to the base station.

8. A transmission delay optimization device for energy harvesting relay-assisted Internet of Things, characterized in that: include: A parameter acquisition module, used to divide the transmission delay K required for device i to transmit data to the relay node into k time slots, and obtain the wireless channel parameters of device i and the relay node in each time slot; A transmission rate calculation module, used to calculate the transmission rate of device i and the relay node according to the wireless channel parameters of device i and the relay node in each time slot; The optimization problem establishment module is used to establish the optimization problem P with the data balance and energy balance at the relay node as constraints, the transmission ratio of device i, the transmission power of device i and the relay node, and the sub-time slot fraction as optimization variables, and the optimization goal of minimizing the transmission delay when each device i transmits a certain amount of data. The expression of the optimization problem P is: P:minK The above formula is the objective function of the optimization problem P, which represents minimizing the transmission delay when each device i transmits a certain amount of data, where K represents the transmission delay required for device i to transmit data to the relay node; The optimization variables include: i,k }、 {p i,k }、{τ k }, K, Among them, α i,k represents the transmission ratio of device i in the kth time slot, r represents the relay node, and p rk represents the transmission power of relay node r in the kth time slot, p i,k represents the transmission power of device i in the kth time slot, τ k represents the subslot fraction within the kth time slot; The above formula is the constraint condition 1 of the optimization problem P, which represents the data balance at the relay node, where T represents the length of a single time slot, represents the transmission rate of relay node r, Q i,k represents the amount of data transmitted by device i in the kth time slot, R i,k Indicates the transmission rate of device i; The above formula is the constraint 2 and constraint 3 of the optimization problem P, which represents the energy balance at the relay node, where E max represents the maximum energy consumption at the relay node, E k represents the energy of the relay node r itself in the kth time slot, E Ak represents the energy obtained by the relay node r from the outside world in the kth time slot; D i (k)≥L,i∈I, The above formula is the constraint condition 4 of the optimization problem P, where D i (k) represents the amount of data transmitted by device i in the first k time slots, L represents the minimum amount of data transmission required by the Internet of Things, and the above formula indicates that the amount of data transmitted by device i in k time slots must exceed the minimum amount of data transmission required by the Internet of Things; 0≤τ k ≤1,k∈K, The above formula is the constraint condition 5-9 of the optimization problem P, which represents the value range of the optimization variable, where: represents the maximum transmission power of device i, represents the maximum transmission power of relay node r, represents the minimum transmission delay required for device i to transmit data to the relay node, K represents the maximum transmission delay required for device i to transmit data to the relay node; The optimization problem solving module is used to decompose the optimization problem P into two sub-problems P-1 and P-2, and use a neural network combined with a deep deterministic policy gradient algorithm to solve the sub-problem P-1. Based on the solution of the sub-problem P-1, a binary search algorithm is used to solve the sub-problem P-2, thereby obtaining the optimal solution to the optimization problem P; Among them, the optimization problem solving module uses a neural network combined with a deep deterministic policy gradient algorithm to solve the subproblem P-1 and performs the following steps: Step S401, determining action space information and state space information, wherein the action space information includes the transmission ratio α of device i i,k , the transmission power of the relay node p rk , the transmission power p of device i i,k and the subslot fraction τ k The state space information includes the amount of data Q transmitted by device i in the current time slot i,k And the energy E of the relay node itself in the kth time slot k ; Step S402, using the actor-critic algorithm framework to construct a neural network model, the neural network model includes an actor network, a critic network and a reward function, the input of the actor network is state space information, and the output is an action space value, the input of the critic network is state space information and action space value, and the output is a state value function, and the reward function is used to describe the reward fed back from the environment after the neural network model takes each action in each state; Step S403, initializing the parameters of the neural network model and defining the loss functions of the actor network and the critic network; Step S404, using an optimizer to optimize the loss functions of the actor network and the critic network; Step S405, randomly extracting data from the experience pool using the experience replay method to train the neural network model, repeatedly updating the actor network and the critic network until the optimal neural network model is obtained; Step S406, input the current state space information into the trained actor network, output the optimal action space value and the maximum value of the transmission data volume of device i in each time slot, and use the optimal action space value as the optimal solution to subproblem P-1.

Images