A method for optimizing the long-term average transmission rate of relay-assisted Internet of Things

Abstract

This invention relates to the field of Internet of Things (IoT) technology, specifically to a method for optimizing the long-term time-average transmission rate of relay-assisted IoT. The method includes: acquiring the wireless channel parameters of the device and relay node i in each time slot; establishing an optimization problem P with data balance and average transmission power balance at the relay node as constraints, and relay selection indicator variables, device transmission power, relay node transmission power, and sub-time slot fractions as optimization variables, with the optimization objective of maximizing the long-term time-average transmission rate of the relay-assisted IoT; transforming optimization problem P into optimization problem P-1 based on the data storage capacity and remaining energy of the relay node, and solving it using a dual-delay deep deterministic policy gradient algorithm to obtain the optimal solution to optimization problem P. This method maximizes the long-term time-average transmission rate of the relay-assisted IoT, improves the data transmission efficiency of the relay-assisted IoT, and thus achieves a better wireless network experience.

Description

Technical Field

[0001] The present invention relates to the technical field of Internet of Things, and in particular to a method for optimizing the long-term average transmission rate of a relay-assisted Internet of Things. Background Art

[0002] Relay-assisted Internet of Things (RAIoT) refers to the technology of using relay nodes to realize data transmission and communication in the Internet of Things. In the Internet of Things, since IoT devices usually have the characteristics of low power consumption, low data rate, and low computing power, their transmission distance and transmission efficiency are limited. RAIoT technology can expand the coverage of the Internet of Things and improve the communication quality by introducing relay nodes. The specific implementation of RAIoT technology is to introduce one or more relay nodes between the IoT device and the base station. The relay node can receive the signal of the IoT device and forward it to the base station. The relay node can communicate with the IoT device and the base station through wireless signals, optical fibers, power lines, etc. The relay node can use its own computing power and storage resources to process and store the data of the IoT device, thereby reducing the burden on the base station. RAIoT technology can be applied to many scenarios, such as urban smart transportation, smart home, smart medical care, etc. By introducing relay nodes, RAIoT technology can improve the coverage and communication quality of the Internet of Things, thereby realizing more reliable and efficient IoT applications.

[0003] The time-average transmission rate refers to the ratio of the amount of data transmitted in a certain period of time to the time interval, usually expressed in bits per second (bps). The time-average transmission rate can reflect the average speed of data transmission and is one of the important indicators of network performance. In network monitoring and measurement, the time-average transmission rate is usually used to evaluate network quality and performance, and as a basis for network optimization and improvement. For relay-assisted Internet of Things, the time-average transmission rate affects the communication efficiency of the Internet of Things and the quality of data transmission. Specifically, a high time-average transmission rate means high data transmission efficiency, which can complete data transmission and reception faster, thereby improving the communication efficiency of the relay-assisted Internet of Things. Moreover, for some applications with high real-time requirements, such as smart transportation, smart medical care, etc., the speed of the time-average transmission rate directly determines the real-time and response speed of the application, and becomes one of the important indicators for evaluating wireless network service quality and user experience.

[0004] The relay-assisted Internet of Things in the prior art mostly uses frequency division multiple access (FDMA) technology for data transmission. In FDMA, the channel bandwidth is divided into several frequency bands, each of which is assigned to a user. The frequency bands between users are independent of each other, so they can communicate at the same time, avoiding interference between users. The data of each user is modulated into a frequency corresponding to the frequency band assigned to it, and then transmitted on the frequency band. The receiving end demodulates according to the frequency band assigned to it and restores the data to the original information. FDMA technology has the advantages of high spectrum utilization and strong anti-interference, but because of the need for frequency allocation, channel resources need to be divided, so frequency division is required during implementation, which is more complex. At the same time, in FDMA, the sub-channels between users are independent of each other, resulting in low spectrum utilization.

[0005] In general, due to the influence of relay-assisted IoT multi-user access technology, the impact of interference between relay nodes on the transmission rate and quality of the network, and the capacity limitations of IoT devices and relay nodes themselves, the current long-term average transmission rate of relay-assisted IoT still cannot achieve efficient data transmission. Summary of the invention

[0006] The technical problem to be solved by the present invention is: At present, the relay-assisted Internet of Things has a technical problem that the long-term time-average transmission rate still cannot achieve efficient data transmission effect. A method for optimizing the long-term time-average transmission rate of the relay-assisted Internet of Things is proposed, which aims to design a relay-assisted Internet of Things with higher transmission performance, optimize the long-term time-average rate of the network, and improve data transmission efficiency, so as to achieve better wireless network experience quality.

[0007] In order to solve the above technical problems, the present invention adopts the following technical solution: a method for optimizing the long-term average transmission rate of a relay-assisted Internet of Things, comprising the following steps:

[0008] Step S1, placing I relay nodes i within the communication range of the base station, represented by the set i = {1, 2, ..., I}, the IoT device sends data to the base station through the relay node, and the device and the relay node as well as the relay node and the base station communicate using a non-orthogonal multiple access transmission mode, dividing the total number of time slots K for the data transmission of the relay node i in the relay-assisted IoT into k time slots, and obtaining the wireless channel parameters of the device and the relay node i in each time slot;

[0009] Step S2, taking the data balance and the average transmission power balance at the relay node as the constraint conditions, taking the relay selection indicator variable, the transmission power of the device, the transmission power of the relay node and the sub-time slot fraction as the optimization variables, and taking maximizing the long-term average transmission rate of the relay-assisted Internet of Things as the optimization goal to establish an optimization problem P;

[0010] Step S3, calculating the data storage capacity and the residual energy of the relay node, and transforming the optimization problem P into the optimization problem P-1 according to the data storage capacity and the residual energy of the relay node, wherein the optimization problem P-1 takes maximizing the average transmission rate within the fixed time slot of the relay-assisted Internet of Things as the optimization goal;

[0011] Step S4, using a double-delayed deep deterministic policy gradient algorithm to solve the optimization problem P-1, thereby obtaining the optimal solution to the optimization problem P.

[0012] Among them, Non-Orthogonal Multiple Access (NOMA) is a wireless communication transmission method that allows multiple users to communicate simultaneously on the same frequency band, improving spectrum utilization efficiency. Specifically, non-orthogonal multiple access technology superimposes the data of different users on the same signal for transmission, and then separates the data of different users through channel decoding technology. This technology can make full use of spectrum resources, improve spectrum utilization efficiency, support large-scale connections, and ensure transmission efficiency while connecting multiple users. Non-orthogonal multiple access technology is widely used and can be used in 5G and 6G mobile communications, the Internet of Things, smart homes and other fields.

[0013] Among them, the total number of time slot transmissions refers to the maximum number of time slots that can be transmitted within a specified time period in a wireless communication system. A time slot refers to the basic unit in which a communication system communicates at fixed time intervals within a period of time. In order to allocate time slots more flexibly, the time slot can be divided into several sub-time slot fractions, that is, a sub-time slot fraction is a fractional part of a time slot. The double-delayed deep deterministic policy gradient algorithm is a deep reinforcement learning algorithm for solving continuous control problems. It is an improved version of the Deep Deterministic Policy Gradient algorithm (DDPG). The present invention improves the performance and stability of the algorithm by introducing two delay networks and some optimization techniques.

[0014] The technical concept of the present invention is as follows: First, in the relay-assisted Internet of Things, based on the data balance at the relay node and the constraint premise of the average transmission power balance, an optimization problem P of the long-term average transmission rate of the relay-assisted Internet of Things that performs data transmission through non-orthogonal multiple access technology is constructed. Since this problem is a complex nonlinear problem, it is very difficult to solve it directly. Therefore, the data storage capacity and residual energy of the relay node are introduced to transform the optimization problem P into the optimization problem P-1. Combined with the analysis of the optimization problem P-1, a double-delay deep deterministic policy gradient algorithm for solving the optimization problem P-1 is proposed, and then the solution of the original optimization problem P is obtained.

[0015] By constructing an optimization problem P of the long-term time-averaged transmission rate of the relay-assisted Internet of Things that transmits data through non-orthogonal multiple access technology, and analyzing the characteristics of the problem, the data storage capacity and residual energy of the relay node are introduced, the optimization problem P is transformed into the optimization problem P-1, and a double-delay deep deterministic policy gradient algorithm is proposed to solve the optimization problem P-1, so as to obtain the optimal solution of the original optimization problem P. Under the constraints of ensuring data balance at the relay node and average transmission power balance, the long-term time-averaged transmission rate of the relay-assisted Internet of Things is maximized, the data transmission efficiency of the relay-assisted Internet of Things is improved, and a better wireless network experience quality is achieved.

[0016] Preferably, in step S2, the expression of the optimization problem P is:

[0017]

[0018] The above formula is the objective function of the optimization problem P, which represents maximizing the long-term average transmission rate of relay node i in the relay-assisted Internet of Things, where K represents the total number of time slots for relay node i in the relay-assisted Internet of Things to transmit data, k represents the number of time slots, and τ k represents the sub-slot fraction within the kth time slot, I represents the total number of relay nodes, and D i,k represents the amount of data received by relay node i in the kth time slot, E[D i,k ] indicates D i,k Seek expectations;

[0019] The optimization variables include: i,k},,{p k},,{p i,k}、{τ k},

[0020] Among them, m i,k Indicates whether the relay node i is selected to receive information in the kth time slot, that is, the relay selection indicator variable, p k represents the transmission power of the device in the kth time slot, p i,krepresents the transmission power of relay node i in the kth time slot;

[0021]

[0022] The above formula is the constraint condition 1 of the optimization problem P, which means that the relay node i is selected to receive information in the kth time slot;

[0023]

[0024] The above formula is the constraint condition 2 of the optimization problem P, which means that the amount of data received at the relay node should not be greater than the amount of data transmitted, that is, the data balance at the relay node, where the amount of data D received by the relay node i in the kth time slot is i,k =m i,k τ k TR k , where T represents the length of a single time slot, R k represents the transmission rate of the device in the kth time slot, D i ′ ,k represents the amount of data transmitted by relay node i in the kth time slot, D i ′ ,k =R i,k (1-τ k )T, where R i,k represents the transmission rate of relay node i in the kth time slot;

[0025]

[0026]

[0027] The above formula is the constraint 3 and constraint 4 of the optimization problem P, which means that the transmission power from the device to the relay node i should be equal to the average transmission power of the device, and the transmission power from the relay node i to the base station should be equal to the average transmission power of the relay node i, that is, the average transmission power is balanced, where, represents the average transmission power of the device in the kth time slot, represents the average transmission power of relay node i in the kth time slot;

[0028] 0≤p k ≤p max ,

[0029]

[0030] 0≤m i,k ≤1

[0031] 0≤τ k ≤1

[0032] The above formula is the constraint condition 5-8 of the optimization problem P, which represents the value range of the optimization variable, where p max Indicates the maximum transmission power of the device. represents the maximum transmission power of relay node i in the kth time slot, if m i,k >0.5, then let m i,k =1, that is, relay node i is considered to be selected for information reception.

[0033] Preferably, in step S3, the method for calculating the data storage capacity of the relay node includes:

[0034]

[0035] Where K represents the total number of time slots in which relay node i transmits data in the relay-assisted IoT, k represents the number of time slots, I represents the total number of relay nodes, and C i,k+1 represents the data storage capacity of relay node i in the k+1th time slot, C i,k represents the amount of data stored in relay node i in the kth time slot, R i,k (1-τ k )T represents the amount of data transmitted by relay node i in the kth time slot, where R i,k represents the transmission rate of relay node i in the kth time slot, τ k represents the sub-slot fraction within the kth time slot, T represents the length of a single time slot, and m i,k τ k TR k represents the amount of data received by relay node i in the kth time slot, where m i,k Indicates whether the relay node i is selected to receive information in the kth time slot, that is, the relay selection indicator variable, R k Indicates the transmission rate of the device in the kth time slot.

[0036] Preferably, in step S3, the method for calculating the residual energy of the relay node includes:

[0037]

[0038] Among them, E i,k+1 represents the remaining energy of relay node i in the k+1th time slot, E i,k represents the energy storage capacity of relay node i in the kth time slot, p i,k (1-τ k )T represents the energy consumed by relay node i in the kth time slot, where p i,k represents the transmission power of relay node i in the kth time slot, E new Represents the energy obtained by the relay node from the outside world.

[0039] Preferably, in step S3, the optimization problem P is converted into the optimization problem P-1 according to the data storage capacity and the residual energy of the relay node, and the expression of the converted optimization problem P-1 is:

[0040]

[0041] The above formula is the objective function of the optimization problem P-1, which represents maximizing the average transmission rate of relay node i in the relay-assisted Internet of Things within a fixed time slot, where K represents the total number of time slots for relay node i in the relay-assisted Internet of Things to transmit data, k represents the number of time slots, and τ k represents the sub-slot fraction within the kth time slot, I represents the total number of relay nodes, and D i,k represents the amount of data received by relay node i in the kth time slot;

[0042] The optimization variables include: i,k},,{p k},,{p i,k}、{τ k},

[0043] Among them, m i,k Indicates whether the relay node i is selected to receive information in the kth time slot, that is, the relay selection indicator variable, p k represents the transmission power of the device in the kth time slot, p i,k represents the transmission power of relay node i in the kth time slot;

[0044]

[0045] The above formula is the constraint 1 of the optimization problem P-1, which means that the relay node i is selected to receive information in the kth time slot;

[0046] C i,k -R i,k (1-τ k )T+m i,k τ k TR k >0

[0047] The above formula is the constraint 2 of the optimization problem P-1, which means that the data storage capacity of the relay node should be greater than zero;

[0048] 0≤p i,k (1-τ k )T≤E max

[0049] The above formula is the constraint 3 of the optimization problem P-1, which means that the energy consumed by relay node i cannot exceed the maximum energy that the relay node can store, where E maxrepresents the maximum energy that the relay node can store;

[0050] E i,k -p i,k (1-τ k )T+E new >0

[0051] The above formula is constraint 4 of the optimization problem P-1, which means that the residual energy of the relay node should be greater than zero;

[0052] 0≤p k ≤p max ,

[0053]

[0054] 0≤m i,k ≤1

[0055] 0≤τ k ≤1

[0056] The above formula is the constraint conditions 5-8 of the optimization problem P-1, which represents the value range of the optimization variable, where p max Indicates the maximum transmission power of the device. represents the maximum transmission power of relay node i in the kth time slot, if m i,k >0.5, then let m i,k =1, that is, relay node i is considered to be selected for information reception.

[0057] Preferably, in step S4, the method of using a double-delayed deep deterministic policy gradient algorithm to solve the optimization problem P-1 includes:

[0058] Step S401, define the state space s at the kth time slot k =(C i,k ,E i,k ), the action space a at the kth time slot k =(m i,k ,p k ,p i,k ,τ k ) and the reward function r;

[0059] Step S402, using the actor-critic algorithm framework to build a neural network model, the neural network model includes an actor network, a first critic network and a second critic network, the input of the actor network is the state space s k , the output is the action space a k , the first critic network and the second critic network are used to estimate the value of the current state space and action space;

[0060] Step S403, initializing the parameters of the neural network model;

[0061] Step S404, collect status report pair (s k ,a k ,r k ,s k+1 ) into the experience pool, where r k represents the reward value returned in the kth time slot, i.e., the solution to the objective function of the optimization problem P-1, s k+1 represents the state space at the k+1th time slot (C i,k+1 ,E i,k+1 );

[0062] Step S405: extract state report pairs (s k ,a k ,r k ,s k+1 ) Train the neural network model, repeatedly update the actor network and the critic network until the optimal neural network model is obtained;

[0063] Step S406: Input the current state space into the trained actor network, output the optimal action space and the returned reward value r k , which is the optimal solution to the objective function of the optimization problem P-1.

[0064] Preferably, the actor network includes an actor online network and an actor target network, the first critic network includes a first critic online network and a first critic target network, and the second critic network includes a second critic online network and a second critic target network. Then, the sub-steps of step S405 include:

[0065] Step A01, extract state reward pairs (s k ,a k ,r k ,s k+1 ) trains the neural network model and calculates the target value using the first critic target network and the second critic target network;

[0066] Step A02: Based on the evaluation value output by the first critic online network and the evaluation value output by the second critic online network The error between the target value and L i , update the parameters of the first critic online network and the second critic online network, and put the state-return pair output by the neural network model into the experience pool;

[0067] Step A03, repeatedly executing step A01 and step A02 until the experience pool reaches the maximum storage capacity N, then stopping updating the first critic online network and the second critic online network;

[0068] Step A04, calculating the error J between the first critic online network and the second critic online network i , according to the error J i Update the parameters of the actor online network;

[0069] Step A05: Repeat steps A01 to A04 until the evaluation values ​​output by the online network of the two critics are equal to the target value or the error L i Within the preset error range, the neural network model at this time is taken as the optimal neural network model.

[0070] Preferably, the sub-steps of step A01 include:

[0071] Return the status to (s k ,a k ,r k ,s k+1 ) is input into the actor-target network and outputs the target action space a in the next state space ′ k ;

[0072] For the target action space a ′ k After adding the perturbation, they are input into the first critic target network and the second critic target network respectively;

[0073] Select the smaller value of the output value of the first critic target network and the output value of the second critic target network;

[0074] The selected smaller value is discounted and weighted, and the reward value is added, and the calculated result is used as the target value.

[0075] As a preferred embodiment, in step A02, the error L i The expression is:

[0076]

[0077] When i=1, L1 represents the evaluation value output by the first critic online network. The error between the target value and represents the evaluation value of the first critic's online network output. When i=2, L2 represents the evaluation value of the second critic's online network output. The error between the target value and represents the evaluation value of the second critic's online network output, y(s k ,ak ) represents the target value;

[0078] According to the error L i The method for updating the parameters in the first critic online network and the second critic online network is:

[0079]

[0080] Among them, when i=1, represents the updated parameters of the first critic online network, represents the parameters of the first critic online network before updating, ρ represents the preset learning rate of the critic online network, represents the gradient descent value of the first critic's online network parameters, when i = 2, represents the updated parameters of the second critic online network, represents the parameters of the second critic online network before updating, Represents the gradient descent value of the second critic online network parameters.

[0081] Preferably, in step A04, the error J between the first critic online network and the second critic online network is i The calculation method is:

[0082]

[0083] Where N represents the maximum storage capacity of the experience pool. When i=1, J1 represents the error of the first critic's online network. represents the evaluation value of the first critic online network output, when i=2, J2 represents the error of the second critic online network, represents the evaluation value of the second critic's online network output;

[0084] According to the error J i The method to update the parameters of the actor online network is:

[0085]

[0086] in, represents the updated parameters of the actor online network, Indicates the parameters of the actor before the online network update, represents the preset learning rate of the actor online network, Represents the gradient descent value of the actor's online network parameters.

[0087] The beneficial technical effects of the present invention include: adopting a method for optimizing the long-term time-average transmission rate of a relay-assisted Internet of Things, constructing an optimization problem P of the long-term time-average transmission rate of the relay-assisted Internet of Things that performs data transmission through non-orthogonal multiple access technology, analyzing the characteristics of the problem, introducing the data storage capacity and residual energy of the relay node, transforming the optimization problem P into an optimization problem P-1, and proposing a double-delay deep deterministic policy gradient algorithm to solve the optimization problem P-1, thereby obtaining the optimal solution of the original optimization problem P, and achieving the maximization of the long-term time-average transmission rate of the relay-assisted Internet of Things under the constraint of ensuring data balance at the relay node and average transmission power balance, improving the data transmission efficiency of the relay-assisted Internet of Things, and thus achieving a better quality of wireless network experience quality; updating the online network of the critic multiple times and then updating the online network of the actor, thereby ensuring that the training of the online network of the actor is more stable; adding disturbances to the target action space of the next state when calculating the target value, thereby making the value evaluation of the subsequent critic network more accurate, and selecting the smaller value of the output value of the first critic target network and the output value of the second critic target network when calculating the target value, thereby suppressing the problem of network over-estimation.

[0088] Other features and advantages of the present invention will be disclosed in detail in the following specific embodiments and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0089] The present invention will be further described below in conjunction with the accompanying drawings:

[0090] Figure 1 The present invention is a flowchart of a method for optimizing the long-term average transmission rate of a relay-assisted Internet of Things according to an embodiment of the present invention.

[0091] Figure 2 Schematic diagram of the principle of the double-delay deep deterministic policy gradient algorithm according to an embodiment of the present invention.

[0092] Figure 3(a) , 3(b) And 3(c) are schematic diagrams of simulation experimental results comparing the average transmission rate and time slot of the technical solution of the present application and the relay-assisted Internet of Things of FDMA when the number of relay nodes in the embodiments of the present invention is 3, 5, and 7, respectively. DETAILED DESCRIPTION

[0093] The technical solutions of the embodiments of the present invention are explained and described below in conjunction with the drawings of the embodiments of the present invention, but the following embodiments are only preferred embodiments of the present invention, not all. Based on the embodiments in the implementation mode, other embodiments obtained by those skilled in the art without creative work are all within the protection scope of the present invention.

[0094] In the following description, terms such as "inside", "outside", "up", "down", "left", "right", etc. that indicate directions or positional relationships are only used to facilitate the description of the embodiments and simplify the description, and do not indicate or imply that the referred device or element must have a specific direction, be constructed and operate in a specific direction. Therefore, it should not be understood as a limitation of the present invention.

[0095] The present application embodiment provides a method for optimizing the long-term average transmission rate of a relay-assisted Internet of Things. Figure 1 , including the following steps:

[0096] Step S1, place I relay nodes i within the communication range of the base station, represented by the set i = {1, 2, ..., I}, the Internet of Things device sends data to the base station through the relay node, and the device and the relay node as well as the relay node and the base station communicate using a non-orthogonal multiple access transmission method, and the total number of time slots K for transmitting data by the relay node i in the relay-assisted Internet of Things is divided into k time slots, and the wireless channel parameters of the device and the relay node i in each time slot are obtained.

[0097] Among them, Non-Orthogonal Multiple Access (NOMA) is a transmission method for wireless communication, which allows multiple users to communicate simultaneously on the same frequency band, thereby improving the efficiency of spectrum utilization. Specifically, non-orthogonal multiple access technology superimposes the data of different users on the same signal for transmission, and then separates the data of different users through channel decoding technology. This technology can make full use of spectrum resources, improve spectrum utilization efficiency, support large-scale connections, and ensure transmission efficiency while connecting multiple users. Non-orthogonal multiple access technology is widely used and can be used in 5G and 6G mobile communications, the Internet of Things, smart homes and other fields. The total number of time slot transmissions refers to the maximum number of time slots that can be transmitted in a specified time period in a wireless communication system. A time slot refers to the basic unit in which a communication system communicates at fixed time intervals over a period of time.

[0098] Since non-orthogonal multiple access transmission is adopted, when transmitting data uplink, the transmission rates of the device and relay node i are respectively expressed as:

[0099]

[0100]

[0101] Among them, i represents the number of relay nodes, k represents the number of time slots, R k represents the transmission rate of the device in the kth time slot, B1 represents the channel bandwidth from the device to the relay node, and p k represents the transmission power of the device in the kth time slot, g krepresents the channel gain from the device to the relay node in the kth time slot, N1 represents the background noise power from the industrial IoT device to the relay node, and R i,k represents the transmission rate of relay node i in the kth time slot, B2 represents the channel bandwidth from the relay node to the base station, and p i,k represents the transmission power of relay node i in the kth time slot, g i,k represents the channel gain from relay node i to the base station in the kth time slot, p j,k represents the transmission power of relay node j in the kth time slot, g j,k represents the channel gain from relay node j to relay node in the kth time slot, and N2 represents the background noise power from relay node to base station.

[0102] Step S2, taking the data balance and the average transmission power balance at the relay node as constraints, the relay selection indicator variable, the transmission power of the device, the transmission power of the relay node and the sub-slot fraction as optimization variables, and maximizing the long-term average transmission rate of the relay-assisted Internet of Things as the optimization goal to establish an optimization problem P.

[0103] In order to allocate time slots more flexibly, a time slot may be divided into a number of sub-time slot fractions, that is, a sub-time slot fraction is a fractional part of a time slot.

[0104] The optimization problem P is expressed as:

[0105]

[0106] The above formula is the objective function of the optimization problem P, which represents maximizing the long-term average transmission rate of relay node i in the relay-assisted Internet of Things, where K represents the total number of time slots for relay node i in the relay-assisted Internet of Things to transmit data, k represents the number of time slots, and τ k represents the sub-slot fraction within the kth time slot, I represents the total number of relay nodes, and D i,k represents the amount of data received by relay node i in the kth time slot, E[D i,k ] indicates D i,k Seek expectations;

[0107] The optimization variables include: i,k},,{p k},,{p i,k}、{τ k},

[0108] Among them, m i,k Indicates whether the relay node i is selected to receive information in the kth time slot, that is, the relay selection indicator variable, p k represents the transmission power of the device in the kth time slot, p i,k represents the transmission power of relay node i in the kth time slot;

[0109]

[0110] The above formula is the constraint condition 1 of the optimization problem P, which means that the relay node i is selected to receive information in the kth time slot. i,k =1 indicates that it is selected, otherwise it indicates that it is not selected. Since this embodiment assumes that only one device needs to transmit data, the sum of the selection indicator variables of all relay nodes in each time slot is equal to 1;

[0111]

[0112] The above formula is the constraint condition 2 of the optimization problem P, which means that the amount of data received at the relay node should not be greater than the amount of data transmitted, that is, the data balance at the relay node, where the amount of data D received by the relay node i in the kth time slot is i,k =m i,k τ k TR k , where T represents the length of a single time slot, R k represents the transmission rate of the device in the kth time slot, D i ′ ,k represents the amount of data transmitted by relay node i in the kth time slot, D i ′ ,k =R i,k (1-τ k )T, where R i,k represents the transmission rate of relay node i in the kth time slot;

[0113]

[0114]

[0115] The above formula is the constraint 3 and constraint 4 of the optimization problem P, which means that the transmission power from the device to the relay node i should be equal to the average transmission power of the device, and the transmission power from the relay node i to the base station should be equal to the average transmission power of the relay node i, that is, the average transmission power is balanced, where, represents the average transmission power of the device in the kth time slot, represents the average transmission power of relay node i in the kth time slot;

[0116] 0≤p k ≤p max ,

[0117]

[0118] 0≤m i,k ≤1

[0119] 0≤τ k ≤1

[0120] The above formula is the constraint condition 5-8 of the optimization problem P, which represents the value range of the optimization variable, where p max Indicates the maximum transmission power of the device. represents the maximum transmission power of relay node i in the kth time slot, m i,k The value range of m is continuous. In this embodiment, if m i,k >0.5, then let m i,k =1, that is, relay node i is considered to be selected for information reception.

[0121] Step S3, calculate the data storage capacity and residual energy of the relay node, and transform the optimization problem P into the optimization problem P-1 according to the data storage capacity and residual energy of the relay node. The optimization problem P-1 takes maximizing the average transmission rate within the fixed time slot of the relay-assisted Internet of Things as the optimization goal.

[0122] Since the optimization problem P contains both nonlinear constraints and multiple continuous variables, it is a complex nonlinear problem and it is very difficult to solve it directly. Therefore, the data storage capacity and residual energy of the relay node are introduced to transform the optimization problem P into the optimization problem P-1.

[0123] In step S3, the method for calculating the data storage capacity of the relay node includes:

[0124]

[0125] Where K represents the total number of time slots in which relay node i transmits data in the relay-assisted IoT, k represents the number of time slots, I represents the total number of relay nodes, and C i,k+1 represents the data storage capacity of relay node i in the k+1th time slot, C i,k represents the amount of data stored in relay node i in the kth time slot, R i,k (1-τ k )T represents the amount of data transmitted by relay node i in the kth time slot, where R i,k represents the transmission rate of relay node i in the kth time slot, τ k represents the sub-slot fraction within the kth time slot, T represents the length of a single time slot, and m i,k τ k TR k represents the amount of data received by relay node i in the kth time slot, where m i,k Indicates whether the relay node i is selected to receive information in the kth time slot, that is, the relay selection indicator variable, R k Indicates the transmission rate of the device in the kth time slot.

[0126] In step S3, the method for calculating the residual energy of the relay node includes:

[0127]

[0128] Among them, E i,k+1 represents the remaining energy of relay node i in the k+1th time slot, E i,k represents the energy storage capacity of relay node i in the kth time slot, p i,k (1-τ k )T represents the energy consumed by relay node i in the kth time slot, where p i,k represents the transmission power of relay node i in the kth time slot, E new Represents the energy obtained by the relay node from the outside world.

[0129] Furthermore, the optimization problem P is transformed into the optimization problem P-1 according to the data storage capacity and the residual energy of the relay node. The expression of the transformed optimization problem P-1 is:

[0130]

[0131] The above formula is the objective function of the optimization problem P-1, which represents maximizing the average transmission rate of relay node i in the relay-assisted Internet of Things within a fixed time slot, where K represents the total number of time slots for relay node i in the relay-assisted Internet of Things to transmit data, k represents the number of time slots, and τ k represents the sub-slot fraction within the kth time slot, I represents the total number of relay nodes, and D i,k represents the amount of data received by relay node i in the kth time slot;

[0132] The optimization variables include: i,k},,{p k},,{p i,k}、{τ k},

[0133] Among them, m i,k Indicates whether the relay node i is selected to receive information in the kth time slot, that is, the relay selection indicator variable, p k represents the transmission power of the device in the kth time slot, p i,k represents the transmission power of relay node i in the kth time slot;

[0134]

[0135] The above formula is the constraint 1 of the optimization problem P-1, which means that the relay node i is selected to receive information in the kth time slot. i,k=1 indicates that it is selected, otherwise it indicates that it is not selected. Since this embodiment assumes that only one device needs to transmit data, the sum of the selection indicator variables of all relay nodes in each time slot is equal to 1;

[0136] C i,k -R i,k (1-τ k )T+m i,k τ k TR k >0

[0137] The above formula is the constraint 2 of the optimization problem P-1, which means that the data storage capacity of the relay node should be greater than zero;

[0138] 0≤p i,k (1-τ k )T≤E max

[0139] The above formula is the constraint 3 of the optimization problem P-1, which means that the energy consumed by relay node i cannot exceed the maximum energy that the relay node can store, where E max Indicates the maximum energy that the relay node can store;

[0140] E i,k -p i,k (1-τ k )T+E new >0

[0141] The above formula is constraint 4 of the optimization problem P-1, which means that the residual energy of the relay node should be greater than zero;

[0142] 0≤p k ≤p max ,

[0143]

[0144] 0≤m i,k ≤1

[0145] 0≤τ k ≤1

[0146] The above formula is the constraint conditions 5-8 of the optimization problem P-1, which represents the value range of the optimization variable, where p max Indicates the maximum transmission power of the device. represents the maximum transmission power of relay node i in the kth time slot, if m i,k >0.5, then let m i,k =1, that is, relay node i is considered to be selected for information reception.

[0147] Step S4, using a double-delayed deep deterministic policy gradient algorithm to solve the optimization problem P-1, thereby obtaining the optimal solution to the optimization problem P.

[0148] The double-delayed deep deterministic policy gradient algorithm is a deep reinforcement learning algorithm for solving continuous control problems. It is an improved version of the Deep Deterministic Policy Gradient (DDPG) algorithm. This embodiment improves the performance and stability of the algorithm by introducing two delay networks and some optimization techniques.

[0149] On the other hand, in this embodiment, in step S4, the method of using the double-delayed deep deterministic policy gradient algorithm to solve the optimization problem P-1 includes:

[0150] Step S401, define the state space s at the kth time slot k =(C i,k ,E i,k ), the action space a at the kth time slot k =(m i,k ,p k ,p i,k ,τ k ) and the reward function r;

[0151] Step S402, using the actor-critic algorithm framework to build a neural network model, the neural network model includes an actor network, a first critic network and a second critic network, the actor network includes an actor online network μ(.|θ μ ) and the actor-target network μ ′ (.|θ μ′ ), the input of the actor network is the state space s k , the output is the action space a k , the First Critic Network includes the First Critic Online Network and the first critic target network Second Critics Network includes Second Critics Online Network and the second critic target network The first critic network and the second critic network are used to estimate the value of the current state space and action space.

[0152] Step S403, initializing the parameters of the neural network model.

[0153] Specifically, let the actor online network parameters θ μ Equal to the actor-target online network parameter θ μ′ , First Critics Online Network Parameters Equal to the first critic target network parameter Second critic online network parameters Equal to the second critic target network parameter

[0154] Step S404, collect status report pair (s k ,a k ,r k ,s k+1 ) into the experience pool, where r k represents the reward value returned in the kth time slot, i.e., the solution to the objective function of the optimization problem P-1, s k+1 represents the state space at the k+1th time slot (C i,k+1 ,E i,k+1 ).

[0155] On the other hand, in this embodiment, the collection status report pair (s k ,a k ,r k ,s k+1 ) into the experience pool are as follows:

[0156] Input state space s k To the actor network, output action space a k ;

[0157] The action space a k Applied to the environment, the environment returns the state space s for the next time slot k+1 and the reward value r k ;

[0158] Will s k 、a k 、r k and k+1 It is recorded as a state-return pair and put into the experience pool.

[0159] Step S405: extract state report pairs (s k ,a k ,r k ,s k+1 ) Train the neural network model and repeatedly update the actor network and the critic network until the optimal neural network model is obtained.

[0160] In this embodiment, the sub-steps of step S405 include:

[0161] Step A01, extract state reward pairs (s k ,a k ,r k ,s k+1) trains the neural network model and calculates the target value using the first critic target network and the second critic target network.

[0162] Step A02: Based on the evaluation value output by the first critic online network and the evaluation value output by the second critic online network The error between the target value and L i , update the parameters in the first critic online network and the second critic online network, and put the state reward pair output by the neural network model into the experience pool.

[0163] Step A03, repeating steps A01 and A02 until the experience pool reaches the maximum storage capacity N, then stopping updating the first critic online network and the second critic online network. This cycle is to increase the amount of data in the experience pool, so the preset maximum storage capacity N is used as the limiting condition for stopping updating.

[0164] Step A04, calculating the error J between the first critic online network and the second critic online network i , according to the error J i Update the parameters of the actor online network. After updating the critic online network multiple times, the actor online network is updated again, thus ensuring that the training of the actor online network is more stable.

[0165] Step A05: Repeat steps A01 to A04 until the evaluation values ​​output by the online network of the two critics are equal to the target value or the error L i Within the preset error range, the network converges, and the neural network model at this time is taken as the optimal neural network model.

[0166] Among them, the error L i Within the preset error range means that the evaluation values ​​output by the online network of the two critics are close to the target value. For example, the preset error range is 0.001.

[0167] Step S406: Input the current state space into the trained actor network, output the optimal action space and the returned reward value r k , that is, the optimal solution of the objective function of the optimization problem P-1, thereby maximizing the long-term time-averaged transmission rate of relay node i in the relay-assisted Internet of Things.

[0168] The overall technical concept of step S4 in this embodiment is: the action space a output by the actor network k will be input into the environment, and the environment will return a reward value r k and the state space s of the next time slot k+1 This reward value r kWill be used to update the critic network. The action space a output by the actor network k It is also input into the critic network. The role of the critic network is to evaluate the actions of the actor network, that is, to calculate the state value function after taking the action. In the actor-critic algorithm framework, the actor network and the critic network are jointly trained through interaction. The action space a output by the actor network k will be input into the environment, and the environment will return a reward value r k and the state space s of the next time slot k+1 This reward value r k It will be used to update the critic network, and the output of the critic network will be used to update the actor network. Through this interaction and update, the actor network and the critic network gradually learn an optimal policy function and state value function, thereby achieving efficient reinforcement learning.

[0169] On the other hand, in this embodiment, the sub-steps of step A01 include:

[0170] Return the status to (s k ,a k ,r k ,s k+1 ) is input into the actor-target network and outputs the target action space a in the next state space ′ k ;

[0171] For the target action space a ′ k After adding the perturbation, they are input into the first critic target network and the second critic target network respectively;

[0172] Select the smaller value of the output value of the first critic target network and the output value of the second critic target network;

[0173] The selected smaller value is discounted and weighted, and the reward value is added, and the calculated result is used as the target value.

[0174] By calculating the target value, in the target action space a of the next state ′ k By adding disturbances to the target network, the subsequent critic network value assessment is more accurate. And by selecting the smaller value of the output value of the first critic target network and the output value of the second critic target network when calculating the target value, the network overestimation problem is suppressed.

[0175] For example, the specific implementation method of calculating the target value using the first critic target network and the second critic target network is:

[0176]

[0177] Among them, y(s k ,a k ) represents the target value, r k (s k ,a k ) represents the reward value, that is, the solution of the objective function of the optimization problem P-1, γ represents the discount factor, which is used to attenuate the impact of future rewards, and the input of the first critic target network is the target action space a ′ k With perturbation, the output of the first critic target network is Similarly, is the output value of the second critic target network.

[0178] For a schematic diagram of the principle of the dual-delay deep deterministic policy gradient algorithm in this embodiment, please refer to the attached Figure 2 .

[0179] On the other hand, in this embodiment, in step A02, the error L i The expression is:

[0180]

[0181] When i=1, L1 represents the evaluation value output by the first critic online network. The error between the target value and represents the evaluation value of the first critic's online network output. When i=2, L2 represents the evaluation value of the second critic's online network output. The error between the target value and represents the evaluation value of the second critic's online network output, y(s k ,a k ) represents the target value;

[0182] According to the error L i The method for updating the parameters in the first critic online network and the second critic online network is:

[0183]

[0184] Among them, when i=1, represents the updated parameters of the first critic online network, represents the parameters of the first critic online network before updating, ρ represents the preset learning rate of the critic online network, represents the gradient descent value of the first critic's online network parameters. Specifically, the gradient descent value of the first critic's online network parameters The calculation method is based on the network parameters The direction of the loss function is to find the effect of the network parameters Find the partial derivative, where the loss function usually uses mean square error (MSE) or cross entropy. When i = 2, represents the updated parameters of the second critic online network, represents the parameters of the second critic online network before updating, Represents the gradient descent value of the second critic online network parameter. The gradient descent value of the second critic online network parameter The calculation method is the same as above The calculation method of is the same as that of , and will not be repeated in this embodiment.

[0185] On the other hand, in this embodiment, in step A04, the error J between the first critic online network and the second critic online network is i The calculation method is:

[0186]

[0187] Where N represents the maximum storage capacity of the experience pool. When i=1, J1 represents the error of the first critic's online network. represents the evaluation value of the first critic online network output, when i=2, J2 represents the error of the second critic online network, represents the evaluation value of the second critic's online network output;

[0188] According to the error J i The method to update the parameters of the actor online network is:

[0189]

[0190] in, represents the updated parameters of the actor online network, Indicates the parameters of the actor before the online network update, represents the preset learning rate of the actor online network, Represents the gradient descent value of the actor's online network parameters.

[0191] On the other hand, in this embodiment, a soft update method is used, that is, an exponential moving average (EMA) method is used to update the parameters of the actor target network, the parameters of the first critic target network, and the parameters of the second critic target network. The specific update method is:

[0192] θ μ′ =βθ μ +(1-β)θ μ′ (7)

[0193]

[0194]

[0195] Among them, β represents the preset smoothing factor, which is generally a smaller value such as 0.001 or 0.005.

[0196] In summary, the technical concept of this embodiment is as follows: First, in the relay-assisted Internet of Things, based on the data balance at the relay node and the constraint premise of the average transmission power balance, an optimization problem P of the long-term average transmission rate of the relay-assisted Internet of Things that performs data transmission through non-orthogonal multiple access technology is constructed. Since this problem is a complex nonlinear problem, it is very difficult to solve it directly. For this reason, the data storage capacity and residual energy of the relay node are introduced to transform the optimization problem P into the optimization problem P-1. Combined with the analysis of the optimization problem P-1, a double-delay deep deterministic policy gradient algorithm for solving the optimization problem P-1 is proposed, and then the solution of the original optimization problem P is obtained.

[0197] By using the long-term average transmission rate optimization method of the relay-assisted Internet of Things proposed in this embodiment, a relay-assisted Internet of Things with higher transmission performance is designed. When the number of relay nodes I is 3, 5, and 7 respectively, a simulation experiment is performed to compare the relationship between the average transmission rate and the time slot of the relay-assisted Internet of Things using FDMA in the prior art and the relationship between the average transmission rate and the time slot of the relay-assisted Internet of Things with better transmission performance designed in this embodiment. The simulation experiment results are shown in the attached figure. Figure 3(a) , 3(b) As shown in Figure 3(c), "-*-" represents the relationship between the time-average transmission rate and the time slot under the relay-assisted Internet of Things using FDMA in the prior art, and "-○-" represents the relationship between the time-average transmission rate and the time slot under the relay-assisted Internet of Things with better transmission performance proposed in this embodiment. It can be seen from the schematic diagram of the simulation experiment results that the time-average transmission rate of the relay-assisted Internet of Things designed in this embodiment is significantly increased compared to the time-average transmission rate of the relay-assisted Internet of Things using FDMA in the prior art. The simulation results verify that the method for optimizing the long-term time-average transmission rate of the relay-assisted Internet of Things proposed in this embodiment has produced substantial and beneficial technical effects in increasing the long-term time-average transmission rate of the relay-assisted Internet of Things, and improves the data transmission efficiency of the relay-assisted Internet of Things.

[0198] The technical solution proposed in this embodiment constructs an optimization problem P of the long-term time-average transmission rate of the relay-assisted Internet of Things that performs data transmission through non-orthogonal multiple access technology, analyzes the characteristics of the problem, introduces the data storage capacity and residual energy of the relay node, transforms the optimization problem P into an optimization problem P-1, and proposes a double-delay deep deterministic policy gradient algorithm to solve the optimization problem P-1, thereby obtaining the optimal solution to the original optimization problem P, and maximizing the long-term time-average transmission rate of the relay-assisted Internet of Things under the constraint of ensuring data balance at the relay node and average transmission power balance, thereby improving the data transmission efficiency of the relay-assisted Internet of Things and achieving a better quality of wireless network experience.

[0199] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Those skilled in the art should understand that the present invention includes but is not limited to the contents described in the drawings and the above specific embodiments. Any modification that does not deviate from the functional and structural principles of the present invention will be included in the scope of the claims.

Claims

1. A method for optimizing the long-term average transmission rate of a relay-assisted Internet of Things, characterized in that: The following steps are involved: Step S1, placing I relay nodes i within the communication range of the base station, represented by the set i = {1, 2, ..., I}, the IoT device sends data to the base station through the relay node, and the device and the relay node as well as the relay node and the base station communicate using a non-orthogonal multiple access transmission mode, dividing the total number of time slots K for the data transmission of the relay node i in the relay-assisted IoT into k time slots, and obtaining the wireless channel parameters of the device and the relay node i in each time slot; Step S2, taking the data balance and average transmission power balance at the relay node as constraints, taking the relay selection indicator variable, the transmission power of the device, the transmission power of the relay node and the sub-time slot fraction as optimization variables, and taking maximizing the long-term average transmission rate of the relay-assisted Internet of Things as the optimization goal to establish an optimization problem P, wherein the expression of the optimization problem P is: The above formula is the objective function of the optimization problem P, which represents maximizing the long-term average transmission rate of relay node i in the relay-assisted Internet of Things, where K represents the total number of time slots for relay node i in the relay-assisted Internet of Things to transmit data, k represents the number of time slots, and τ k represents the sub-slot fraction within the kth time slot, I represents the total number of relay nodes, and D i,k represents the amount of data received by relay node i in the kth time slot, E[D i,k ] indicates D i,k Seek expectations; The optimization variables include: i,k }、{p k }、{p i,k }、{τ k }, Among them, m i,k Indicates whether the relay node i is selected to receive information in the kth time slot, that is, the relay selection indicator variable, p k represents the transmission power of the device in the kth time slot, p i,k represents the transmission power of relay node i in the kth time slot; The above formula is the constraint condition 1 of the optimization problem P, which means that the relay node i is selected to receive information in the kth time slot; The above formula is the constraint condition 2 of the optimization problem P, which means that the amount of data received at the relay node should not be greater than the amount of data transmitted, that is, the data balance at the relay node, where the amount of data D received by the relay node i in the kth time slot is i,k =m i,k τ k TR k , where T represents the length of a single time slot, R k represents the transmission rate of the device in the kth time slot, D i ′ ,k represents the amount of data transmitted by relay node i in the kth time slot, D i ′ ,k =R i,k (1-τ k )T, where R i,k represents the transmission rate of relay node i in the kth time slot; The above formula is the constraint 3 and constraint 4 of the optimization problem P, which means that the transmission power from the device to the relay node i should be equal to the average transmission power of the device, and the transmission power from the relay node i to the base station should be equal to the average transmission power of the relay node i, that is, the average transmission power is balanced, where, represents the average transmission power of the device in the kth time slot, represents the average transmission power of relay node i in the kth time slot; 0≤p k ≤p max , 0≤m i,k ≤1 0≤τ k ≤1 The above formula is the constraint condition 5-8 of the optimization problem P, which represents the value range of the optimization variable, where p max Indicates the maximum transmission power of the device. represents the maximum transmission power of relay node i in the kth time slot, if m i,k >0.5, then let m i,k =1, that is, relay node i is considered to be selected for information reception; Step S3, calculate the data storage capacity and residual energy of the relay node, and transform the optimization problem P into the optimization problem P-1 according to the data storage capacity and residual energy of the relay node, wherein the optimization problem P-1 takes maximizing the average transmission rate within the fixed time slot of the relay-assisted Internet of Things as the optimization goal, wherein the expression of the transformed optimization problem P-1 is: The above formula is the objective function of the optimization problem P-1, which represents maximizing the average transmission rate of relay node i in the relay-assisted Internet of Things within a fixed time slot, where K represents the total number of time slots for relay node i in the relay-assisted Internet of Things to transmit data, k represents the number of time slots, and τ k represents the sub-slot fraction within the kth time slot, I represents the total number of relay nodes, and D i,k represents the amount of data received by relay node i in the kth time slot; The optimization variables include: i,k }、{p k }、{p i,k }、{τ k }, Among them, m i,k Indicates whether the relay node i is selected to receive information in the kth time slot, that is, the relay selection indicator variable, p k represents the transmission power of the device in the kth time slot, p i,k represents the transmission power of relay node i in the kth time slot; The above formula is the constraint 1 of the optimization problem P-1, which means that the relay node i is selected to receive information in the kth time slot; C i,k -R i,k (1-τ k )T+m i,k τ k TR k >0 The above formula is the constraint 2 of the optimization problem P-1, which means that the data storage capacity of the relay node should be greater than zero; 0≤p i,k (1-t k )T≤E max The above formula is the constraint 3 of the optimization problem P-1, which means that the energy consumed by relay node i cannot exceed the maximum energy that the relay node can store, where E max represents the maximum energy that the relay node can store; AND i,k -p i,k (1−τ k )T+E new >0 The above formula is constraint 4 of the optimization problem P-1, which means that the residual energy of the relay node should be greater than zero; 0≤m i,k ≤1 0≤τ k ≤1 The above formula is the constraint conditions 5-8 of the optimization problem P-1, which represents the value range of the optimization variable, where p max Indicates the maximum transmission power of the device. represents the maximum transmission power of relay node i in the kth time slot, if m i,k >0.5, then let m i,k =1, that is, relay node i is considered to be selected for information reception; Step S4, using a double-delayed deep deterministic policy gradient algorithm to solve the optimization problem P-1, thereby obtaining the optimal solution to the optimization problem P.

2. A method for optimizing the long-term average transmission rate of a relay-assisted Internet of Things according to claim 1, characterized in that: In step S3, the method for calculating the data storage capacity of the relay node includes: Where K represents the total number of time slots in which relay node i transmits data in the relay-assisted IoT, k represents the number of time slots, I represents the total number of relay nodes, and C i,k+1 represents the data storage capacity of relay node i in the k+1th time slot, C i,k represents the amount of data stored in relay node i in the kth time slot, R i,k (1-τ k )T represents the amount of data transmitted by relay node i in the kth time slot, where R i,k represents the transmission rate of relay node i in the kth time slot, τ k represents the sub-slot fraction within the kth time slot, T represents the length of a single time slot, and m i,k τ k TR k represents the amount of data received by relay node i in the kth time slot, where m i,k Indicates whether the relay node i is selected to receive information in the kth time slot, that is, the relay selection indicator variable, R k Indicates the transmission rate of the device in the kth time slot.

3. A method for optimizing the long-term average transmission rate of a relay-assisted Internet of Things according to claim 2, characterized in that: In step S3, the method for calculating the residual energy of the relay node includes: Among them, E i,k+1 represents the remaining energy of relay node i in the k+1th time slot, E i,k represents the energy storage capacity of relay node i in the kth time slot, p i,k (1-τ k )T represents the energy consumed by relay node i in the kth time slot, where p i,k represents the transmission power of relay node i in the kth time slot, E new Represents the energy obtained by the relay node from the outside world.

4. A method for optimizing the long-term average transmission rate of a relay-assisted Internet of Things according to claim 3, characterized in that: In step S4, the method of using the double-delayed deep deterministic policy gradient algorithm to solve the optimization problem P-1 includes: Step S401, define the state space s at the kth time slot k =(C i,k ,E i,k ), the action space a at the kth time slot k =(m i,k ,p k ,p i,k ,τ k ) and the reward function r; Step S402, using the actor-critic algorithm framework to build a neural network model, the neural network model includes an actor network, a first critic network and a second critic network, the input of the actor network is the state space s k , the output is the action space a k , the first critic network and the second critic network are used to estimate the value of the current state space and action space; Step S403, initializing the parameters of the neural network model; Step S404, collect status report pair (s k ,a k ,r k ,s k+1 ) into the experience pool, where r k represents the reward value returned in the kth time slot, i.e., the solution to the objective function of the optimization problem P-1, s k+1 represents the state space at the k+1th time slot (C i,k+1 ,E i,k+1 ); Step S405: extract state report pairs (s k ,a k ,r k ,s k+1 ) Train the neural network model, repeatedly update the actor network and the critic network until the optimal neural network model is obtained; Step S406: Input the current state space into the trained actor network, output the optimal action space and the returned reward value r k , which is the optimal solution to the objective function of the optimization problem P-1.

5. A method for optimizing the long-term average transmission rate of a relay-assisted Internet of Things according to claim 4, characterized in that: The actor network includes an actor online network and an actor target network, the first critic network includes a first critic online network and a first critic target network, and the second critic network includes a second critic online network and a second critic target network. Then, the sub-steps of step S405 include: Step A01, extract state reward pairs (s k ,a k ,r k ,s k+1 ) trains the neural network model and calculates the target value using the first critic target network and the second critic target network; Step A02: Based on the evaluation value output by the first critic online network and the evaluation value output by the second critic online network The error between the target value and L i , update the parameters of the first critic online network and the second critic online network, and put the state-return pair output by the neural network model into the experience pool; Step A03, repeatedly executing step A01 and step A02 until the experience pool reaches the maximum storage capacity N, then stopping updating the first critic online network and the second critic online network; Step A04, calculating the error J between the first critic online network and the second critic online network i , according to the error J i Update the parameters of the actor's online network; Step A05: Repeat steps A01 to A04 until the evaluation values ​​output by the online network of the two critics are equal to the target value or the error L i Within the preset error range, the neural network model at this time is taken as the optimal neural network model.

6. A method for optimizing the long-term average transmission rate of a relay-assisted Internet of Things according to claim 5, characterized in that: The sub-steps of step A01 include: Return the status to (s k ,a k ,r k ,s k+1 ) is input into the actor-target network and outputs the target action space a in the next state space ′ k ; For the target action space a ′ k After adding the perturbation, they are input into the first critic target network and the second critic target network respectively; Select the smaller value of the output value of the first critic target network and the output value of the second critic target network; The selected smaller value is discounted and weighted, and the reward value is added, and the calculated result is used as the target value.

7. A method for optimizing the long-term average transmission rate of a relay-assisted Internet of Things according to claim 6, characterized in that: In step A02, the error L i The expression is: When i=1, L1 represents the evaluation value output by the first critic online network. The error between the target value and represents the evaluation value of the first critic's online network output. When i=2, L2 represents the evaluation value of the second critic's online network output. The error between the target value and represents the evaluation value of the second critic's online network output, y(s k ,a k ) represents the target value; According to the error L i The method for updating the parameters in the first critic online network and the second critic online network is: Among them, when i=1, represents the updated parameters of the first critic online network, represents the parameters of the first critic online network before updating, ρ represents the preset learning rate of the critic online network, represents the gradient descent value of the first critic's online network parameters, when i = 2, represents the updated parameters of the second critic online network, represents the parameters of the second critic online network before updating, Represents the gradient descent value of the second critic online network parameters.

8. A method for optimizing the long-term average transmission rate of a relay-assisted Internet of Things according to claim 5, characterized in that: In step A04, the error J between the first critic online network and the second critic online network is i The calculation method is: Where N represents the maximum storage capacity of the experience pool. When i=1, J1 represents the error of the first critic's online network. represents the evaluation value of the first critic online network output, when i=2, J2 represents the error of the second critic online network, represents the evaluation value of the second critic's online network output; According to the error J i The method to update the parameters of the actor online network is: in, represents the updated parameters of the actor online network, Indicates the parameters of the actor before the online network update, represents the preset learning rate of the actor online network, Represents the gradient descent value of the actor's online network parameters.

Images