Combined uplink and downlink channel access and energy collection method for wireless energy-carrying communication

Abstract

The invention discloses a combined uplink and downlink channel access and energy collection method for wireless energy-carrying communication, and belongs to the technical field of wireless communication. A channel access method is divided into three layers, namely downlink channel access based on non-orthogonal multiple access, uplink channel access based on dynamic frame time slot Aloha starting continuous interference elimination and downlink wireless energy collection, multi-layer Q learning is adopted, downlink superposed signal power distribution is learned by a downlink layer, and wireless energy collection is realized. The uplink layer learns the time slot selection and transmission probability of the uplink device, and the stateless layer learns the time frame size and the power division ratio of the system. According to the method, the energy evolution of an infinite energy-carrying communication user, the change of variable channel gain information at the time of user energy collection and data transmission and the tradeoff of different uplink and downlink transmission proportions are fully considered, and experiments show that the method can obtain higher uplink and downlink communication and rate in a wireless energy-carrying communication network.

Description

Technical Field

[0001] The present invention relates to the technical field of wireless communications, and in particular to a combined uplink and downlink channel access and energy collection method for wireless energy-carrying communications. Background Art

[0002] Future IoT networks will consist of low-power devices that sense their environment and transmit data to a gateway. The gateway can then use the data from the device to instruct the device to perform a sensing task or control an actuator. Devices have given rise to wireless energy-carrying communication technology by simultaneously delivering RF energy using the existing spectrum used for data transmission. More and more IoT networks equipped with RF-powered devices are operating over a shared medium. Unlike traditional networks where devices have wireless energy, in RF-powered IoT networks, devices must first collect RF energy before they can transmit or / and receive data. In these scenarios, channel access is a key technology that facilitates uplink and downlink transmissions on the same channel. Since conflicts may occur when devices upload their data to the gateway, a key issue is to determine the uplink and downlink channel access parameters given an unknown number of competing devices.

[0003] In addition, the lifetime of the device and the amount of data collected are limited by the available energy. Therefore, in order to ensure long-term operation of the device, it is necessary to manage the available energy of the device that relies on hybrid access points for wireless energy harvesting. Wireless energy-carrying communication supports two modes: time switching and power splitting. When using a power splitter, the receiver divides the power of the received signal between its energy harvester and the data decoder. In addition, there is a trade-off between energy harvesting and information decoding, which affects the uplink and downlink rates respectively. Specifically, the less energy collected will result in lower uplink transmission power, which will undoubtedly hurt the uplink rate. Correspondingly, the less power allocated to information decoding will directly hurt the downlink rate. Therefore, a key issue in energy harvesting is to optimize the power split ratio of the receiver.

[0004] At present, the traditional wireless energy-carrying communication network channel access optimization mainly considers the pre-allocation of time slots / subcarriers and only optimizes one time frame. It adopts an ideal linear RF energy conversion model, does not consider the energy evolution of the device and the imperfect channel gain information, and lacks consideration for the actual situation. Summary of the invention

[0005] The purpose of the present invention is to provide a joint uplink and downlink channel access and energy collection method for wireless energy-carrying communication, which learns the optimal channel access method and energy collection method under different states in each stage through multi-layer machine learning to obtain the best uplink and downlink communication and rate.

[0006] In order to achieve the above object, the technical solution adopted by the present invention is:

[0007] A method for joint uplink and downlink channel access and energy collection for wireless energy-carrying communication, comprising the following steps:

[0008] Step S1, constructing a time-varying channel gain model to obtain a channel gain between a user and a hybrid access point in each time frame;

[0009] Step S2, constructing an energy evolution model, collecting energy through a radio frequency energy collector equipped by each user, and obtaining the energy level of the user according to the energy evolution model;

[0010] Step S3, constructing a wireless energy-carrying communication downlink Markov decision model, where the downlink state is constructed as the channel state of all devices, the downlink action is constructed as the power allocation of the superimposed signal, and the reward is constructed as the downlink throughput;

[0011] Step S4, constructing a wireless energy-carrying communication uplink Markov decision model, where the uplink state is constructed as the channel state of a single device and its existing power, the uplink action is constructed as the transmission probability and transmission time slot selection of a single device, and the reward is constructed as the uplink throughput;

[0012] Step S5: Centralized Q learning is adopted in the downlink layer, and the hybrid access point is used as an agent. The hybrid access point selects the action corresponding to the maximum Q value in the current state according to the channel state of all devices in the current time frame, determines the power allocation of the signal, and then calculates the sum rate of the downlink of the current time frame as a reward based on the transmission result.

[0013] Step S6: Distributed Q learning is adopted in the uplink layer. Each device acts as an agent. Each device selects the action corresponding to the maximum Q value in the current state according to its own channel state and real-time power in the current time frame, determines its transmission probability and time slot selection, and then calculates the sum rate of the uplink in the current time frame as a reward based on the transmission result.

[0014] Step S7, using stateless Q learning to optimize the energy collection of step S2, the system acts as an agent, and in each cycle the system selects the action with the highest probability, controls the uplink frame size and the downlink power split ratio, and then the system collects the rewards of the uplink and downlink of the current cycle as the stateless reward R κ , The system updates the Q value and probability mass function according to the reward value until the system converges, thereby optimizing energy harvesting.

[0015] Furthermore, in step S1, time is divided into frames and indexed by t, and the channel gain is obtained according to the following formula:

[0016] In the formula, represents the channel gain between user i and the hybrid access point at time frame t, d i represents the distance between user i and the hybrid access point, n represents the path loss exponent, and λ represents an exponential random variable with a mean of 1.

[0017] Furthermore, in step S2, the energy level evolves according to the following formula:

[0018] In the formula, B max represents the battery capacity of each user, represents the energy collected by user i in time frame t, represents the energy consumed by user i in uplink transmission in time frame t, where

[0019] In the formula, η represents the energy conversion efficiency, θ represents the power division ratio, represents the received power at the user, τ d Indicates the length of the downlink period in each frame.

[0020] Please note that users will only transmit if their energy is sufficient to transmit at least one package, and each transmission will use up all current energy. Assuming the smallest package is 21 bytes, and transmitting 1 bit requires 18 nJ of energy, then transmitting one package requires at least 21×8×18=3024 nJ.

[0021] Furthermore, in step S3, the downlink Markov decision model specifically includes: the downlink state is constructed as the channel state of all devices, that is, All possible states constitute the downlink state set, namely The downlink action is constructed as the power allocation of the superimposed signal, i.e. All possible actions constitute the downlink action set, namely Rewards are structured as downlink throughput.

[0022] Furthermore, in step S4, the uplink Markov decision model specifically includes: the uplink state is constructed as the channel state of a single device and its current power All possible states of each device constitute the uplink state set of the device Uplink actions are constructed as single device transmission probability and transmission time slot selection All possible actions of each device constitute the uplink action set of the device Rewards are structured as uplink throughput.

[0023] Furthermore, in step S5, during the system warm-up phase, the hybrid access point randomly selects a downlink action according to its state, and updates the state-action corresponding Q value according to the following Bellman equation: Q(s t ,a t )=(1-α)Q(s t ,a t )+α(r(s t ,a t )+γmaxQ(s t+1 ,a t+1 ))

[0024] In the formula, Q(s t ,a t ) is state s t Take action a t The corresponding Q value, r(s t ,a t ) is state s t Take action a t The reward obtained, maxQ(s t+1 ,a t+1 ) is the maximum Q value corresponding to the next time slot state predicted by the system, α is the learning parameter, γ is the discount factor, γ∈[0,1].

[0025] Furthermore, in step S5, the power allocation should satisfy the SIC of the device for successful decoding and Where P is the total transmit power of the hybrid access point, is the transmission power allocated to user i in time frame t,

[0026] The sum rate of the downlink is obtained according to the following formula:

[0027] Where W is the bandwidth, θ is the power split ratio, i is the current user, N is the total number of users, is the channel gain of user i in time frame t, is the signal transmission power allocated to user i in time frame t, and user j is the user other than user i, is the transmission power of the signal allocated to user j in time frame t, and n0 is the noise power.

[0028] Furthermore, in step S6, during the system warm-up phase, each device first randomly selects an upward action according to its state, and updates the corresponding Q value according to the Bellman equation.

[0029] Further, in step S6, the sum rate of the uplink is obtained according to the following formula:

[0030] Where W is the bandwidth, i is the current user, and N is the total number of users. is the channel gain of user i in time frame t, is the signal transmission power of user i in time frame t, user j is the user other than user i, is the channel gain of user j in time frame t, is the signal transmission power of user j in frame t, and n0 is the noise power.

[0031] Furthermore, in step S7, the system updates the Q value according to the reward value using the following formula: Q(a)←Q(a)+λ(r(a)-Q(a))

[0032] In the formula, Q(a) represents the system Q value corresponding to action a, λ represents the learning rate, λ∈[0,1], and r(a) represents the stateless reward function value corresponding to action a;

[0033] The probability mass function is:

[0034] In the formula, a i represents the current action, a represents all possible actions, Pr(a i ) indicates selecting action a i The probability of Q(a i ) indicates action a i The corresponding system Q value, T represents the continuous time frame.

[0035] By adopting the above technical solution, the present invention can achieve the following beneficial effects:

[0036] (1) The present invention divides the channel access method into three levels: downlink channel access based on non-orthogonal multiple access (NOMA), uplink channel access based on dynamic frame time slot Aloha with successive interference cancellation (SIC), and downlink wireless energy collection. Then, the channel access and energy collection parameters in each scenario are determined through multi-layer learning, which fully considers the energy evolution of wireless energy-carrying communication users, the change of user energy collection and data transmission with time-varying channel gain information, and the trade-off of different uplink and downlink link transmission ratios. The uplink user and downlink hybrid access point can learn the optimal channel access method under different states in each stage through multi-layer machine learning. When the channel model is unknown, the channel state is accurately judged through a continuous learning process, so as to select the optimal channel access method to obtain the best uplink and downlink communication and rate.

[0037] (2) The present invention allows hybrid access points and devices to learn the best transmission strategy through a multi-layer Q learning (Multi-Q) solution. The downlink hybrid access point uses downlink NOMA to allocate different powers to the data of all users and superimpose them into a composite signal for transmission to all users. Each user collects energy from the received signal according to power segmentation and uses the SIC decoder to decode the information. Uplink users use frame time slot Aloha for uplink transmission. The hybrid access point does not allocate fixed time slots to the device and is equipped with a SIC decoder for information decoding. For each system state, the hybrid access point learns to optimize its downlink transmission power allocation, the device learns the time slot selection and transmission probability of each frame in the uplink, and the system learns the frame size and the optimal power segmentation ratio corresponding to different frame sizes. Multi-Q does not assume non-causal information and state transition probability, but actually considers nonlinear energy conversion models, energy evolution of devices, and imperfect causal channel gains.

[0038] (3) Simulation results show that Multi-Q achieves an average sum rate of 44b / s / Hz, which is 6 times that of Aloha, 2.3 times that of TDMA, and 30% more than polling. This is because Multi-Q, a learning-based method, can flexibly schedule the system to respond to different network conditions. Therefore, compared with Aloha, TDMA, etc., Multi-Q is able to obtain the best transmission strategy. The present invention can be applied to a variety of IoT deployments, such as smart agriculture, smart transportation, smart cities, etc., to better provide charging solutions and communications for agricultural sensors, parking sensors, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1is a working flow chart of the wireless energy-carrying communication system in an embodiment of the present invention;

[0040] Figure 2 It is a time frame structure diagram in the present invention;

[0041] Figure 3 It is a flow chart of the multi-layer learning algorithm in the present invention;

[0042] Figure 4 is a flowchart of centralized Q learning at the downlink layer in the present invention;

[0043] Figure 5 is a flow chart of distributed Q learning at the uplink layer in the present invention;

[0044] Figure 6 It is a stateless Q learning flow chart in the present invention;

[0045] Figure 7 is the rate convergence curve of the uplink and downlink in the present invention;

[0046] Figure 8 is the sum rate convergence curve of different learning parameters in the present invention;

[0047] Fig. 9 The influence of different HAP transmission powers on the average sum rate per frame and the average uplink rate per frame and the downlink rate per frame in the present invention;

[0048] Fig.10 The influence of different user positions on the average sum rate per frame and the average uplink rate per frame and the downlink rate per frame in the present invention;

[0049] Fig.11 This is the influence of different power division ratios on the average sum rate per frame, the average uplink rate per frame, and the downlink rate per frame in the present invention. DETAILED DESCRIPTION

[0050] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0051] like Figure 1 As shown, the wireless energy-carrying communication network in this embodiment includes a hybrid access point (HAP) providing services for N energy-harvesting users / devices. Each device is represented by i, where i∈[1,2,…,N]. HAP uses NOMA in the downlink, and the device uses frame time slot Aloha for uplink transmission. HAP is equipped with a SIC decoder for uplink information decoding. Time is divided into frames and indexed by t, refer to Figure 2At the beginning of each time frame t, HAP sends pilot symbols for channel estimation. After that, each frame is divided into a downlink period and an uplink period, each with a length of τ d and τ u In the downlink period, the device uses power splitting to divide the received power into two parts, namely energy collection and information decoding. After that, there are M time slots in the uplink, and the devices select a time slot from the M time slots to transmit their data packets.

[0052] The present invention provides a combined uplink and downlink channel access and energy collection method for wireless energy-carrying communication, comprising the following steps:

[0053] Step S1, constructing a time-varying channel gain model to obtain the channel gain between the user and the hybrid access point in each time frame; the channel gain is obtained according to the following formula:

[0054] In the formula, represents the channel gain between user i and the hybrid access point at time frame t, d i represents the distance between user i and the hybrid access point, n represents the path loss exponent, and λ represents an exponential random variable with a mean of 1.

[0055] Step S2, constructing an energy evolution model, collecting energy through a radio frequency energy collector equipped with each user, and obtaining the energy level of the user according to the energy evolution model.

[0056] Each user is equipped with a RF energy harvester, such as the P2110B RF energy harvester. Represents the received power at the user, which is calculated as follows: In this formula, is the channel gain of user i in frame t, and P is the transmitting power. According to power division, Part of the power is transferred to the energy harvester. The RF energy conversion process is nonlinear, which is a function of the received power. Therefore, this application adopts a nonlinear energy harvesting model based on reality. The energy conversion efficiency is represented by η, which ranges from [0,1]. The energy conversion efficiency is calculated as in, Ω=1 / (1+e ab )

[0057] Where M represents the maximum energy harvesting power, and the values ​​of a and b depend on the circuit of the RF energy harvester.

[0058] make Expressed as the energy collected by user i in time frame t, the collected energy is calculated as:

[0059] In the formula, η represents the energy conversion efficiency, θ represents the power division ratio, represents the received power at the user, τ d Indicates the length of the downlink period in each frame.

[0060] The energy level evolves according to the following formula:

[0061] In the formula, B max represents the battery capacity of each user, represents the energy collected by user i in time frame t, represents the energy consumed by user i in uplink transmission in time frame t.

[0062] Step S3, constructing a wireless energy-carrying communication downlink Markov decision model, where the downlink state is constructed as the channel state of all devices, i.e. All possible states constitute the downlink state set, namely The downlink action is constructed as the power allocation of the superimposed signal, i.e. All possible actions constitute the downlink action set, namely Rewards are structured as downlink throughput.

[0063] Step S4: construct a wireless energy communication uplink Markov decision model, where the uplink state is constructed as the channel state of a single device and its current power. All possible states of each device constitute the uplink state set of the device Uplink actions are constructed as single device transmission probability and transmission time slot selection All possible actions of each device constitute the uplink action set of the device Rewards are structured as uplink throughput.

[0064] Step S5: Centralized Q learning is adopted in the downlink layer, and the hybrid access point is used as an agent. The hybrid access point selects the action corresponding to the maximum Q value in the current state according to the channel state of all devices in the current time frame, determines the power allocation of the signal, and then calculates the sum rate of the downlink of the current time frame as a reward based on the transmission result.

[0065] First, initialize the learning parameters α, γ∈[0,1] and the centralized Q table of centralized Q learning. During the system warm-up phase, the hybrid access point randomly selects a downlink action based on its state and updates the state-action corresponding Q value according to the following Bellman equation: Q(s t ,a t )=(1-α)Q(s t ,a t )+α(r(s t ,a t )+γmaxQ(s t+1 ,a t+1 ))

[0066] In the formula, Q(s t ,a t ) is state s t Take action a t The corresponding Q value, r(s t ,a t ) is state s t Take action a t The reward obtained (i.e., the downlink throughput in frame t), maxQ(s t+1 ,a t+1 ) is the maximum Q value corresponding to the next time slot state predicted by the system, α is the learning parameter, γ is the discount factor, γ∈[0,1].

[0067] After the warm-up period ends, at the beginning of each time slot, the hybrid access point will Select the action corresponding to the maximum Q value with probability (1-ε), and decay the ε value, that is, Determine the power distribution of its signal And ensure that the power allocation meets the device's SIC successful decoding and In this formula, P is the total transmit power of the hybrid access point. is the transmission power allocated to user i in time frame t, and then the sum rate of the downlink is calculated as a reward based on its transmission results, that is, the downlink throughput.

[0068] The sum rate of the downlink is obtained according to the following formula:

[0069] Where W is the bandwidth, θ is the power split ratio, i is the current user, N is the total number of users, is the channel gain of user i in time frame t, is the signal transmission power allocated to user i in time frame t, and user j is the user other than user i, is the transmission power of the signal allocated to user j in time frame t, and n0 is the noise power.

[0070] Furthermore, the device continues to update the Q value according to the Bellman formula according to the current selected action and the downlink throughput value. This step is repeated for each subsequent time slot until the channel access decision process is completed.

[0071] Step S6: Distributed Q learning is adopted in the uplink layer. Each device acts as an agent. Each device selects the action corresponding to the maximum Q value in the current state according to its own channel state and real-time power in the current time frame, determines its transmission probability and time slot selection, and then calculates the sum rate of the uplink in the current time frame as a reward based on the transmission result.

[0072] First, each device initializes its independent distributed Q learning parameters α, γ∈[0,1] and distributed Q table. In the system warm-up phase, each device first randomly selects an upward action according to its state and updates the corresponding Q value according to the Bellman formula: Q(s t ,a t )=(1-α)Q(s t ,a t )+α(r(s t ,a t )+γmax Q(s t+1 ,a t+1 ))

[0073] In the formula, Q(s t ,a t ) is state s t Take action a t The corresponding Q value, r(s t ,a t ) is state s t Take action a t The reward obtained (i.e., the downlink throughput in frame t), maxQ(s t+1 ,a t+1 ) predicts the maximum Q value corresponding to the next time slot state of each device, α is the learning parameter, γ is the discount factor, γ∈[0,1].

[0074] After the warm-up, at the beginning of each time slot, each device will Select the action corresponding to the maximum Q value with probability (1-ε), and decay the ε value, that is, Determine its transmission probability and time slot selection Then the sum rate of the uplink is calculated as a reward based on its transmission results, namely the uplink throughput.

[0075] The sum rate of the uplink is obtained according to the following formula:

[0076] Where W is the bandwidth, i is the current user, and N is the total number of users. is the channel gain of user i in time frame t, is the signal transmission power of user i in time frame t, user j is the user other than user i, is the channel gain of user j in time frame t, is the signal transmission power of user j in frame t, and n0 is the noise power.

[0077] Furthermore, the device continues to update the Q value according to the Bellman formula according to the current selected action and the uplink throughput value. This step is repeated for each subsequent time slot until the channel access decision process is completed.

[0078] Step S7, using stateless Q learning to optimize the energy collection of step S2, the system acts as an agent, and in each cycle the system selects the action with the highest probability, controls the uplink frame size and the downlink power split ratio, and then the system collects the rewards of the uplink and downlink of the current cycle as the stateless reward R κ , The system updates the Q value and probability mass function according to the reward value until the system converges, thereby optimizing energy harvesting.

[0079] First, the system initializes the learning rate λ∈[0,1], probability mass function (PMF) and stateless Q table of stateless Q learning. The probability mass function calculates the probability of taking each action a i The probability is expressed as:

[0080] In the formula, a i represents the current action, a represents all possible actions, Pr(a i ) indicates selecting action a i The probability of Q(a i ) indicates action a i The corresponding system Q value, T represents the continuous time frame.

[0081] During the warm-up phase, the system randomly selects an action. After the warm-up, at the beginning of each cycle κ, the system will select the action a with the highest probability with probability (1-ε) κ = argmax Pr(a), and decays the value of ε. κ= [M, θ] controls the uplink frame size M and the downlink power split ratio θ. Afterwards, the system collects uplink and downlink rewards during period κ to obtain the cumulative sum rate of uplink and downlink during this period as the stateless reward of period κ:

[0082] Then, the system updates the Q value based on the reward value using the following formula: Q(a)←Q(a)+λ(r(a)-Q(a))

[0083] In the formula, Q(a) represents the system Q value corresponding to action a, λ represents the learning rate, λ∈[0,1], and r(a) represents the stateless reward function value corresponding to action a;

[0084] The probability mass function is:

[0085] In the formula, a i represents the current action, a represents all possible actions, Pr(a i ) indicates selecting action a i The probability of Q(a i ) indicates action a i The corresponding system Q value, T represents the continuous time frame.

[0086] The present invention adopts multi-layer Q learning, namely Multi-Q, in which the downlink layer learns the downlink superposition signal power allocation, the uplink layer learns the uplink device time slot selection and transmission probability, and the stateless layer learns the system time frame size and power division ratio. Figure 3 The Multi-Q framework is shown. Traditional Q-learning is used for the downstream and upstream layers, while stateless Q-learning is used for the stateless layers. All layers use ε-greedy for action selection. Therefore, after the warm-up phase, each agent has ε probability to randomly select an action, and then the value of ε is decayed to ensure convergence.

[0087] Specifically, at the beginning of each cycle κ, the system uses stateless Q learning to learn the probability mass function PMF to calculate the probability of taking each action. κ=[M,θ] to determine the power split ratio θ for energy collection in the current cycle and the frame size M for uplink. Among them, θ represents the proportion of received power dedicated to information decoding. The remaining (1-θ) part of the received power will be sent to the energy collector. During the warm-up period, the system randomly determines the uplink frame size and the downlink power split ratio. For each frame size and power split ratio, the Q table will update the Q value according to the sum rate. After several cycles, the system will select the frame size and power split ratio with the highest Q value. Frame sizes and power allocation ratios that obtain high system sum rates will obtain higher Q values. Each time an action is selected, its Q value will be updated based on its past rewards, current rewards, and possible future rewards.

[0088] During each downlink time frame of this cycle, HAP uses downlink layer centralized Q learning as an agent. HAP calculates the channel status of all devices. Select the action corresponding to the maximum Q value, that is Determine the power distribution of its signal All signals are then superimposed together according to this power allocation, and the resulting composite signal is transmitted to all users. The Q-table will update the Q values ​​corresponding to different power allocations under different channel conditions based on the downlink throughput obtained by the selected action. Power allocations that achieve high throughput will achieve high Q values. Each time a power allocation is selected, its Q value will be updated based on its past throughput, current throughput, and predicted future throughput. After several cycles, the HAP will mainly select the power allocation with the highest Q value in pursuit of high downlink throughput. Therefore, as the learning process converges, for each given channel state, the optimal power allocation will achieve the highest Q value.

[0089] During each uplink time frame, users use frame time slot Aloha for uplink transmission, and the hybrid access point is equipped with a SIC decoder to decode the information. Each device is an intelligent agent that learns its uplink behavior independently. Each device learns its uplink behavior based on its channel status and real-time power. Select the action corresponding to the maximum Q value: Determine its transmission probability and time slot selection Then each user uses the transmission probability ρ that he has learned independently i Select the learned time slot δ i Use the full available energy for uplink transmission. This means that in each frame, each device will learn to choose a specific uplink transmission time slot and transmission probability. Similar to the downlink layer, the Q-table will update the Q value of each action-state pair until convergence. Therefore, for a given channel state, the uplink transmission time slot and transmission probability with the highest transmission rate will obtain the highest Q value.

[0090] In order to further illustrate the effectiveness and superiority of the present invention, a simulation experiment is conducted on the suboptimal decision fusion rule adopted by the present invention, and the suboptimal decision fusion rule is compared with the existing optimal decision fusion rule.

[0091] First, we studied convergence, with the user placed at distances of 1, 5, and 9 meters from the HAP, and ran the simulator for 200 iterations, with each iteration containing 150 frames. Figure 7 The uplink and downlink rates are plotted in . The warm-up period is 15000 frames. Figure 7 , we can see that the uplink and downlink rates converge after 140 iterations. Specifically, the downlink rate converges to around 34b / s / Hz, and the uplink rate converges to around 13.66b / s / Hz.

[0092] Second, we study the uplink and downlink layer learning rates, including the uplink learning rate α u , downlink learning rate α d , stateless learning rate λ, discount factor γ and warmup period. Figure 8 It can be seen that each learning parameter combination converges to a different sum rate. Due to the short warm-up period, the system will experience more randomness during convergence. When the stateless layer uses a suitable learning rate λ, the system converges faster, from Figure 8 It can be seen that when the learning rate λ = 0.1, the system converges fastest and more stably. Therefore, considering all factors, this scheme prefers λ = 0.1. u =0.1,α d =0.1, γ=0.9, warmup=10000.

[0093] Next we compare Multi-Q with polling, TDMA, and Aloha. The polling protocol is used for both uplink and downlink transmissions; that is, the HAP transmits downlink data to each device in turn, and the devices transmit uplink data to the HAP in turn. For TDMA and frame-slotted Aloha, these protocols are only used for uplink transmissions. Both protocols consider downlink NOMA and uniform power distribution. Then during the uplink, TDMA allocates a dedicated time slot to each device. As for frame-slotted Aloha, devices with sufficient energy randomly compete for uplink time slots, and the HAP is equipped with a SIC decoder for information decoding. We will measure and compare the performance of these protocols in terms of average system sum rate, average downlink transmission rate, and average uplink transmission rate.

[0094] In addition, we also studied different HAP transmit powers, device locations, and power allocation ratios. Each simulation has 30,000 time frames, and the results of the last 300 frames after convergence are collected and the average of 10 simulation runs is plotted. The frame size of TDMA is 3, and the path loss exponent is n = 2.7. In terms of computational complexity, Multi-Q involves three layers of Q learning. By analyzing the computational complexity of each server at time t, for the downlink layer and uplink layer, each server needs to determine the Q value of each state-action pair; for the stateless layer, the server needs to determine the Q value of each action. For each layer, the Q value is updated only based on the server's reward. In addition, Multi-Q is able to adapt to larger-scale networks. However, the computational complexity of the downlink layer may increase with the increase in scale because it requires global information on the server.

[0095] For different HAP transmission powers, Fig. 9 (a) shows the average sum rate of uplink and downlink. In all methods, the total rate of uplink and downlink increases with the increase of HAP transmission power. Specifically, both uplink and downlink rates increase with the increase of HAP transmission power, as shown in Fig. 9 (b) The reason for the increase in uplink rate is that the user collects more energy at a higher HAP transmit power. Therefore, the user is able to transmit at a higher transmit power. Similarly, a higher HAP transmit power results in a higher downlink rate. Multi-Q performance is best when changing the HAP transmit power from 1W to 5W.

[0096] from Fig. 9 (a) It can be clearly seen that Multi-Q can always achieve the highest sum rate when the HAP transmit power is 1 W to 5 W. In addition, the average rate of Multi-Q is 44.3 b / s / Hz, which is 6 times that of Aloha, 2.3 times that of TDMA, and 30% higher than polling. Fig. 9(b) shows that Multi-Q achieves an average downlink transmission rate of 31.9b / s / Hz, which is the highest among all methods and 6.6b / s / Hz higher than polling. However, the downlink rate of TDMA and frame slot Aloha is only zero. This is because both TDMA and frame slot Aloha adopt uniform power allocation for each user during the downlink period, and uniform power allocation will lead to decoding failure. In addition, Multi-Q outperforms polling in both uplink and downlink. Specifically, the uplink and downlink average rates of Multi-Q are 12.4b / s / Hz and 31.9b / s / Hz, respectively, which are 6.6b / s / Hz and 4.1b / s / Hz higher than polling, respectively. This is because Multi-Q users use the entire downlink cycle to receive data. Polling users only receive data when the HAP polls data. In terms of downlink, polling encounters idle time slots when the user does not obtain enough energy. However, Multi-Q can dynamically adjust the frame size and transmission probability according to the user's battery level and channel conditions to avoid idle time slots. Overall, Multi-Q has a significant advantage in sum rate, which is 6 times that of the frame time slot Aloha method. At the same time, Multi-Q also shows a great advantage in downlink rate, which is 26% higher than polling and 100% higher than TDMA and frame time slot Aloha.

[0097] For different user locations, we consider five groups of user locations. The distance (in meters) from each user to the HAP is as follows: [5, 5, 5], [4, 5, 6], [3, 5, 7], [2, 5, 8], and [1, 5, 9]. The total distance from each group of devices to the HAP is 15 meters. The users are moved to different locations to obtain different channel gain conditions. The SIC decoding is improved with the channel gain difference. Specifically, the HAP transmission power is 3 watts, and the frame size of Aloha and TDMA is 3. The path loss exponent is n = 2.7. As Fig.10As shown in (a), the average sum rate of uplink and downlink increases when the distance difference of users is larger. This is because the channel gain of users encounters significant differences when placing users at different distances from the HAP. Therefore, the energy collected by users varies greatly. This also means that there will be a user located near the HAP that transmits at high power, while another user is located far away from the HAP and uses low transmit power. This difference in transmit power helps increase the number of successful SIC decoding. Overall, Multi-Q outperforms all other methods. When the user positions differ greatly, Multi-Q achieves an average sum rate of 32b / s / Hz for different locations. Meanwhile, the average sum rates of Aloha, polling, and TDMA are 2.5b / s / Hz, 29b / s / Hz, and 4b / s / Hz, respectively. When the user is 1 meter, 5 meters, and 9 meters away from the HAP, the sum rate of Multi-Q is 44b / s / Hz, which is 6 times that of Aloha, 3 times that of TDMA, and 30% higher than polling. The reason why Multi-Q performs better is that the downlink rate of Aloha and TDMA is zero, e.g. Fig.10 (b). Since both Aloha and TDMA use uniform power allocation in the downlink, they always encounter failures during downlink transmission. Multi-Q outperforms polling because Multi-Q is able to simultaneously learn the frame size and transmission probability to avoid idle time slots. Overall, Multi-Q consistently achieves the highest sum rate across different user locations, which is an average of 11 times that of Aloha, 7 times that of TDMA, and 10% higher than polling. In addition, Multi-Q shows a great advantage in downlink rate, which is 100% higher than TDMA and Aloha, and 5% higher than polling.

[0098] For different initial power split ratios, we set the users at 1 m, 5 m, and 9 m distances from the HAP. The HAP transmission power is 3 W, the frame size of TDMA and Aloha is 3, and the path loss exponent is n = 2.7. The power split ratio is gradually changed from 0 to 1 with a step size of 0.1. Initially, the power allocation ratio is zero, which means that all the received power is used for energy harvesting. After that, the power allocation ratio is increased to 0.1. Therefore, 10% of the power is directed for data reception and the remaining 90% is used for energy harvesting. The power split ratio is then increased in steps of 0.1 until it reaches 1.0. Reference Fig.11 (a), Multi-Q achieves a sum rate of about 44b / s / Hz; it is able to converge to the optimal power split ratio starting from any initial ratio. The polled sum rate continues to increase as the power split ratio increases to 0.9, because when the power split ratio is increased, more power is allocated for downlink data reception and less power is used for energy harvesting. Fig.11(b) It can be seen that polling achieves an average rate of 25.3b / s / Hz for the downlink and 5.7b / s / Hz for the uplink. It can be seen that the downlink rate of polling is about 5 times its uplink rate. Therefore, even though the uplink rate of polling is reduced, the sum rate of polling is increased because the increase in its downlink rate is greater than the decrease in the uplink rate. When the power split ratio is increased from 0.9 to 1.0, the sum rate of polling decreases because all the downlink power is allocated to the energy harvester. When the power split ratio is increased from 0 to 1.0, the sum rate of Aloha and TDMA decreases because the users use a uniform power allocation for the downlink, resulting in no difference between each user's transmission, so that each user cannot decode the received data packets. In addition, a higher power split ratio will result in less energy harvesting and uplink transmission power, so the uplink rate is significantly lower.

[0099] We also study the performance of Aloha with different frame sizes. Fig.11 As shown in (a), Aloha achieves the highest sum rate when the frame size is 1. Although larger frames mean fewer collisions, frame size 1 performs better than larger frames. This is because we take SIC into account, which allows multiple concurrent transmissions to be decoded in the same time slot. In addition, smaller frame sizes extend the transmission period, so users can transmit for longer periods of time. A frame size of 1 means that all transmissions occur in one time frame, but only transmissions whose SINR of the transmitting user power meets the SIC decoding conditions will be successful. Under the Aloha method, users do not learn transmission probabilities to avoid SIC decoding failures, and all users can only transmit concurrently in the same time slot. Multi-Q improves the probability of successful SIC decoding at the receiving end by learning the optimal time frame size, uplink transmission probability and time slot selection. Therefore, Multi-Q performs best among all methods.

[0100] Multi-Q achieves an average sum rate of 44b / s / Hz, which is 9.8 times more than TDMA, 6.5 times more than Aloha when the frame size is 1, and 1.4 times higher than polling. This is because Multi-Q simultaneously learns the power split ratio, frame size, uplink transmission probability, uplink time slot selection, and downlink power allocation for all users. Specifically, Multi-Q learns the optimal frame size and time slot selection to avoid uplink decoding failures and idle time slots. Learning the downlink power allocation for each user enhances SIC decoding success. It also learns the power split ratio to balance the uplink and downlink rates. Overall, for different preset split ratios, Multi-Q shows a huge advantage over all other methods in terms of sum rate. In addition, Multi-Q always achieves the highest downlink rate and uplink rate, which are 31.5b / s / Hz and 12.5b / s / Hz, respectively.

[0101] The present invention fully considers the actual situation of wireless energy-carrying communication networks, adopts nonlinear radio frequency energy conversion and energy evolution of equipment, and considers a causal channel gain model. Therefore, the method has good applicability in real wireless energy-carrying communication networks.

[0102] Finally, it should be noted that the parts of the present invention that are not described in detail are all prior art. Those of ordinary skill in the art can understand that the above are only preferred examples of the invention and are not intended to limit the invention. Although the invention is described in detail with reference to the aforementioned examples, those of ordinary skill in the art can still modify the technical solutions recorded in the aforementioned examples, or replace some of the technical features therein with equivalents. Any modifications, equivalent replacements, etc. made within the spirit and principles of the invention should be included in the scope of protection of the invention.

Claims

1. A method for joint uplink and downlink channel access and energy collection for wireless energy-carrying communication, characterized in that: The following steps are involved: Step S1, constructing a time-varying channel gain model to obtain a channel gain between a user and a hybrid access point in each time frame; Step S2, constructing an energy evolution model, collecting energy through a radio frequency energy collector equipped by each user, and obtaining the energy level of the user according to the energy evolution model; Step S3, constructing a wireless energy-carrying communication downlink Markov decision model, where the downlink state is constructed as the channel state of all devices, the downlink action is constructed as the power allocation of the superimposed signal, and the reward is constructed as the downlink throughput; Step S4, constructing a wireless energy-carrying communication uplink Markov decision model, where the uplink state is constructed as the channel state of a single device and its existing power, the uplink action is constructed as the transmission probability and transmission time slot selection of a single device, and the reward is constructed as the uplink throughput; Step S5: Centralized Q learning is adopted in the downlink layer, and the hybrid access point is used as an agent. The hybrid access point selects the action corresponding to the maximum Q value in the current state according to the channel state of all devices in the current time frame, determines the power allocation of the signal, and then calculates the sum rate of the downlink of the current time frame as a reward based on the transmission result. Step S6: Distributed Q learning is adopted in the uplink layer. Each device acts as an agent. Each device selects the action corresponding to the maximum Q value in the current state according to its own channel state and real-time power in the current time frame, determines its transmission probability and time slot selection, and then calculates the sum rate of the uplink in the current time frame as a reward based on the transmission result. Step S7, using stateless Q learning to optimize the energy collection of step S2, the system acts as an agent, and in each cycle the system selects the action with the highest probability, controls the uplink frame size and the downlink power split ratio, and then the system collects the rewards of the uplink and downlink of the current cycle as the stateless reward R κ , The system updates the Q value and probability mass function according to the reward value until the system converges, thereby optimizing energy harvesting.

2. The method for joint uplink and downlink channel access and energy collection of wireless energy-carrying communication according to claim 1, characterized in that: In step S1, time is divided into frames and indexed by t, and the channel gain is obtained according to the following formula: In the formula, represents the channel gain between user i and the hybrid access point at time frame t, d i represents the distance between user i and the hybrid access point, n represents the path loss exponent, and λ represents an exponential random variable with a mean of 1.

3. The method for joint uplink and downlink channel access and energy collection of wireless energy-carrying communication according to claim 2, characterized in that: In step S2, the energy level evolves according to the following formula: In the formula, B max represents the battery capacity of each user, represents the energy collected by user i in time frame t, represents the energy consumed by user i in uplink transmission in time frame t, in, In the formula, η represents the energy conversion efficiency, θ represents the power division ratio, represents the received power at the user, τ d Indicates the length of the downlink period in each frame.

4. The method for joint uplink and downlink channel access and energy collection of wireless energy-carrying communication according to claim 1, characterized in that: In step S3, the downlink Markov decision model specifically includes: the downlink state is constructed as the channel state of all devices, that is, All possible states constitute the downlink state set, namely The downlink action is constructed as the power allocation of the superimposed signal, i.e. All possible actions constitute the downlink action set, namely Rewards are structured as downlink throughput.

5. The method for joint uplink and downlink channel access and energy collection of wireless energy-carrying communication according to claim 4, characterized in that: In step S4, the uplink Markov decision model specifically includes: the uplink state is constructed as the channel state of a single device and its current power All possible states of each device constitute the uplink state set of the device Uplink actions are constructed as single device transmission probability and transmission time slot selection All possible actions of each device constitute the uplink action set of the device Rewards are structured as uplink throughput.

6. The method for joint uplink and downlink channel access and energy collection of wireless energy-carrying communication according to claim 5, characterized in that: In step S5, during the system warm-up phase, the hybrid access point randomly selects a downlink action according to its state, and updates the state-action corresponding Q value according to the following Bellman equation: Q(s t ,a t )=(1-α)Q(s t ,a t )+α(r(s t ,a t )+γmaxQ(s t+1 ,a t+1 )) In the formula, Q(s t ,a t ) is state s t Take action a t The corresponding Q value, r(s t ,a t ) is state s t Take action a t The reward obtained, maxQ(s t+1 ,a t+1 ) is the maximum Q value corresponding to the next time slot state predicted by the system, α is the learning parameter, γ is the discount factor, γ∈[0,1].

7. The method for joint uplink and downlink channel access and energy collection of wireless energy-carrying communication according to claim 6, characterized in that: In step S5, the power allocation should satisfy the requirement that the SIC of the device is successfully decoded and Where P is the total transmit power of the hybrid access point, is the transmission power allocated to user i in time frame t, The sum rate of the downlink is obtained according to the following formula: Where W is the bandwidth, θ is the power split ratio, i is the current user, N is the total number of users, is the channel gain of user i in time frame t, is the signal transmission power allocated to user i in time frame t, and user j is the user other than user i, is the transmission power of the signal allocated to user j in time frame t, and n0 is the noise power.

8. The method for joint uplink and downlink channel access and energy collection of wireless energy-carrying communication according to claim 7, characterized in that: In step S6, during the system warm-up phase, each device first randomly selects an upward action according to its state, and updates the corresponding Q value according to the Bellman equation.

9. A method for joint uplink and downlink channel access and energy collection for wireless energy-carrying communication according to claim 8, characterized in that: In step S6, the sum rate of the uplink is obtained according to the following formula: Where W is the bandwidth, i is the current user, and N is the total number of users. is the channel gain of user i in time frame t, is the signal transmission power of user i in time frame t, user j is the user other than user i, is the channel gain of user j in time frame t, is the signal transmission power of user j in frame t, and n0 is the noise power.

10. The method for joint uplink and downlink channel access and energy collection of wireless energy-carrying communication according to claim 1, characterized in that: In step S7, the system updates the Q value according to the reward value using the following formula: Q(a)←Q(a)+λ(r(a)-Q(a)) In the formula, Q(a) represents the system Q value corresponding to action a, λ represents the learning rate, λ∈[0,1], and r(a) represents the stateless reward function value corresponding to action a; The probability mass function is: In the formula, a i represents the current action, a represents all possible actions, Pr(a i ) indicates selecting action a i The probability of Q(a i ) indicates action a i The corresponding system Q value, T represents the continuous time frame.

Images