Channel access and energy scheduling method and device based on deep reinforcement learning

Abstract

The present application relates to a channel access and energy scheduling method and system based on deep reinforcement learning, belonging to the field of wireless communication, aiming at the application scene of wireless devices in an energy shortage and limited state observation value, such as wireless devices working in earthquake disaster areas and other difficult to frequently use human intervention, the method of the present application can use a distributed deep reinforcement learning neural network to obtain a relatively excellent energy scheduling and channel access strategy, not only enabling the wireless device to have certain energy management and scheduling capability, but also reducing the information collision between wireless devices as much as possible, while improving the throughput of the whole communication system.

Description

Technical Field

[0001] This disclosure relates to wireless communication, and more particularly to a method and apparatus for channel access and energy scheduling based on deep reinforcement learning. Background Technology

[0002] According to reports, the number of IoT devices in wireless networks is gradually increasing and will reach a very large number in the future. At the same time, IoT devices usually consume energy to operate, but the devices' batteries are limited, which requires manual replacement of IoT device batteries, thus increasing the overall maintenance cost.

[0003] Fortunately, the emerging energy harvesting (EH) technology in recent years enables IoT devices to collect energy from their surroundings, thus achieving energy recycling. However, this technology also has a significant drawback: the energy collected by wireless devices from the environment is random and unpredictable. This necessitates that wireless devices possess energy management and scheduling systems to rationally manage their energy consumption in complex energy environments, maximizing their operational time.

[0004] In recent years, for wireless communication systems with energy dissipation (EH), some scholars have studied many access strategies and energy scheduling methods to enable the network to effectively improve throughput by making low-efficiency use of collected energy. The specific methods can be divided into the following three types:

[0005] (1) The traditional method is to use algorithms such as dynamic programming to plan the overall action of the communication system. Its main drawback is that it requires knowing all the information of the system, which is very difficult to achieve in practice.

[0006] (2) Traditional methods have limited practical application value because they require knowledge of the system parameters. Therefore, learning-based methods have gradually emerged and been applied to wireless communication systems with energy harvesting. However, reinforcement learning-based methods require more space as the system size increases.

[0007] (3) In recent years, researchers have successfully combined reinforcement learning and deep learning, resulting in deep reinforcement learning (among which Deep-Q Network (DQN) is a representative example). This combination can effectively improve stability and handle large-scale state spaces, and some scholars have applied it to wireless communication networks to improve network throughput. Although deep reinforcement learning has solved the above problems to some extent, current deep reinforcement learning is centralized. Centralized deep reinforcement learning requires a "central controller" that needs to know all the information of the system, such as the channel state and the energy of the UE, before it can make decisions. However, there is a lack of a device that can play this role, so centralized deep reinforcement learning has limitations in wireless communication networks. Summary of the Invention

[0008] To address the aforementioned limitations of existing technologies, the present invention aims to provide a superior energy scheduling and channel access method. This method employs a "distributed deep reinforcement learning" approach, applied to a communication model, where each UE (User Equipment) makes its own decisions based on its state observations. Experimental results demonstrate the feasibility of this method. It not only enables wireless devices to possess certain energy management and scheduling capabilities but also minimizes information collisions between wireless devices, thereby improving the overall throughput of the communication system. The distributed deep reinforcement learning refers to equipping each wireless device with a neural network, allowing the device to make its own decisions based on its observations.

[0009] To achieve the above objectives, the technical solution of the present invention is as follows.

[0010] Firstly, this invention proposes a channel access and energy scheduling method based on deep reinforcement learning. The user equipment module obtains the first T1 time slots of user equipment UE i, including the success or failure of transmission on each channel and the remaining battery capacity of user equipment UE i, as input information. This input information is then fed into the pre-trained deep reinforcement learning network corresponding to user equipment UE i. The deep reinforcement learning network outputs the optimal action for user equipment UE i, where i = 1, 2, ..., N, and N is the total number of user equipment UEs, greater than or equal to the number of channels; where T1 is a set value; the action is either user equipment UE i choosing to consume energy to transmit information to the access point at time slot t, or user equipment UE choosing to suspend transmission at time slot t.

[0011] The above technical solution is applicable to wireless communication systems with one access point (AP, such as a base station) and multiple wireless user devices, where the number of channels is less than or equal to the number of devices. In this solution, the success or failure of the first T time slots of information transmission and the remaining capacity are used as input information to a deep reinforcement learning network. The aim is to obtain a superior energy scheduling and channel access strategy through the deep reinforcement learning network, thereby minimizing information collisions between wireless devices and improving the throughput of the entire communication system. This technical solution is particularly suitable for wireless devices operating in earthquake-stricken areas or other environments where frequent human intervention is difficult. When wireless devices may be in application scenarios with limited energy and finite state observations, by adopting this solution, wireless user devices can collect a certain amount of energy from the surrounding environment at the beginning of each time slot as needed using energy harvesting devices. The collected energy is then used to send information to the access point (AP), automatically optimizing energy usage and information transmission.

[0012] In the above technical solution, the remaining capacity is obtained through an energy model; in the energy model, the UE collects energy and stores it in its corresponding battery, and the energy collection process satisfies the Markov property. If the battery capacity reaches the upper limit B... max If the excess energy is discarded, the UE will send information to the access point to consume energy.

[0013] The battery evolution equation for UE i from time slot t to t+1 is:

[0014] B i [t+1]=min{B max B i [t-1]-P i [t]+E i [t]}

[0015] B i [t+1] represents the remaining battery capacity at time slot t+1 of uE i, B i [t-1] represents the remaining battery capacity at time slot t-1 of uE i, and P i [t] represents the energy consumption value when the UE transmits information at time t in time slot i, E i [t] represents the energy collected by the i-th UE at time slot t.

[0016] In the above technical solution, the success or failure status of the first T time slots of information transmission is obtained through the channel model;

[0017] In the channel model, the channel state is represented by a matrix C[t] of size N×K, where N is the number of UEs and K is the number of channels. The value of the element in the i-th row and j-th column of the matrix represents the amount of information that the i-th UE can send from the j-th channel. The amount of information is the channel state.

[0018] If more than one UE simultaneously selects the same channel to send information, a collision will occur on that channel. All UEs that select this channel will fail to send information, and the access point (AP) will not receive the information. In other words, a single channel can only be used for the information transmission of a single UE.

[0019] In the above technical solution, the implementation of deep reinforcement learning networks includes the following steps:

[0020] S1. Integrate the input information of user equipment UE i obtained by the user equipment module into the current state s. i [t], i = 1, 2, ..., N, where N is the total number of user equipment (uE);

[0021] S2. Select action A using the ε-greedy algorithm. i [t], i = 1, 2, ..., N;

[0022] S3, User Equipment (UE) i executes the selected action A i [t], i = 1, 2, ..., N;

[0023] S4. Actions based on each User Equipment (UE) i [t], obtain the action set A[t] = {A i [t], i = 1, 2, ..., N;

[0024] S5. Based on the action set A[t], obtain a vector R[t] of size 1×N to represent the reward vector. Each value in the reward vector is the reward value obtained by the corresponding UE. At the same time, the environment state S[t] is transitioned to the next state S[t+1].

[0025] S[t]={S i [t], i = 1, 2, ..., N}

[0026] S[t+1]={s i [t+1], i = 1, 2, ..., N}

[0027] R[t]={r i [t], i = 1, 2, ..., N}

[0028] S6, the user equipment module will send the s corresponding to UE i i [t],A i [t], Ri [t],s i [t+1] Save and update the deep reinforcement learning network.

[0029] In the above technical solution, the reward value is set as follows:

[0030] If the user equipment (UE) i chooses not to send information, it will receive a reward value of 0.

[0031] If user equipment UE i selects channel j to send information, and there is another user equipment UE sending information through channel j, then user equipment UE i receives a reward value of -1.

[0032] If user equipment UE i selects channel j to send information, and other user equipment UEs do not select this channel to send information, the reward value obtained by the user equipment UE is the channel state of the channel selected by this user equipment UE.

[0033] In the above technical solution, step S6 includes the following steps:

[0034] S61, User Equipment Module will s i [t],A i [t], R i [t],s i [t+1] Store the experience in the experience pool corresponding to each user equipment (UE). If the number of experience stored in the experience pool meets the requirements of D groups of samples, then execute step S62; otherwise, proceed to step S1. Wherein, D represents the capacity of the experience pool.

[0035] S62. Based on the set number of samples, randomly sample from the experience pool, calculate the target value target[i], and update the estimation neural network DQN1[i].

[0036] S63. Update the target neural network DQN2[i] for each time interval T1.

[0037] In the above technical solution, to make the Q value calculated by the neural network more accurate, the user equipment UE i is equipped with a set of neural networks. This set of neural networks consists of an estimation neural network DQN1[i] and a target neural network DQN2[i] with the same network structure. However, the network structure of the neural networks corresponding to each UE can be the same or different. For each UE located in a different position in the communication system, the environment they observe is also different. For example, the channel state they observe on the same channel is also different. Therefore, the input of each UE's neural network is determined according to its own observed environment.

[0038] In the above technical solution, the target value target[i] is calculated as follows:

[0039] If the next time slot is the terminating time slot T2, then target[i] = R i [t]; otherwise:

[0040] target[i] = R i [t]+γQ2′(s i [t+1])

[0041] In the formula: Y is the discount factor, Q2′(s) i [t+1]) represents the target neural network DQN2[i] corresponding to user equipment UE i in time slot t, and the state s. i [t+1] is the Q value corresponding to taking the optimal action.

[0042] In the above technical solution, the deep reinforcement learning network is updated using gradient descent, and the loss is calculated using the following loss function:

[0043] L = (target[i] - Q1(s) i [t])) 2

[0044] In the formula: Q1(s) i [t]) represents the estimation neural network DQN1[i] corresponding to user equipment UE i in time slot t, and the state s. i [t] is the Q value corresponding to taking the optimal action.

[0045] Secondly, the present invention provides a computer-readable storage device, which is a storage medium storing a computer program that can be loaded by a processor and executed by any of the methods described above. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 , one A schematic diagram of the communication model in a specific implementation method;

[0048] Figure 2 , one A schematic diagram illustrating the application of deep reinforcement learning networks for decision-making in one specific implementation method;

[0049] Figure 3 , one A pseudocode illustration of a specific implementation method using a deep reinforcement learning network;

[0050] Figure 4 In one specific implementation, the communication model is a comparison of the throughput of two schemes when using energy scheme one with 3 wireless devices and 2 channels;

[0051] Figure 5 In one specific implementation, the communication model is a comparison of the throughput of two schemes when using energy scheme one with 4 wireless devices and 3 channels;

[0052] Figure 6 In one specific implementation, the communication model is a comparison of the throughput of two schemes when using energy scheme one with 5 wireless devices and 4 channels;

[0053] Figure 7 In one specific implementation, the communication model is a comparison chart of the throughput of the two schemes when using energy scheme two with 3 wireless devices and 2 channels;

[0054] Figure 8 In one specific implementation, the communication model is a comparison chart of the throughput of the two schemes when using energy scheme two with 4 wireless devices and 3 channels;

[0055] Figure 9 In one specific implementation, the communication model is a comparison of the throughput of the two schemes when using energy scheme two with 5 wireless devices and 4 channels;

[0056] Figure 10 In one specific implementation, the communication model is shown in the throughput comparison diagram of the two schemes when using energy scheme two with 5 wireless devices and 4 channels and an increased penalty value for information collision.

[0057] In the figure: the red line represents the throughput of the communication system under the method of this invention, and the blue line represents the throughput of the communication system under the CSMA method. Detailed Implementation

[0058] This invention is used in wireless communication systems comprising one access point (AP, such as a base station) and multiple wireless user equipment (User Equipment). At the beginning of each time slot, each User Equipment collects energy from its surroundings using an energy harvesting device. This energy is then used to send information to the AP. When a User Equipment is in a state of energy scarcity and limited observations, a distributed deep reinforcement learning neural network can be used to obtain a superior energy scheduling and channel access strategy, thereby minimizing information collisions between User Equipment and improving the throughput of the entire communication system. Its application scenarios include wireless equipment operating in earthquake-stricken areas or other locations where frequent human intervention is difficult.

[0059] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0060] In recent years, deep reinforcement learning methods have been widely used. However, many centralized deep reinforcement learning methods assume that there is a "central controller" in the system. This controller needs to know all the real-time information of the system, such as the channel state and the energy state of all wireless devices. However, in reality, there are often no devices that can play this role. Therefore, centralized deep reinforcement learning is more inclined to theoretical research. Although it can achieve high throughput (centralized management will not have the problem of information collision because centralization is equivalent to assigning tasks to each device from a global perspective), it is difficult to implement in practice.

[0061] Therefore, this invention applies a distributed deep reinforcement learning method to our communication system. The goal is to equip each wireless device with a neural network and then use a distributed algorithm to train the neural network of the wireless device. By occupying relatively little space, a better energy scheduling and channel access strategy can be obtained to improve the throughput of the communication system. The throughput is measured by the amount of information that the access point (AP) can receive, so that our model has high throughput while being closer to practical applications.

[0062] In one specific implementation, a model of the overall wireless communication system is first defined, including a time model, a channel model, and an energy model, as follows:

[0063] (I) System and Time Model

[0064] Unless otherwise specified, N and K represent the number of user equipment and the number of channels, respectively, in this invention.

[0065] First, a communication model was designed, assuming that N User Equipments (UEs) send information to one Access Point (AP) through K links or orthogonal channels (assuming N≥K). Time is divided into time slots. At the beginning of each time slot, the UE can obtain energy from its surroundings and select its action based on a neural network. The overall system model diagram is shown below. Figure 1 As shown. The action is that the user equipment UE i chooses to consume energy to send information to the access point at time slot t, or the user equipment UE chooses to suspend and not send information at time slot t.

[0066] (II) Energy Model

[0067] At the beginning of each time slot, the user equipment (UE) i uses energy harvesting technology to collect a certain amount of energy from the surrounding environment and then stores it in its own battery.

[0068] In this invention, the energy harvesting process of the UE is defined as having Markov properties, and the maximum capacity of the UE's battery is B. max Its size is 10 units of energy, and the energy collected by the i-th UE at time slot t is defined as E. i [t], the remaining battery energy of the i-th UE in time slot t is B. i [t]. For ease of subsequent analysis, we define a minimum energy unit E0. In summary, E... i [t] = nE0, B i [t] = mE0, where n and m are both non-negative integers.

[0069] In each specific time slot, the UE first uses EH to collect a certain amount of energy from the surrounding environment, if the battery capacity reaches the upper limit B. max If the UE fails to transmit the signal, it will discard excess energy. The UE can choose to consume energy to send information to the AP or suspend transmission while waiting for the next time slot to collect more energy. For ease of analysis, we assume that information transmission is the only source of energy consumption, and that the battery is ideal and does not leak energy. Similarly, we assume that the energy consumed by the UE to activate the EH circuit is negligible.

[0070] (III) Channel Model

[0071] At time slot t, a channel state matrix C[t] of size N×K is defined, where N and K represent the number of UEs and the number of channels, respectively. The element C in the i-th row and j-th column of the matrix is... i，j [t] represents the channel state of the j-th channel observed by the i-th UE at time slot t, where 0 ≤ C i，j [t]≤3, for the same channel, each UE may also observe different channel states. Specifically: In some embodiments, the number of UEs is exactly equal to the number of channels, allowing wireless devices to autonomously manage and schedule their energy to maximize their operating time; in other embodiments, the number of UEs is greater than the number of channels, reducing information collisions between wireless devices and improving system throughput while considering energy scheduling.

[0072] For ease of subsequent analysis, we assume that UEs send information at a fixed rate of 3. Therefore, the actual amount of information that the AP can receive is the channel state of the channel selected by the UE. For example, at time slot t, the value of the i-th row and j-th column of the channel state matrix C[t] is 3. This means that the i-th UE observes that the channel state of the j-th channel is 3. If the i-th UE chooses to use the j-th channel to send information to the AP, then the AP actually receives 3 units of information (we use the actual amount of information received by the AP as the throughput indicator).

[0073] Secondly, a single channel can only be used for information transmission of a single UE. This means that at any given moment, if more than one UE selects the same channel to send information, a collision will occur on that channel, and all UEs that select this channel to send information will fail to send, and the access point (AP) will not receive the information.

[0074] User equipment (UE) i's first T1 timeslot information is used to determine the success or failure of each channel information transmission, employing a 1×(T1×K) vector H. i [t] records, where K is the number of channels; starting from the first element, every K elements represent the success or failure of information transmission in a time slot, and the element value is the corresponding channel state, which is the amount of information that can be transmitted in the channel. In this embodiment, T1 is set to 3, so at time slot t, each UE maintains its own vector H of size 1×3K. i [t] is used to record whether each uE successfully transmitted in each channel during the first three time slots, vector H i The first K elements of [t] represent the successful information transmission records of the i-th UE three time slots ago, the (K+1)th to (2K)th elements represent the successful information transmission records of the i-th UE two time slots ago, and the (2K+1)th to (3K)th elements represent the successful information transmission records of the i-th UE in the previous time slot. A successful information transmission record records the channel selected by each UE in the first three time slots and whether the transmission was successful. For example, if a UE selects channels 1, 2, and 3 in the first three time slots respectively, and the channel status observed by this UE is good during these three time slots (so the channel status is 3), and no other UE selects these three channels to transmit information during these three time slots, then no information collision occurs. Therefore, the corresponding H of this UE is... i [t] = [3, 0, 0, 0, 3, 0, 0, 0, 3]. This example is only an idealized case. In reality, if this UE selected the same channel as other UEs in the previous time slot, a collision would occur, and its corresponding value should be 1. For each uE, H i[t] can be used as part of the input to their respective neural networks.

[0075] At time slot t, each UE can only select one channel to send information at most. Therefore, we can obtain a vector A[t] of size 1×N to represent the set of actions of all UEs. A certain value A in the vector... i [t] indicates which channel the i-th UE selected to transmit information at time slot t, and 0 ≤ A i [t]≤K, 0 indicates that the UE chooses to suspend and not send information in this time slot, and other values ​​indicate that the UE has chosen the corresponding channel to send information.

[0076] At time slot t, for a given action set vector A[t], a vector P[t] of size 1×N can be obtained, representing the energy consumption of all UEs in this time slot. The i-th element in the vector represents the energy unit consumed by UE i in this time slot. If UE i does not choose to send information, the corresponding element is 0. If it chooses a channel to send information, the corresponding element is P, where P is the fixed energy consumption value when UE chooses to send information, and its size is 4 units of energy.

[0077] In summary, the battery evolution equation for UE i from time slot t to t+1 can be obtained as: B i [t+1]=min{B max B i [t-1]-P i [t]+E i [t]}.

[0078] (iv) Setting up distributed reinforcement learning algorithms and neural networks

[0079] Each UE in the corresponding communication model is equipped with a corresponding deep reinforcement learning network. For each UE located at different positions in the communication system, the environment they observe is also different. For example, the channel state they observe for the same channel is also different. Therefore, the input information of each UE's neural network is determined according to the environment it observes. The neural network DQN1[i] of the i-th UE at time slot t is mainly input by the vector H describing whether the transmission was successful or not in the first T1 time slots as described in the channel model. i [t] and the remaining battery energy described in the energy model is B. i [t] is composed of T1, which is a set value, such as 3. Its output is the optimal action of user equipment UE i, i = 1, 2, ..., N. The action is either user equipment UE i choosing to consume energy to send information to the access point at time slot t, or user equipment UE choosing to suspend and not send information at time slot t.

[0080] Meanwhile, the neural network structure of each UE is the same, which is a three-layer fully connected deep network with 64 and 32 nodes in the middle layer, respectively. The learning rate Y is 0.0001, and the batch size for training is 64. The output Q(s,a) of the neural network DQN1[i] is the score of each action made by the UE under the given environmental state, abbreviated as Q value. In the setting of this invention, the Q value is the score of the channel selected by each UE. Action 0 means not sending information, and other action values ​​mean that the UE selects the corresponding channel to send information. The neural network will score and output each action.

[0081] Because the parameters of the neural network are initially relatively random, in order to ensure the selection of the optimal action (i.e., the action with the largest Q-value) while also maintaining a certain exploration probability (i.e., the possibility of selecting a non-optimal action that may yield a larger reward value in the future), deep reinforcement learning networks employ the ε-greedy algorithm to select actions. This algorithm selects a random action with a probability of ε and the action with the largest Q-value with a probability of 1-ε. The user equipment (UE) i executes the optimal action A. i [t], then calculate the reward value for the selected action and update the environment state.

[0082] In this invention, the reward value setting is described as follows: If the UE chooses not to send information, it receives a reward value of 0. If the UE chooses to send information, but unfortunately another UE chooses the same channel to send information, the UE receives a reward value of -1. If the UE chooses a certain channel to send information, and other UEs do not choose this channel to send information, the reward value obtained by the UE is the channel state C of the channel it has selected, as observed by the UE. i，j [t] (i.e., the actual number of messages received by the AP).

[0083] To make the Q-value calculated by the neural network more accurate, this invention adopts the form of Fixed Q-targetsDQN, that is, there are two neural networks at each UE end, namely an estimation neural network and a target neural network, and their parameters w target and w eval Initially, it's the same. During training, the target neural network is used to compute the target value, and then the parameters w of the estimated neural network are updated using gradient descent. eval Since the parameters of the target neural network are constant, this will undoubtedly make the Q-value calculated by the estimating neural network closer to the target value calculated by the target neural network. Because the target neural network is also randomly initialized, it must also be continuously updated to adjust the parameters w. targetContinuous optimization is employed. The method involves directly transferring all parameters of the estimated neural network to the target neural network after training many batches, i.e., assigning values ​​to w. target =w eval The training algorithm for distributed reinforcement learning will be briefly described below.

[0084] The above process is as follows Figure 2 As shown, in Figure 3 The pseudocode for the above process is provided in the document, namely:

[0085] S0. The user equipment module is initialized first.

[0086] The user equipment module initializes each UE's neural network estimation neural network and target neural network, as well as its respective experience storage record experience pool D. The experience pool D records the environmental state observed by each UE in each previous time slot, the choices it made, the rewards it received, and the environmental state observed in the next time slot.

[0087] S1. For each UE i at a certain time slot t, the input information of UE i obtained by the user equipment module is integrated into the current state s. i [t], i = 1, 2, ..., N.

[0088] Each UE first obtains its own observations from the environment as input to its own deep neural network.

[0089] S2. Then, based on the output of the neural network, the ε-greedy algorithm is used to select action A. i [t], i = 1, 2, ..., N.

[0090] S3, User Equipment (UE) i executes the selected action A i [t], i = 1, 2, ..., N.

[0091] S4. Actions based on each User Equipment (UE) i [t], obtain the action set A[t] = {A i [t], i = 1, 2, ..., N}.

[0092] S5. Based on the action set A[t], interact with the environment to obtain a vector R[t] of size 1×N representing the reward vector. Each value in the reward vector is the reward value obtained by the corresponding UE. At the same time, the environment state S[t] is transitioned to the next state S[t+1].

[0093] S[t]={s i [t], i = 1, 2, ..., N}

[0094] S[t+1]={si [t+1], i = 1, 2, ..., N}

[0095] R[t]={r i [t], i = 1, 2, ..., N}

[0096] S6. The user equipment module will transfer the corresponding s to uE i. i [t],A i [t], R i [t],s i [t+1] Save and update the deep reinforcement learning network, specifically:

[0097] S61, The user equipment module will transfer the s corresponding to UE i i [t],A i [t], R i [t],s i [t+1] Store the experience in the experience pool corresponding to each user equipment (UE). If the number of experience stored in the experience pool meets the requirements of D groups of samples, then execute step S62; otherwise, proceed to step S1. Wherein, D represents the capacity of the experience pool.

[0098] S62. Based on the set number of samples, randomly sample from the experience pool, calculate the target value target[i], and update the estimated neural network DQN1[i].

[0099] The target value target[i] is calculated as follows:

[0100] If the next time slot is the terminating time slot T2, then target[i] = R i [t]; otherwise:

[0101] target[i] = R i [t]+γQ2′(s i [t+1])

[0102] In the formula: γ is the discount factor, Q2′(s i [t+1]) represents the target neural network DQN2[i] corresponding to user equipment uE i in time slot t, and the state s. i [t+1] is the Q value corresponding to taking the optimal action.

[0103] The estimated neural network DQN1[i] is updated using gradient descent, and the loss is calculated using the following loss function:

[0104] L = (target[i] - Q1(s) i [t])) 2

[0105] In the formula: Q1(s) i[t]) represents the estimation neural network DQN1[i] corresponding to user equipment UE i in time slot t, and the state s. i [t] is the Q value corresponding to taking the optimal action.

[0106] S63. At regular intervals, the parameters of the estimated neural network DQN1[i] are directly copied to the target neural network DQN2[i], i.e., w target =w eval .

[0107] Repeat steps S61-S63, the specific number of repetitions depending on the total number of time slots to be performed in the experiment.

[0108] When applying the above-trained deep neural network model:

[0109] The user equipment module obtains the first T1 timeslot information of user equipment uE i, the status of whether the transmission was successful or not on each channel, and the remaining battery capacity of user equipment uE i as input information. The input information is then fed into the trained deep reinforcement learning network corresponding to user equipment UE i. The deep reinforcement learning network outputs the optimal action of user equipment UE i, where i = 1, 2, ..., N, and T1 is a set value.

[0110] (V) Distributed Channel Access and Energy Management Framework

[0111] The distributed channel access and energy management framework involves each UE using its own environmental state (including collision data and energy status from previous time slots) to determine its action for each time slot. This minimizes collisions with other UEs under limited observation conditions while efficiently utilizing its own energy, thereby improving throughput for both itself and the overall system. It can be divided into three main parts: environmental state transition, distributed action selection, and distributed reward distribution. First, each UE observes its local environmental state from the overall system environment and uses this local observation as input to its neural network. Then, each UE's neural network outputs its own action—this is distributed action selection. Next, all UE actions are aggregated to obtain an action set. Rewards are distributed to each UE based on its selected action for training its neural network—this is distributed reward distribution. Finally, based on the overall action set, the UE interacts with the environment to transition to the next state—this is environmental state transition.

[0112] (VI) Experimental Results and Analysis

[0113] To demonstrate that the method of this invention can be applied to communication systems, it is compared with other methods (the comparison method is CSMA). The results show that the method of this invention can achieve better throughput than CSMA in this communication system. The specific experimental results are analyzed and summarized as follows.

[0114] 1. Comparison scheme setup

[0115] The method proposed in this invention is compared with CSMA / CA. This comparison is made because CSMA / CA excels at preventing UE collisions on the channel, and many applications and protocols still use this method. Therefore, this comparison verifies the feasibility of the proposed method. If the final experimental results are similar to CSMA / CA, it does not mean the proposed method is infeasible. In fact, the proposed method ultimately surpasses CSMA / CA, as shown below. This precisely verifies that the proposed method can also effectively prevent UE collisions on the channel. Specific experimental results are shown below.

[0116] The following describes how to configure CSMA.

[0117] At time slot t, each UE has its own contention window, which is an N×K matrix CW[t]. Initially, each element in the vector has a size of 7 or 15 (depending on the experimental settings). After a collision, the size doubles (consistent with the conventional CSMA binary backoff algorithm), with a maximum value of 511. Then, each UE randomly generates an N×K backoff matrix BO[t] based on its own contention window. The element value BO in the i-th row and j-th column of the matrix is... i，j [t] represents a backoff value randomly generated by the i-th uE in the j-th channel based on its own contention window size. The randomly generated value is 0 ≤ BO. i，j [t]≤CW i，j [t]. At the beginning of each time slot, all elements in matrix BO[t] are decremented by 1, indicating that each UE is listening to various channels. In this invention, the energy consumed by the UE listening to channels in CSMA is ignored. However, in reality, the energy consumed by the listening part of CSMA can be very large. For example, if a certain uE randomly receives a large backoff value, it will spend a lot of time listening to the channel, thus consuming a large amount of energy. When each uE reaches its own backoff vector BO... iWhen the value of [t] is 0, it indicates that the UE, after listening for a certain period of time, believes that no other UE is occupying this channel. At this time, the UE selects this channel to send information. Here, the time spent decreasing the backoff value in each CSMA slot is ignored, i.e., the time spent by the UE using the CSMA selection action is ignored, as is the energy spent by each UE listening to the channel in CSMA. Therefore, according to the above process, each UE will determine its own action for this slot. It is worth noting that in the CSMA setting of this invention, if a UE selects a certain channel to send information, other UEs will hear that this channel is occupied and will not listen to this channel again. However, if the backoff value of multiple UEs on a specific channel is 0 at the same time, after listening, multiple UEs will all believe that no other UE is occupying this channel and will select this channel to send information. In this case, information collision will still occur.

[0118] 2. Experimental Design

[0119] Since the energy collected by wireless devices through EH is limited and unpredictable, this invention limits the energy to a relatively low state to simulate the working conditions of the UE in a real environment. Our expectation is that the UE can make its own action choices based on its own state observations in a low energy state, thereby reasonably avoiding information collisions and improving its own and the system's throughput.

[0120] In the experiment, two different energy schemes were set up for comparison. First, in the first energy scheme, at the beginning of each time slot, the UE had a 0.8 probability of not receiving energy, a 0.1 probability of receiving 1 unit of energy, and a 0.1 probability of receiving 2 units of energy. Second, in the second energy scheme, the UE had a 0.4 probability of not receiving energy, a 0.3 probability of receiving 1 unit of energy, a 0.2 probability of receiving 2 units of energy, and a 0.1 probability of receiving 3 units of energy. Simultaneously, three different uE and channel numbers were set for each energy scheme: 3 UEs and 2 channels, 4 UEs and 3 channels, and 5 UEs and 4 channels.

[0121] The throughput performance of the proposed solution was verified using Python code, and compared with comparative solutions. The experiment ran for 100 episodies, each containing 1000 time slots, meaning each experiment ran approximately 100,000 time slots. After each episode, the environment was reset, a new randomized environment state was generated, and then the next episode was run. Figures 4 to 10In the graph, the horizontal axis represents the number of episodes that have been run (one episode contains 1000 time slots), and the vertical axis represents the throughput of different UEs and the total throughput of all UEs.

[0122] 1. First energy scheme

[0123] At the start of each time slot, the UE has a probability of 0.8 of not obtaining energy, a probability of 0.1 of obtaining one unit of energy, and a probability of 0.1 of obtaining two units of energy.

[0124] Figure 4 This diagram illustrates the throughput of a communication system with three wireless devices and two channels. Figure 4 In the diagram, counting from top to bottom, the first three graphs show the throughput performance of three individual UEs, while the fourth graph shows the total throughput of all UEs. As can be seen from the graphs, as the training process progresses, the throughput achieved by the method described in this invention is almost equal to that of the CSMA / CA method. It's important to note that the initial lower throughput of the DQN method is because the neural network is not yet fully trained. The true throughput achievable by the deep reinforcement learning network only becomes apparent after the neural network converges. Initially, the neural network parameters are relatively random, requiring training to modify the parameters and improve performance until the neural network converges. It's also worth noting that since the energy and time consumed by the CSMA / CA listening channel are ignored, in practical situations, the method described in this invention should outperform the CSMA / CA method.

[0125] Figure 5 This diagram illustrates the throughput of a communication system with four wireless devices and three channels. Figure 6 The diagram illustrates the throughput of a communication system with 5 wireless devices and 4 channels. Similar to the previous diagram, the first four figures from top to bottom show the throughput performance of each UE, while the last figure shows the sum of the throughput of all UEs. The two figures demonstrate that, even with a relatively idealized view of CSMA / CA, the proposed solution achieves a level almost identical to the comparative solutions.

[0126] 2. Second energy scheme

[0127] At the start of each time slot, the UE has a probability of 0.4 of not obtaining energy, a probability of 0.3 of obtaining one unit of energy, a probability of 0.2 of obtaining two units of energy, and a probability of 0.1 of obtaining three units of energy.

[0128] Figure 7The diagram illustrates the throughput of a communication system with three wireless devices and two channels. The first three graphs from top to bottom show the throughput of the three UEs, while the last graph shows the sum of the throughput of all UEs. It can be seen that, ignoring some shortcomings of the comparative scheme, the method of this invention achieves almost no difference in the sum of the throughput of all UEs compared to the comparative scheme. Therefore, in practical situations, the method of this invention can achieve better results.

[0129] Figure 8 The diagram illustrates the throughput of a communication system with four wireless devices and three channels. Although the results obtained by the two methods shown in the diagram are similar, the data shows that when using the CSMA / CA method, the average throughput of all UEs is 1.39, while the average throughput obtained by the method of this invention is 1.45. Based on this experimental result, the method of this invention is more advantageous.

[0130] Figure 9 A throughput diagram is provided for a communication system with 5 wireless devices and 4 channels. Ignoring some drawbacks of CSMA / CA, the average throughput of all UEs using CSMA / CA is 1.63, while the average throughput achieved by the method of this invention is 1.84, demonstrating that the energy scheduling of the method of this invention is better. Overall, the method of this invention can achieve greater advantages as the number of UEs and channels increases.

[0131] Figure 10 and Figure 9 The difference is Figure 10 The experimental environment increased the penalty for information collisions; that is, when multiple UEs simultaneously transmit information on the same channel, a larger penalty value (i.e., throughput) is obtained. It can be seen that after increasing the penalty, the method of this invention achieves better results than CSMA / CA. This is because, with increased penalty, DQN tries to avoid information collisions, which is also the effectiveness of this invention in reducing information collisions. Looking at the data, the average throughput of all UEs using the CSMA / CA method is 1.20, while the average throughput achieved by the method of this invention is 1.65, showing a significant difference. The difference is not so obvious in the graph because the vertical axis has a larger range.

[0132] 3. Summary and Analysis of Experimental Results

[0133] In summary, all experimental results show that the method of this invention achieves throughput performance on par with, or even better than, the comparative experimental scheme (CSMA / CA) in energy-scarce environments. Note that this performance ignores the time and energy consumed by CSMA in monitoring the channel. In real-world environments, if this energy consumption is taken into account, the method of this invention can achieve better results than CSMA. Furthermore, the experimental results show that the method of this invention has a greater advantage when there are more UEs and channels, and when energy is scarce but not excessively tight. Especially when energy is slightly more flexible, the method of this invention surpasses CSMA / CA with a significant improvement, which reflects the energy scheduling aspect of the method. In additional experiments, when the collision penalty is increased, the method of this invention also performs well in reducing collisions. Therefore, by comparing the method of this invention with CSMA / CA, it can be seen that the proposed method can be applied to this communication system and achieve excellent throughput performance. It achieves a superior distributed access and energy scheduling strategy even when each UE only observes a portion of the environmental state and there is no communication between UEs.

[0134] Based on this invention, by adding or replacing some other environmental factors, such as other channel state settings, and then comparing them with traditional solutions using other deep learning methods or the method of this invention, some results can also be achieved.

[0135] Through the above description of the embodiments, those skilled in the art can clearly understand that this disclosure can be implemented using software plus necessary general-purpose hardware, or it can be implemented using dedicated hardware including dedicated integrated circuits, dedicated CPUs, dedicated memory, dedicated components, etc. Generally, any function performed by a computer program can be easily implemented using corresponding hardware, and the specific hardware structure used to implement the same function can be diverse, such as analog circuits, digital circuits, or dedicated circuits. However, for this disclosure, software implementation is more often a preferred implementation method.

[0136] Although embodiments of the present invention have been described above in conjunction with the accompanying drawings, the present invention is not limited to the specific embodiments and application fields described above. The specific embodiments described above are merely illustrative and instructive, and not restrictive. Those skilled in the art can make many other forms based on the guidance of this specification and without departing from the scope of protection of the claims of the present invention, and all of these are within the scope of protection of the present invention.

Claims

1. A channel access and energy scheduling method based on deep reinforcement learning, characterized in that: The user equipment module obtains the first T1 timeslot information of user equipment UE i, the status of whether the transmission was successful or not on each channel, and the remaining battery capacity of user equipment UE i as input information. The input information is then fed into the pre-trained deep reinforcement learning network corresponding to user equipment UE i. The deep reinforcement learning network outputs the optimal action of user equipment UE i, i=1,2,…,N, where N is the total number of user equipment UEs, which is greater than or equal to the number of channels. in: T1 is the set value; The action is either that the user equipment UE i chooses to consume energy to send information to the access point at time slot t, or that the user equipment UE chooses to suspend and not send information at time slot t. The implementation of deep reinforcement learning networks includes the following steps: S1. Integrate the input information of user equipment UE i obtained by the user equipment module into the current state. i = 1, 2, ..., N, where N is the total number of user equipment (UEs); S2, Adopt Algorithm selects action , i=1,2,…,N; S3, User Equipment (UE) i performs the selected action. , i=1,2,…,N; S4. Actions based on each User Equipment (UE) Obtain the action set ; S5, based on action sets A 1×N vector R[t] is obtained to represent the reward vector, where each value in the reward vector is the reward value obtained by the corresponding UE, and the environment state S[t] is transitioned to the next state S[t+1]. ； ； ； S6, The user equipment module will send the UE i corresponding to Save and update the deep reinforcement learning network.

2. The method according to claim 1, characterized in that, The remaining battery capacity of user equipment UE i satisfies the energy model; In the energy model, user equipment (UE) i collects energy and stores it in its corresponding battery. The energy collection process satisfies the Markov property. If the battery capacity reaches its upper limit... If the excess energy is wasted, the user equipment (UE) i will consume energy by sending information to the access point. The battery evolution equation for user equipment (UE) i from time slot t to t+1 is: The remaining battery capacity of user equipment (UE) at time slot t+1. The remaining battery capacity of user equipment (UE) at time slot t-1. This represents the energy consumption of the user equipment (UE) when transmitting information in time slot t. Let be the energy collected by the i-th UE at time slot t.

3. The method according to claim 1, characterized in that, User equipment (UE) i's first T1 timeslot information is used to determine the success or failure of each channel information transmission, employing a 1×(T1×K) vector. The record contains K, which is the number of channels. Starting from the first element, each K element represents the status of whether the information in a time slot was successfully transmitted. The element value is the corresponding channel status, which is the amount of information that can be transmitted in the channel.

4. The method according to claim 3, characterized in that, Information transmission satisfies the channel model. In the channel model, if more than one user equipment (UE) simultaneously selects the same channel to send information, information collision will occur on this channel. All UEs that select this channel to send information will fail to send, and the access point (AP) will not receive the information.

5. The method according to claim 1, characterized in that, The reward value is set as follows: If user equipment (UE) i chooses not to send information, it receives a reward value of 0. If user equipment UE i selects channel j to send information, and there is another user equipment UE sending information through channel j, then user equipment UE i receives a reward value of -1. If user equipment UE i selects channel j to send information, and other user equipment UEs do not select this channel to send information, the reward value obtained by the user equipment UE is the channel state of the channel selected by this user equipment UE.

6. The method according to claim 1, characterized in that, Step S6 includes the following steps: S61, User Equipment Module will The experience is stored in the experience pool corresponding to each user equipment (UE). If the number of experience samples stored in the experience pool meets the requirements of D groups, then step S62 is executed; otherwise, step S1 is executed. Here, D represents the capacity of the experience pool. S62. Based on the set number of samples, randomly sample from the experience pool, calculate the target value target[i], and update the estimation neural network DQN1[i]. S63. Update the target neural network DQN2[i] for each time interval T1.

7. The method according to claim 6, characterized in that, The target value target[i] is calculated as follows: If the next time slot is the terminating time slot T2, then target[i] = ; otherwise: target[i]= In the formula: As a discount factor, For user equipment (UE) i in time slot t, the target neural network DQN2[i] is used to set the state. The Q value corresponding to taking the optimal action.

8. The method according to claim 7, characterized in that, Deep reinforcement learning networks are updated using gradient descent, and the loss is calculated using the following loss function: In the formula: For user equipment (UE) i in time slot t, the estimated neural network DQN1[i] is used to estimate the state. The Q value corresponding to taking the optimal action.

9. A computer-readable storage device, characterized in that: The device is a storage medium storing a computer program that can be loaded by a processor and executed according to any one of claims 1 to 8.

Images