A high-density internet of things channel access method based on deep reinforcement learning

Abstract

The application relates to the field of wireless communication, in particular to a high-density Internet of Things channel access method based on deep reinforcement learning, which comprises the following steps: initializing a communication network when a new node joins the communication network; after each initialization, carrying out data transmission according to a superframe period, wherein the superframe period comprises a data transmission period and a control period, in the control period, a node listens to an access contention window and a maximum backoff index broadcast by a sink node, and then the maximum backoff times are determined, in the data transmission period, the node carries out channel access according to channel access parameters broadcast in the control period in the last superframe period; after the data transmission period ends, the sink node selects an access parameter corresponding to the maximum action value according to a Q-learning algorithm as an access parameter of the next superframe period; the application avoids the generation of a large number of collisions in a high-density network environment, improves network throughput, and reduces a packet loss rate.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of wireless communication, in particular to a high-density Internet of Things channel access method based on deep reinforcement learning. BACKGROUND

[0002] With the continuous in-depth research and development of wireless sensor networks (WSNs), it has penetrated into various Internet of Things fields and is an important part of the Internet of Things. This is also due to the improvement of IEEE 802.15.4 protocol, which is a low-cost, widely deployed, low-energy, and highly reliable solution for WSNs. In the IEEE 802.15.4 protocol, the carrier sense multiple access with collision avoidance (CSMA / CA) mechanism based on competition is widely used to realize channel access, and its execution efficiency affects the performance of the entire wireless personal area network. Therefore, in order to improve the overall network throughput and more reasonably adjust the channel access opportunities of each node, research on channel access and communication algorithms is a hot topic.

[0003] In recent years, channel access schemes involve either the CSMA / CA mechanism based on contention-based channel access or the TSCH technology based on time slot division channel allocation. TSCH is not suitable for network scenarios with too many nodes due to the communication overhead caused by time slot allocation, while the CSMA / CA mechanism is not flexible enough in dealing with channel access in high-density or specific scenarios. Currently, many improvements based on this mechanism have been proposed. With the development of artificial intelligence, it is possible to combine intelligent algorithms with traditional network mechanisms. The present application combines reinforcement learning with the traditional CSMA / CA mechanism and dynamically adjusts the corresponding parameters of the CSMA / CA mechanism according to the network throughput, so that the CSMA / CA mechanism can adapt to high-density network environments and achieve low-collision channel access and network communication, which is friendly to the deployment of dense wireless networks.

[0004] Currently, the channel access strategy of wireless sensor network mainly includes two types: channel access scheme based on competition and channel allocation scheme based on time slot division. The channel access strategy is determined by the medium access control (MAC) layer in the network protocol stack, and we mainly consider the channel access strategy in the IEEE 802.15.4 standard. When the number of network nodes is moderate, both channel access strategies can achieve good results, but if the nodes are in a high-density state within a certain range, the channel allocation based on time slot division will bring huge communication overhead, and the process of maintaining control information between nodes will become very cumbersome. While the traditional channel access scheme based on competition will not have difficulty in implementation with the increase of the number of nodes, but due to the lack of adjustment to the network status in high-density environment, the network performance such as throughput will be greatly reduced. SUMMARY

[0005] Due to the shortcomings of the traditional wireless sensor network channel access strategy, the application proposes a high-density Internet of Things channel access method based on deep reinforcement learning, which specifically includes the following steps:

[0006] When a new node joins the communication network, the communication network is initialized;

[0007] After each initialization is completed, data transmission is performed according to the superframe period. In a superframe period, it includes a data transmission period and a control period. In the control period, the node listens to the access contention window, the maximum backoff index and the maximum backoff times broadcast by the sink node. In the data transmission period, the node performs channel access according to the channel access parameters broadcast in the control period of the last superframe period;

[0008] After the data transmission period ends, the sink node selects the access parameter corresponding to the maximum action value as the access parameter of the next superframe period according to the Q-learning algorithm.

[0009] Further, when a new node joins the communication network, the initialization process of the communication network includes the following steps:

[0010] The initialization process is divided into an initial period and an initial data transmission period;

[0011] The initial data transmission period includes multiple data transmission slots, and each data transmission slot performs data transmission according to a fixed contention window, backoff index and maximum backoff times;

[0012] According to the data transmission of each time slot, the average values of the delay, packet loss rate and throughput in the initial data transmission period are calculated.

[0013] Further, the values of the contention window, backoff exponent and maximum backoff number in each time slot of the initial data transmission period are set to the values obtained by the last reinforcement learning, and if no reinforcement learning has been performed before, the values of the contention window, backoff exponent and maximum backoff number are all set to 2.

[0014] Further, the process of calculating the average values of the delay, packet loss rate and throughput in an initial data transmission period comprises the following steps:

[0015] delay 平均 (t+1) = k (|delay(t+1) - delay(t) |) + (1-k)delay 平均 (t)

[0016] throughput 平均 (t+1) = k (|throughput(t+1) - throughput(t) |) + (1-k)throughput 平均 (t)

[0017] packetloss 平均 (t+1) = k (|packetloss(t+1) - packetloss(t) |) + (1-k)throughput 平均 (t)

[0018] wherein, delay 平均 (t) is the average value of the delay in the tth data transmission time slot; throughput 平均 (t) is the average value of the throughput in the tth data transmission time slot; packetloss 平均 (t) is the average value of the packet loss rate in the tth data transmission time slot; k is a weight factor; delay(t) is the network delay in the tth data transmission time slot; throughput(t) is the network throughput in the tth data transmission time slot; packetloss(t) is the packet loss rate in the tth data transmission time slot; and || represents the absolute value.

[0019] Further, the action value function in the Q-learning algorithm is represented as:

[0020]

[0021] wherein, q(s t ,a t ) represents the value of action a t in state s t ; γ is a discount factor, α is a learning rate; and r t is a reward value. obtaining a next state with a maximum action value function.

[0022] Further, the reward value r t The obtaining process comprises:

[0023] When the current time slot has packet loss rate rising, time delay rising and throughput decreasing compared with the previous time slot, the reward value is -3;

[0024] When the current time slot has two of the three conditions of packet loss rate rising, time delay rising and throughput decreasing compared with the previous time slot, and the reward function value does not rise compared with the previous time slot, the reward value is -2;

[0025] When the current time slot has one of the three conditions of packet loss rate rising, time delay rising and throughput decreasing compared with the previous time slot, and the reward function value does not rise compared with the previous time slot, the reward value is -1;

[0026] When the current time slot has one of the three conditions of packet loss rate rising, time delay rising and throughput decreasing compared with the previous time slot, and the reward function value rises compared with the previous time slot, the reward value is 0;

[0027] When the current time slot has one of the three conditions of packet loss rate rising, time delay rising and throughput decreasing compared with the previous time slot, and the reward function R' rises, the reward value is 1;

[0028] When the current time slot has none of the three conditions of packet loss rate rising, time delay rising and throughput decreasing compared with the previous time slot, and the reward function value rises compared with the previous time slot, the reward value is 2.

[0029] Further, the reward function R' is expressed as:

[0030]

[0031] wherein R' is the reward function of the current time slot, which is calculated according to the data of the current time slot and the data of the previous time slot; m, u and n are weight factors, 0≤m≤1, 0≤u≤1, 0≤n≤1 and u+n+m=1; delay'(t) is the comparative network time delay of the tth data transmission time slot; delay(t) is the network delay of the tth data transmission time slot; delay 平均 is the average value of the network delay obtained in the whole initial transmission process; throughput(t) is the throughput of the tth data transmission time slot; throughput'(t) is the comparative throughput of the tth data transmission time slot; throughput 平均packetloss(t) is the average value of the final throughput obtained by completing the entire initial transmission process; packetloss'(t) is the comparative packet loss rate of the tth data transmission time slot; packetloss(t) is the packet loss rate of the tth data transmission time slot; packetloss 平均 packetloss(t) is the average value of the final packet loss rate obtained by completing the entire initial transmission process.

[0032] Further, in the process of selecting the access parameter corresponding to the maximum action value as the access parameter of the next superframe period according to the Q-learning algorithm, the sink node sets the maximum value and the minimum value for the access parameter, and if the value of the access parameter after the action is executed exceeds the set range, the value after the action is executed is consistent with the value at the previous moment, which is represented as:

[0033]

[0034] Wherein, x t+1 represents the value of an access parameter x at t+1 moment, x min is the minimum value of the access parameter x; x max is the maximum value of the access parameter x; x∈{CW, macMaxBE, macMaxCSMABackoffs}, CW represents the contention window length, macMaxBE represents the maximum backoff index, and macMaxCSMABackoffs represents the maximum backoff number.

[0035] Compared with the fixed contention window, the maximum backoff index and the maximum backoff number used in the channel access of the conventional CSMA / CA mechanism based on competition, the application avoids the generation of a large number of collisions in a high-density network environment, and compared with the scheme in the prior art which only considers a single network physical quantity, the application improves the network throughput while reducing the packet loss rate. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 It is a reinforcement learning modeling in the application of a high-density Internet of Things channel access method based on deep reinforcement learning;

[0037] Figure 2 It is a Markov decision process used in the application of a high-density Internet of Things channel access method based on deep reinforcement learning;

[0038] Figure 3 It is a superframe structure used in the application of a high-density Internet of Things channel access method based on deep reinforcement learning;

[0039] Figure 4 It is a flowchart of the application of a high-density Internet of Things channel access method based on deep reinforcement learning;

[0040] Figure 5The application discloses a multi-hop network topology in a high-density Internet of Things channel access method based on deep reinforcement learning. DETAILED DESCRIPTION

[0041] The technical solutions in the embodiments of the application will be clearly and completely described below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, rather than all the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by a person of ordinary skill in the art without creative work are within the protection scope of the application.

[0042] The application discloses a high-density Internet of Things channel access method based on deep reinforcement learning

[0043] When a new node joins the communication network, the communication network is initialized;

[0044] After each initialization is completed, data transmission is performed according to a superframe period, and in one superframe period, a data transmission period and a control period are included; in the control period, a node listens to an access contention window, a maximum backoff index and a maximum backoff number broadcast by a sink node; and in the data transmission period, the node performs channel access according to channel access parameters broadcast in the control period in a previous superframe period.

[0045] After the data transmission period ends, the sink node selects an access parameter corresponding to a maximum action value according to a Q-learning algorithm as an access parameter of a next superframe period.

[0046] The traditional competition-based CSMA / CA mechanism adopts a fixed contention window, a maximum backoff index and a maximum backoff number in channel access, which can cause a large number of collisions in a high-density network environment. Some similar schemes propose to modify the size of the contention window through reinforcement learning, and mainly aim at a single-hop network such as wifi. However, since a multi-hop network needs to consider not only the global throughput of the network, but also the amount of data received by the root node, that is, the amount of effective data processed, only modifying the contention window is far from enough, and can even cause the global throughput to rise but the packet loss rate to be too high, and finally the amount of effective data processed by the root node to be low.

[0047] In the embodiment, the reinforcement learning is used to improve the traditional backoff mechanism, for example, Figure 5In the multi-network, the sink node as the root node manages multiple sensor nodes as child nodes and routing nodes, the sink node in the network timely informs the whole network of the most reasonable channel access related parameters as the intelligent agent, which are applied when the nodes compete for the channel, and the collision rate of the whole network is reduced. The method models the adjustment of the contention window, the maximum backoff index and the maximum backoff number in the channel access as a Markov decision process, and realizes the improvement of the system performance through the reinforcement learning algorithm.

[0048] Specifically, the reinforcement learning model constructed in the embodiment is as shown in the figure Figure 1 In the model, the intelligent agent is in a state s t selects an action a t In the embodiment, the action selected is one of increasing, decreasing or keeping unchanged one of the parameters of the current contention window, the maximum backoff index or the maximum backoff number, and then the next state s t+1 and the reward value r t+1 of the next state are obtained under the action of the environment, after which the state and the reward value are updated for the selection of the action in the next round until the iteration ends.

[0049] The adjustment process of the parameters in the embodiment is described as a Markov decision process, which is composed of a state space S, an action space A, a state transition probability P and a reward R, and is represented as MDP=(S,A,P,R), wherein:

[0050] The state space S is represented as S={s1,s2,...,s t ,...,s m}, s t represents the maximum backoff number, the contention window and the maximum BE value after n*t (n represents the time slots required to collect each reward) time slots, that is, the state space S is the combination of the selectable contention window, the maximum BE value and the maximum backoff number, s t ∈[CW,macMaxBE,macMaxCSMABackoffs].

[0051] The action space A is represented as A={a0,a1,...,a t ,...,a n}, a t represents the adjustment mode corresponding to the value of the maximum backoff number, the contention window and the maximum backoff index selected in the state s t a t ∈{1,2,3,4,5,6,7}, which is specifically represented as:

[0052]

[0053] The maximum backoff times, the contention window and the value of the maximum backoff index are also changed within a certain range, such as the selection interval of the contention window size is [1, 2, 3, 4, 5, 6], if the current contention window is 1 and the selected action is also 1, or the current contention window is 6 and the selected action is 2, when it is judged that the modification is not within the preset interval, the state remains unchanged, that is, not only the action 7 is adopted to make the state unchanged.

[0054] The state transition probability represents the probability of the current state s t The next action a t is performed t+1 , and then the state s t+1 is transferred to another state s t . t The probability distribution of the state s t after the action a t is performed in the current state s t+1 is denoted as p(s t |s t , a ).

[0055] In the embodiment, the reward R is determined by the reward function, and the changes of the delay, the throughput and the packet loss rate of two adjacent time slots, specifically, the reward value is determined according to the following rules:

[0056] When the packet loss rate, the delay and the throughput of the current time slot increase compared with the previous time slot, the reward value is -3;

[0057] When two of the three conditions that the packet loss rate, the delay and the throughput of the current time slot increase compared with the previous time slot are met, and the reward function value of the current time slot increases compared with the previous time slot, the reward value is -2;

[0058] When one of the three conditions that the packet loss rate, the delay and the throughput of the current time slot increase compared with the previous time slot are met, and the reward function value of the current time slot does not increase compared with the previous time slot, the reward value is -1;

[0059] When one of the three conditions that the packet loss rate, the delay and the throughput of the current time slot increase compared with the previous time slot are met, and the reward function value of the current time slot does not increase compared with the previous time slot, the reward value is 0;

[0060] When one of the three conditions that the packet loss rate, the delay and the throughput of the current time slot increase compared with the previous time slot is met, and the reward function R' increases, the reward value is 1;

[0061] When the current time slot has no packet loss rate rise, time delay rise and throughput drop compared with the previous time slot, and the reward function value rises compared with the previous time slot, the reward value is 2.

[0062] In the embodiment, the reward function R' is expressed as:

[0063]

[0064] Wherein, m, u and n are weight factors, 0≤m≤1, 0≤u≤1, 0≤n≤1 and u+n+m=1; for the convenience of understanding, the three weight factors are set to be the same in the embodiment; delay'(t) is the comparative network time delay of the tth data transmission time slot; delay(t) is the network delay of the tth data transmission time slot; delay 平均 is the average value of the network delay finally obtained in the whole initial transmission process; throughput(t) is the throughput of the tth data transmission time slot; throughput'(t) is the comparative throughput of the tth data transmission time slot; throughput 平均 is the average value of the throughput finally obtained in the whole initial transmission process; packetloss'(t) is the comparative packet loss rate of the tth data transmission time slot; packetloss(t) is the packet loss rate of the tth data transmission time slot; packetloss 平均 is the average value of the packet loss rate finally obtained in the whole initial transmission process.

[0065] The algorithm of the setting process of the reward R is as follows:

[0066]

[0067]

[0068] Wherein, R d =delay'-delay, delay represents the network delay of the current time slot, and delay' represents the comparative time delay, which is updated after the reward value is calculated each time, and when the reward value, i.e. the value of R', is a positive value, the comparative time delay delay' of the current time slot is updated to the network delay of the current time slot, otherwise no update is performed; R t =throughput-throughput', throughput represents the throughput of the current time slot, and throughput' represents the comparative throughput, which is updated after the reward value is calculated each time, and when the reward value, i.e. the value of R', is a positive value, the comparative throughput delay' of the current time slot is updated to the network delay of the current time slot, otherwise no update is performed; R ppacketloss' represents the contrast packet loss rate, and packetloss represents the packet loss rate of the current time slot. The network delay is the same. After the reward value is calculated each time, the contrast packet loss rate is updated. When the value of the reward value, i.e., R', is positive, the contrast packet loss rate packetloss' of the current time slot is updated as the network delay of the current time slot, otherwise, no update is performed.

[0069] Figure 2 is a schematic diagram of the Markov decision process of the embodiment. The selection range of the maximum backoff number and the contention window size is [1, 2, 3, 4, 5, 6], and the maximum backoff index range is [2, 3, 4, 5, 6, 7]. Of course, this range can be adjusted according to actual conditions. The range set here is mainly for a high-density network environment with 200-500 nodes, so there are a total of 216 states (6x6x6).

[0070] In order to facilitate all nodes to modify the contention window, the maximum backoff index and the maximum backoff number, and to set the reward, a wireless sensor network superframe structure based on the RPL (IPv6 Routing Protocol for Low-Power and Lossy Networks) protocol is designed. The communication period is divided into an initialization period and a superframe period. When the node starts the initialization period, the sink node starts the superframe period after completing the initialization period within a certain time. After the other nodes complete the networking in the initialization period, the starting time of the superframe period can be estimated according to the parent node to complete synchronization.

[0071] The superframe structure is shown in Figure 3 A superframe period includes a data transmission period and a control period. In the control period, the node listens to the access contention window, the maximum backoff index and the maximum backoff number broadcast by the sink node. In the data transmission period, the node performs channel access according to the channel access parameters broadcast in the control period of the previous superframe period. When a new node joins the communication network, the initialization process of the communication network includes the following steps:

[0072] The initialization process is divided into an initial period and an initial data transmission period.

[0073] The initial data transmission period includes multiple data transmission time slots. Each data transmission time slot performs data transmission according to the fixed contention window, backoff index and maximum backoff number.

[0074] The average values of the delay, packet loss rate and throughput in the initial data transmission period are calculated according to the data transmission of each time slot.

[0075] The average values of the delay, the packet loss rate and the throughput corresponding to the initial data transmission period after the new node is added are taken as the values of the delay, the packet loss rate and the throughput of the first time slot of the first superframe period after the new node is added, that is, the control period of the first superframe period after the new node is added is calculated according to the average values of the delay, the packet loss rate and the throughput corresponding to the initial data transmission period.

[0076] As Figure 3 , an initial data transmission period contains multiple data transmission time slots, [2, 2, 2] is selected as the initial values of the contention window CW, the maximum backoff index MaxBE and the maximum backoff number MaxCSMA Backoffs in this period, and then the average change amount of the packet loss rate and the throughput and the delay in this period is calculated as the reward evaluation basis for subsequent reinforcement learning training. After each new node is added, the new evaluation basis is recalculated after the initial period to facilitate subsequent training. If it is the second round into the initial period, the contention window CW, the maximum backoff index MaxBE and the maximum backoff number MaxCSMA Backoffs are the combination obtained by the last reinforcement training, which is to confirm whether the previous training result is still applicable in the new network state.

[0077] The calculation of the average values of the delay, the packet loss rate and the throughput in the initial data transmission period includes the following steps:

[0078] delay 平均 (t+1)=k(|delay(t+1)-delay(t))+(1-k)delay 平均 (t)

[0079] throughput 平均 (t+1)=k(|throughput(t+1)-throughput(t)|)+(1-k)throughput 平均 (t)

[0080] packetloss 平均 (t+1)=k(|packetloss(t+1)--packetloss(t)|)+(1-k)throughput 平均 (t)

[0081] Wherein, delayaverage(t) is the average value of the delay of the tth data transmission time slot; throughputaverage(t) is the average value of the throughput of the tth data transmission time slot; packetlossaverage(t) is the average value of the packet loss rate of the tth data transmission time slot; k is a weight factor; delay(t) is the network delay of the tth data transmission time slot; throughput(t) is the throughput of the network of the tth data transmission time slot; packetloss(t) is the packet loss rate of the tth data transmission time slot; || represents taking an absolute value.

[0082] The superframe period includes a data transmission period and a control period. In the data transmission period, each node listens to the uplink transmission working channel and uses single channel or multi-channel for data packet transmission. In the control period, each node listens to the control channel, not only receives the control information broadcasted by the sink node to ensure the construction and maintenance of the network topology, but also receives the CW and other parameters broadcasted by the sink node.

[0083] The channel access process uses the uniform contention window, the maximum backoff index and the maximum backoff times broadcasted by the sink node in the control period. Then the node which needs to transmit data listens to the channel after randomly backoff for multiple time slots according to the backoff index in the transmission period. When the channel is idle within the contention window period, the node obtains the channel access opportunity. If the channel is busy within the contention window period, the node modifies the backoff index and listens to the channel again after re-backoff. If the backoff times are greater than the maximum backoff times, it means that the channel access fails, and the current data packet to be transmitted is discarded. The algorithm of the channel access part is shown in the following figure:

[0084]

[0085]

[0086] The learning process, after an action adjustment period, the sink node calculates the reward in the current state according to the network throughput in the last period, updates the value function of the reinforcement learning model, and then uses the ε-greedy strategy to reselect a combination of the contention window, the maximum backoff index and the maximum backoff times to broadcast to other nodes. After an action adjustment period, the above process is repeatedly executed to optimize the three parameters of the contention window, the maximum backoff index and the maximum backoff times.

[0087] The embodiment adopts an epsilon-greedy strategy for selecting the optimal action. Specifically, the action with the maximum q value is selected as the optimal choice. If there are multiple actions with the maximum q value, such as when the q table values are all zero at the beginning, a random optimal action is selected from among them. In addition, the epsilon-greedy strategy also adopts an exploration mechanism to avoid the algorithm falling into a local optimum. This exploration occurs with a certain probability, otherwise, endless exploration will sacrifice too much short-term benefit, thereby affecting the overall performance of the network. The sink node acts as an agent and executes the epsilon-greedy strategy, which can select the optimal action in the following way:

[0088]

[0089] wherein a t is divided into two types: development and exploration. argmax a Q(s t ,a) represents the action with the maximum q value in the current state s t , which is selected with a probability of 1-epsilon; a t ∈A represents a randomly selected action belonging to the action space A, which is selected with a probability of epsilon.

[0090] The system adopts the value function updating method of the Q-learning algorithm, and the calculation formula of the action value function is as follows:

[0091]

[0092] wherein gamma is a discount factor, alpha is a learning rate, is the maximum action value of the next state, and the application can also adopt other reinforcement learning algorithms, such as a deep Q network.

[0093] The method can enable the wireless sensor network to achieve dynamic channel access in a high-density environment and cause fewer collisions. The application gives a preferred embodiment as shown in Figure 4 , which includes the following steps:

[0094] 101. Model the channel access problem as a Markov decision process;

[0095] 102. The sink node acts as an agent and selects the optimal action a using the epsilon-greedy strategy in the state S;

[0096] 103. In the control period of the superframe structure, the sink node broadcasts the contention window CW, the maximum backoff index MaxBE, and the maximum backoff number MaxCSMA Backoffs;

[0097] 104. In the data transmission period, the nodes compete for the channel according to the broadcasted channel access parameters;

[0098] 105. If a channel access opportunity is obtained within the maximum number of backoffs, data transmission is performed, otherwise the current data packet is dropped;

[0099] 106. The reward is calculated according to the network throughput, latency and packet loss rate of the channel contention process;

[0100] 107. The action value function is updated according to the reward;

[0101] 108. If the wireless network stops, the process ends, otherwise, return to step 103 to broadcast the contention window CW, the maximum backoff index MaxBE and the maximum number of backoffs MaxCSMA Backoffs corresponding to the action selected in step 107.

[0102] While the embodiments of the application have been illustrated and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made therein without departing from the spirit and scope of the application, which is defined by the appended claims and their equivalents.

Claims

1. A high dense Internet of Things channel access method based on deep reinforcement learning, characterized in that, The method comprises the following steps: The communication network is initialized when a new node joins the communication network, and the initialization process comprises the following steps: The initialization process is divided into an initial period and an initial data transmission period; The initial data transmission period comprises a plurality of data transmission time slots, and each data transmission time slot performs data transmission according to a fixed contention window, backoff index and maximum backoff number; The average values of the delay, packet loss rate and throughput in the initial data transmission period are calculated according to the data transmission conditions of each time slot; After each initialization, data transmission is performed according to a superframe period, and one superframe period comprises a data transmission period and a control period. In the control period, the nodes listen to the access contention window, maximum backoff index and maximum backoff number broadcast by the sink node, and in the data transmission period, the nodes perform channel access according to the channel access parameters broadcast in the control period of the previous superframe period. After the data transmission period ends, the sink node selects the access parameters corresponding to the maximum action value according to the Q-learning algorithm as the access parameters of the next superframe period.

2. The method of claim 1, wherein, In the initial data transmission period, the values of the contention window, backoff index and maximum backoff number are the values obtained through the last reinforcement learning. If reinforcement learning has not been performed before, the values of the contention window, backoff index and maximum backoff number are all set to 2. 3.The high dense Internet of Things channel access method based on deep reinforcement learning of claim 1, wherein, The process of calculating the average values of the delay, packet loss rate and throughput in one initial data transmission period comprises the following steps: delay 平均 (t+1) = k(|delay(t+1) - delay(t)) + (1 - k)delay 平均 (t) throughput 平均 (t+1) = k (| throughput (t+1) - throughput (t) |) + (1 - k) throughput (t) 平均 (t) packetloss 平均 (t+1) = k (|packetloss(t+1) - packetloss(t) |) + (1 - k) throughput 平均 (t) wherein delay 平均 (t) is the average value of the delay of the tth data transmission time slot; throughput 平均 (t) is the average value of the throughput of the tth data transmission time slot; packetloss 平均 (t) is the average value of the packet loss rate of the tth data transmission time slot; k is a weighting factor; delay(t) is the network delay of the tth data transmission time slot; throughput(t) is the throughput of the network of the tth data transmission time slot; packetloss(t) is the packet loss rate of the tth data transmission time slot; || denotes taking the absolute value.

4. The method of claim 1, wherein, The action value function update process in the Q-learning algorithm is represented as: where q(s t ,a t ) represents the value of taking action a t in state s t ; γ is a discount factor, α is a learning rate; r t is a reward value; Obtain the next state with the maximum action value function.

5. The method of claim 1, wherein, In the process of selecting the access parameters corresponding to the maximum action value according to the Q-learning algorithm as the access parameters of the next superframe period, the maximum value and minimum value of the access parameters are set. If the value of the access parameter after the action is performed exceeds the set range, the value after the action is performed is consistent with the value at the previous moment, which is represented as: wherein x t+1 denotes the value of an access parameter x at time t + 1, x min is the minimum value of the access parameter x; x max is the maximum value of the access parameter x; x e {CW, macMaxBE, macMaxCSMABackoffs}, CW denotes the contention window length, macMaxBE denotes the maximum backoff exponent, and macMaxCSMABackoffs denotes the maximum number of backoffs.

Images