An intelligent congestion control method for the Internet of Things based on deep reinforcement learning

Abstract

The present invention belongs to the field of computer technology, and specifically relates to an intelligent congestion control method for the Internet of Things based on deep reinforcement learning, which constructs an Internet of Things system; a terminal node sends a data packet to a cluster head node; the cluster head node sets a buffer queue according to the data packet sent by the terminal node, and obtains current cluster head node status information; the congestion status of the cluster head node is determined according to the current cluster head node status information, and the congestion status is sent to a relay node; the relay node obtains status information of each node in the Internet of Things according to the congestion status, and sends the status information to a sink node; the sink node optimizes the status information of the Internet of Things using a DQN algorithm, obtains a congestion control strategy, and allocates queue tasks in the cluster head node to complete congestion control; the present invention introduces a feedback recovery mechanism, fully considers the feedback loss and delay problems caused by channel quality, saves bandwidth resources and reduces data packet delivery delay.

Description

Technical Field

[0001] The present invention belongs to the field of computer technology, and in particular relates to an Internet of Things intelligent congestion control method based on deep reinforcement learning. Background Art

[0002] The Internet of Things (IoT) is a system consisting of interconnected computing devices, machinery, digital devices, and objects with unique identifiers. The IoT connects everyday devices such as vehicles, watches, and air conditioners to the internet and collects sensor data from these devices without requiring any human or computer interaction. However, the massive amounts of data generated by the IoT accumulate rapidly, increasing the demand for networks with higher throughput, reliability, and predictability. Congestion management in the IoT is one of the most challenging tasks in improving network service quality. IoT communication network technology standards include: ad-hoc networks, communication network architectures, and IP technologies for low-power networks for smart objects.

[0003] In resource-constrained IoT applications, IoT devices are subject to limitations in battery power, computing power, storage, and bandwidth. The primary task of IoT devices is to collect data within their monitoring range. Terminal devices use cluster heads to transmit this information to the next node or directly to sink nodes, also known as sink nodes. Network congestion typically occurs when the upper capacity limit of network resources falls short of the network's demand for these resources. Furthermore, in the IoT, sudden bursts of data traffic, overflows of node buffer queues, and data collisions during node data transmission can all contribute to network congestion. Network congestion can lead to increased packet loss, wasted network bandwidth, increased network latency, and even, if congestion worsens, system crashes. Summary of the Invention

[0004] In order to solve the problems existing in the above-mentioned prior art, the present invention proposes an intelligent congestion control method for the Internet of Things based on deep reinforcement learning, which comprises: constructing an Internet of Things system, which comprises a terminal node, a cluster head node, a relay node and a sink node; the terminal node sends a data packet to the cluster head node; the cluster head node sets a buffer queue according to the data packet sent by the terminal node, and obtains the current cluster head node status information; the congestion status of the cluster head node is determined according to the current cluster head node status information, and the congestion status is sent to the relay node; the relay node obtains the status information of each node in the Internet of Things according to the congestion status, and sends the status information to the sink node; the sink node optimizes the status information of the Internet of Things using an improved DQN algorithm to obtain a congestion control strategy, and allocates queue tasks in the cluster head node according to the congestion control strategy to complete congestion control.

[0005] Beneficial effects of the present invention:

[0006] The present invention uses the node buffer queue length, the upper node buffer queue length, the number of peer nodes and the node packet sending rate as congestion indicators. The number of peer nodes can well help nodes to select backoff values, and the increase or decrease in the number of peer nodes can well help intelligent agents to adapt to bursts of network traffic. The present invention uses implicit congestion notification, uses ACK feedback to carry congestion information, and finally adopts the change of node packet sending rate to control network congestion. In response to the problem of ACK feedback loss, the present invention introduces a feedback recovery mechanism, fully considers the feedback loss and delay problems caused by channel quality, saves bandwidth resources and reduces data packet delivery delay. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 This is a structural diagram of the Internet of Things of the present invention;

[0008] Figure 2 This is a flow chart of the channel access of the non-slotted CSMA / CA algorithm of the present invention;

[0009] Figure 3 Schematic diagram of reinforcement learning of the present invention;

[0010] Figure 4 This is a diagram of the DRL-JMCC mechanism architecture of the present invention;

[0011] Figure 5 This is a deep data network structure diagram of the DRL-JMCC mechanism of the present invention;

[0012] Figure 6 Schematic diagram of the ACK feedback recovery mechanism of the present invention;

[0013] Figure 7 This is a flow chart of the DRL-JMCC algorithm of the present invention. DETAILED DESCRIPTION

[0014] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0015] A deep reinforcement learning-based IoT structure diagram, such as Figure 1 As shown in the figure, the terminal device node can be called a downlink node, and the cluster head node is an uplink node. Nodes need to access the channel before transmitting data, and the terminal nodes usually share the wireless channel. Channel sharing will cause channel competition, which will result in low channel utilization and thus low data delivery rate. In addition, since WSNs support reliable transmission, that is, each round of data packet transmission will rely on the acknowledgment (ACK) mechanism to confirm whether the transmission is successful, the delay or loss of ACK and its late arrival will lead to a decrease in network throughput. In addition, network resources are overloaded, that is, the rate at which data packets reach the node is greater than the rate at which data packets are sent, which will cause the packet dequeue rate of the node buffer queue to be lower than the queue entry rate, thereby causing the node buffer queue to overflow. As shown in the figure, Figure 1 As shown in Figure 1, congestion in the IoT can be categorized into node congestion and link congestion. Node congestion is primarily characterized by the overflow of the node's buffer queue. Since the cluster head node is responsible for delivering data for the entire cluster, its buffer queue needs to accommodate a greater volume of traffic. Therefore, node congestion is more likely to occur at the cluster head node. Link congestion can arise from a variety of factors, including channel contention, data collisions, and poor channel quality. The likelihood of link congestion increases as the number of nodes competing for the channel increases.

[0016] An intelligent congestion control method for the Internet of Things based on deep reinforcement learning comprises: constructing an Internet of Things system, the system comprising a terminal node, a cluster head node, a relay node, and a sink node; the terminal node sending a data packet to the cluster head node; the cluster head node setting a buffer queue according to the data packet sent by the terminal node, and obtaining current cluster head node status information; determining the congestion status of the cluster head node according to the current cluster head node status information, and sending the congestion status to the relay node; the relay node obtaining status information of each node in the Internet of Things according to the congestion status, and sending the status information to the sink node; the sink node optimizing the status information of the Internet of Things using an improved DQN algorithm to obtain a congestion control strategy, and allocating queue tasks in the cluster head node according to the congestion control strategy to complete congestion control.

[0017] In this embodiment, network congestion control generally includes three stages, namely congestion detection, congestion notification, and congestion control.

[0018] During the congestion detection phase, the main indicators used to detect congestion are: buffer queue occupancy, channel load, and packet service time. Buffer queue occupancy can be used to accurately identify the degree of node congestion, channel load can indicate the level of channel contention at the current node, and packet service time indicates the difference between the time a packet arrives at the Medium Access Control (MAC) layer and the time it is delivered upwards. A larger difference indicates greater network congestion. This method primarily uses node buffer queue length and channel load for congestion detection.

[0019] During the congestion notification phase, congestion notifications are divided into explicit and implicit notifications. Explicit notifications inform neighboring nodes of congestion by sending additional control information, but this inevitably incurs additional network resource overhead and can, to a certain extent, increase network congestion. In contrast, implicit notifications utilize information piggybacking to avoid incurring additional overhead.

[0020] During the congestion control phase, congestion control can be categorized into centralized and distributed approaches based on the congestion control architecture. In centralized congestion control, congestion control measures are executed by sink nodes. Sink nodes guide the entire network topology and periodically collect congestion reports from device nodes across the network. After calculation, they issue congestion control measures to the device nodes hop by hop. In distributed congestion control, device nodes execute corresponding congestion control algorithms based on observed congestion conditions.

[0021] During data transmission, the IEEE 802.15.4 protocol is used for data transmission. IEEE 802.15.4 is a protocol standard developed for low-rate wireless personal area networks and is currently one of the most widely used standard protocols in the Internet of Things. This standard defines the physical layer and data link layer of the communication protocol stack.

[0022] The MAC layer belongs to the data link sublayer and is responsible for allocating communication resources to all device nodes and then forwarding data to the physical layer. It is mainly used to avoid data transmission conflicts caused by channel sharing in wireless networks. There are three common MAC protocols: contention-based random access MAC protocol, scheduling-based fixed access MAC protocol, and hybrid MAC protocol that combines contention scheduling. Compared with contention MAC protocol, because contention MAC protocol can allocate channels according to node needs and respond to changes in the number of nodes and changes in network load, fixed access MAC protocol is prone to nodes not transmitting data for a long time, resulting in resource waste. Hybrid MAC protocol has high protocol complexity, high protocol overhead and cost.

[0023] The Carrier Sense Multiple Access with Collision Avoid (CSMA / CA) protocol is a typical contention-based MAC protocol that aims to avoid data collisions caused by stations during data transmission. In the Internet of Things, the CSMA / CA protocol is divided into slotted CSMA / CA and non-slotted CSMA / CA according to whether it supports the beacon network working mode. The former is more complex to implement than the latter. The present invention adopts the non-slotted CSMA / CA algorithm. Figure 2 As shown, the method includes:

[0024] Step 1: Initialize the parameters of the MAC layer data packet of the cluster head node, including the backoff number NB, backoff index BE, frame retransmission number FR and contention window CW;

[0025] Step 2: The cluster head node senses the channel idle state of the relay node. If the relay node is in the idle state, the cluster head node accesses the channel. Otherwise, the node is in the [0,2 BE -1] Randomly select a backoff number to back off;

[0026] Step 3: After the node backs off, it performs idle channel detection. If the current channel is idle, it will receive the node access information after the backoff. Otherwise, the NB value increases by 1 and the BE value increases by 1.

[0027] Step 4: Determine the current backoff times and the maximum CSMA backoff value. When the value of NB exceeds the maximum CSMA backoff value set by the MAC layer, the node fails to access the channel in this round. Otherwise, return to step 2.

[0028] Specifically, the transmission of MAC layer data packets is controlled by four parameters: Number of Backoff (NB), Backoff Exponent (BE), Frame Retries (FR), and Contention Window (CW). In the non-slotted CSMA / CA algorithm, CW is set to 1, so once a node senses that the channel is idle, it can access the channel. Clear Channel Assessment (CCA) is used to detect whether the channel is available. The node needs to be in [0,2 BE -1] Randomly select a backoff number to back off, thus avoiding conflicting access to the wireless channel. After the backoff is over, if the channel is detected to be idle, channel access is performed, otherwise this round of competition fails, the NB value increases by 1, the BE value increases by 1, and BE needs to satisfy min(BE,macMaxBE). macMaxBE is the maximum backoff index set by the MAC layer. Once NB>macMaxCSMABackoff, that is, the NB value exceeds the maximum CSMA backoff value set by the MAC layer, the node fails to access the channel in this round. In addition, to ensure reliable transmission, the CSMA / CA protocol can support data retransmission. After the data packet is sent, it is necessary to wait for confirmation of the data and confirm the successful delivery of the data by sending ACK. If the ACK is not collected within the specified time, it means that the target node has not received the data packet, and the data will be retransmitted. Once the retransmission limit is reached, the data packet transmission fails.

[0029] Deep Neural Networks (DNNs), also known as deep learning, are the foundation of current artificial intelligence applications. "Deep" refers to the multiple hidden layers within a neural network, which can learn more complex and abstract multidimensional data features. DNNs input the data to be analyzed into the network. There are two types of input networks: feedforward networks and recurrent networks. A feedforward network is a static network where information is transmitted in one direction. The network's output depends solely on the current input and lacks memory. Recurrent neural networks, on the other hand, use neurons with self-feedback. The network's output is related to both the current input and the output at the previous moment, thus possessing memory. The long short-term memory (LSTM) network used later in this method is a type of recurrent neural network (RNN).

[0030] Reinforcement learning (RL) is a branch of machine learning that excels at controlling complex systems. The decision-making entity in RL is called an agent. Decision-making problems in RL are solved using the Markov Decision Process (MDP) problem. An MDP consists of a four-tuple (S, A, P, R), where S represents the state set, A represents the action set, P represents the state transition probability of the agent from its current state to the next, and R represents the reward function. In complex systems, agents continuously observe states, perform actions, and receive rewards as they interact with the environment. Figure 3 A schematic diagram of reinforcement learning is given. Unlike supervised learning, RL does not require labeled data or human knowledge for model training. Instead, it uses rewards (i.e., feedback signals from the environment) to update the policy mapping from observed states to corresponding actions.

[0031] In reinforcement learning, Π refers to the strategy, which represents the mapping from state to action. The goal of MDP is to find a strategy to maximize the cumulative discounted reward, which is represented by Ut and is defined as:

[0032]

[0033] Among them, a t =π(s t ), γ is the discount rate, γ∈[0,1], γ determines the importance of future rewards compared to current rewards. If γ=0, the agent only considers maximizing the current reward value, and if γ=1, the agent strives to maximize the long-term cumulative reward value. It can be seen that U t With randomness, when the agent is in s t Finish action a t When U t The randomness of depends on all states and actions that occur after time t+1. If we evaluate the conditional expectation of all states and actions after time t+1, we get:

[0034]

[0035] where Q π (s,a) is called the action-value function, Q π (s,a) only depends on the state s, action a and strategy π at the current time t. When the agent adopts different strategies, the action value function Q π The value of (s,a) is also different. If we want to exclude the influence of strategy π on the action-value function, we need to introduce the optimal action-value function, which is defined as follows:

[0036]

[0037] Approximate Q by continuously updating the Q function * , where the formula for updating the Q function is:

[0038]

[0039] Where α is the learning rate, and α > 0, it affects the learning speed of the agent. The Q-learning algorithm does not require constructing a model for learning; it simply maintains a Q-table, the size of which depends on the dimensions of the state and action. Q-learning exhibits excellent learning performance in reinforcement learning problems with discrete and small sets of states and actions.

[0040] In the early days, tabular reinforcement learning (RL) methods were primarily used to solve related decision-making problems. However, due to the dimensionality of the input information, tabular RL methods require a lot of computation when the dimensionality of the state space or action space is high. Deep reinforcement learning (DRL) is a new method proposed in recent years that addresses this dilemma by introducing deep learning (DL) technology, namely, using deep neural networks as function approximators. In addition to its ability to handle high-dimensional state and action spaces, DRL also offers the advantage of extracting more accurate feature representations than traditional heuristic methods.

[0041] Deep Q Network (DQN) is the most basic DRL algorithm based on value learning. DQN is represented by Q(s, a; θ) as a value evaluator. It evaluates the value of each state-action pair. In order to allow the agent to learn the best strategy from past interactions with the environment, that is, the state-action pair with the maximum value, DQN also introduces Experience Replay to store the experience quadruple (s t ,a t ,r t ,s t+1 ), and a batch of experience is sampled in each training step to train the agent. The DQN parameters are updated using the Temporal-Difference (TD) algorithm, which uses the mean square error between the current Q value and the target Q value as the loss function and continuously updates the DQN parameters θ to reduce the loss function. The loss function is defined as:

[0042]

[0043] Since the DQN algorithm uses the target Q network to select actions and calculate values, it leads to the problem of overestimation of value. In order to reduce the impact of this problem, a new model loss function is constructed, which is expressed as:

[0044]

[0045] in, Expresses expectation, r t represents the reward obtained by the agent at time t, γ represents the reward discount rate obtained by the agent, Q represents the action value function, s t+1 represents the state of the agent at time t+1, a represents the action set, θ represents the weight parameter of the target network, and θ′ represents the weight parameter of the evaluation network.

[0046] The optimal selection strategy learned using a deep Q network includes: the intelligent agent obtains environmental information and inputs the obtained environmental information into the policy network to obtain the action at the current moment; the reward function is used to calculate the reward of the intelligent agent's current action based on the action at the current moment; the current network state, task action, immediate reward and network state at the next moment are stored as a four-tuple in the priority experience replay array; the policy network and value network are trained using the data in the preferred experience replay array to update the Q function; the loss function of the model is calculated and the loss function is optimized using the cumulative discounted reward. When the cumulative discounted reward is maximized, the loss function converges, the model training is completed, and the optimal selection strategy is output.

[0047] A joint MAC layer channel competition access congestion control mechanism based on deep reinforcement learning (DRL-JMCC). The DRL-JMCC mechanism mainly consists of three parts: network environment, Double DQN architecture and ACK feedback recovery mechanism. Figure 4 A detailed architecture diagram of the DRL-JMCC mechanism is provided. The network environment serves as the agent's interaction environment, providing appropriate state and reward information during training. The Double DQN architecture is used for agent training. The ACK feedback recovery mechanism serves as an auxiliary structure, allowing the agent to more effectively cope with poor wireless channel quality during training.

[0048] The process of training the agent using the DRL-JMCC mechanism includes: the state of the DRL-JMCC mechanism is composed of the downlink agent node buffer queue length, the uplink node buffer queue length, the number of downlink nodes sharing the channel and the agent's packet sending rate at the last moment.

[0049] The action A of the agent is a two-tuple (V, T), which refers to the multiple value of the agent's packet sending rate and the backoff time, respectively. The backoff time is an integer. The specific action definition is as follows:

[0050]

[0051] The agent's reward is set as the difference between the number of packets successfully delivered by the agent and the number of packets lost by the network within a fixed period. A successfully delivered packet refers to the successful receipt of an ACK confirmation for the packet. The reward formula is as follows:

[0052] r t =R t -L t

[0053] Among them, R t Indicates the number of data packets received by the sink node within time t, L t Indicates the number of network packet losses within time t.

[0054] The present invention adopts the Double DQN architecture, which first uses experience replay and greedy strategy to accelerate the convergence of the algorithm. The greedy algorithm is manifested in that when the intelligent agent makes a decision, there is a probability of ∈ (∈<1) randomly selecting unknown actions for exploration, and the remaining 1-∈ is to select the optimal action. Experience replay will be mentioned in detail in the ACK feedback recovery mechanism. The deep neural network structure of the DRL-JMCC mechanism is as follows Figure 5 As shown in the figure, the DNN's input layer receives state input, while the hidden layer consists of an LSTM and two fully connected layers. The LSTM has 128 neurons, while the two fully connected layers have 128 and 64 neurons, respectively. The LSTM helps the agent record historical learning information. Because nodes rely on historical information when accessing channels, the introduction of the LSTM facilitates the agent's action decision-making.

[0055] The ACK feedback recovery mechanism focuses on the loss and delay of ACK data packets. This method adopts the experience replay mechanism when training the intelligent agent. Experience refers to the three elements of state, action and reward. The purpose of experience replay is to reduce the dependency between experiences and to improve the utilization of data. The ACK feedback recovery mechanism involves two experience replay pools, namely incomplete experience replay and complete experience replay. If the ACK feedback is not received within the specified time, the current experience is deemed to be incomplete experience and needs to be placed in the incomplete experience replay pool. If the subsequent ACK feedback is delayed, it will make up for the previous lack of information and make the incomplete experience become complete experience and place it in the complete experience pool. Once the ACK feedback is still not received within the set time, it is deemed that the ACK feedback is lost and the incomplete experience will be discarded. The specific ACK feedback recovery mechanism diagram is shown as follows. Figure 6 shown.

[0056] like Figure 7 As shown in the figure, the specific DRL-JMCC mechanism implementation algorithm steps are as follows:

[0057] Step 1: Initialize the network, set an experience replay pool Φ with a capacity of N, evaluate the network weight parameter θ, and the target network weight parameter θ', set θ' = θ, and the network structure of the evaluation network and the target network is exactly the same. This mechanism sets the maximum number of training rounds to M and the current number of training rounds to N e , the total time of each round of training process is T. In addition, after the node channel competes for access, the initialized parameters need to add the number of node retransmissions to limit the node retransmission.

[0058] Step 2: The agent starts to interact with the network environment. First, it also determines N e Check whether the maximum number of training rounds M has been reached. If so, go to step 13; otherwise, go to step 3.

[0059] Step 3: Before each round of agent training begins, the network environment needs to be reset, that is, the initialization parameters are set to the original values ​​by default, and the training time step t is initially 0.

[0060] Step 4: The agent will record the training time during the training process. Once t>T, it means that this round of training is over and jump to step 12. Otherwise, execute step 5.

[0061] Step 5: Randomly select an action with probability ∈ as the bound, otherwise let the evaluation network select the action that produces the best expected reward in the target action value function.

[0062] Step 6: Execute the channel contention access algorithm. Before performing channel access backoff, the node needs to check whether the maximum number of retransmissions has been reached, and then perform the corresponding action. Figure 4-6 Although no explicit retransmission limit is given, the channel contention access algorithm already includes retransmission limits. If channel access is successful, proceed to step 7; otherwise, proceed to step 4.

[0063] Step 7: Check whether the node that successfully accesses the channel successfully receives ACK. If yes, execute step 7; otherwise, execute step 8.

[0064] Step 8: Execute the ACK feedback recovery mechanism. If a late ACK is received within the specified time, execute step 9. Otherwise, the incomplete experience is discarded, the current round of data transmission fails, and execute step 6.

[0065] Step 9: Store the complete experience in the complete experience pool, randomly sample small batches of experience, and find the minimum value of the loss function through gradient descent to update the evaluation network parameters θ.

[0066] Step 10: Determine whether the target network parameter update period has been reached. If so, proceed to step 11; otherwise, jump to step 4.

[0067] Step 11: Update the target network parameters, θ′=θ, and jump to step 4.

[0068] Step 12: Number of training rounds N e Add 1.

[0069] Step 13: The agent training is completed.

[0070] This invention introduces deep reinforcement learning into network congestion control, eliminating the need for pre-data collection. It allows the agent to directly and adaptively interact with the environment and continuously learn optimal strategies. Secondly, the addition of a node-greedy strategy to deep reinforcement learning further facilitates exploratory node learning, helping the agent learn more beneficial actions under traditional rules. Subsequently, during agent training, this invention fully considers the possibility of network congestion, namely, considering both node congestion and link congestion as possible types of network congestion, and incorporates the burst characteristics of network traffic into the network environment, further increasing the robustness of the method.

[0071] The above embodiments further illustrate the purpose, technical solutions and advantages of the present invention in detail. It should be understood that the above embodiments are only preferred implementation plans of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made to the present invention within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. An IoT intelligent congestion control method based on deep reinforcement learning, characterized in that: include: Build an Internet of Things system, which includes terminal nodes, cluster head nodes, relay nodes and sink nodes; terminal nodes send data packets to cluster head nodes; The cluster head node sets up a buffer queue according to the data packets sent by the terminal node and obtains the current cluster head node status information; The current cluster head node status information includes: buffer queue occupancy, channel load and data packet service time; the buffer queue occupancy is used to determine the node congestion, the channel load is used to determine the channel contention level of the current node, and the data packet service time is used to determine the difference between the time when the data packet arrives at the media access control layer and the time when it is delivered upward; the congestion status of the cluster head node is determined based on the current cluster head node status information and the congestion status is sent to the relay node; the relay node obtains the status information of each node in the Internet of Things based on the congestion status and sends the status information to the sink node; the sink node uses the improved DQN algorithm to optimize the status information of the Internet of Things to obtain the congestion control strategy, and allocates the queue tasks in the cluster head node according to the congestion control strategy to complete the congestion control; The congestion status is sent to the relay node by using a non-slotted CSMA / CA algorithm to transmit the congestion status to the relay node. The specific process includes: S1: Initialize the parameters of the MAC layer data packet of the cluster head node, which include the backoff number NB, backoff index BE, frame retransmission number FR and contention window CW; S2: The cluster head node senses the channel idle state of the relay node. If the relay node is in the idle state, the cluster head node accesses the channel. Otherwise, the node is in the [0,2 BE -1] Randomly select a backoff number to back off; S3: After the node backs off, it performs idle channel detection. If the current channel is idle, it will receive the access information of the node after backing off. Otherwise, the NB value increases by 1 and the BE value increases by 1. S4: Determine the current backoff count and the maximum CSMA backoff value. If the value of NB exceeds the maximum CSMA backoff value set by the MAC layer, the node fails to access the channel in this round. Otherwise, return to step S2. The process of optimizing the state information of the Internet of Things using the improved DQN algorithm includes: Step 1: Network initialization, which includes an experience replay pool Φ with a capacity of N, evaluation network weight parameters θ, target network weight parameters θ; maximum number of training rounds M, current number of training rounds N e , the total time T of each round of training process; Step 2: The agent interacts with the network environment to determine whether the current number of training rounds has reached the maximum number of training rounds. If so, the optimized IoT state information is output; otherwise, step 3 is executed. Step 3: Reset the network environment, that is, set the initialization parameters to the original values ​​by default, and the training time step t is initially 0; Step 4: The agent obtains the training time t of the current network. If t>T, the training ends and jumps to step 12. Otherwise, it executes step 5. Step 5: Use the deep Q network to learn the optimal selection strategy and select the optimal action based on the optimal selection strategy; specifically, the agent obtains environmental information and inputs the obtained environmental information into the policy network to obtain the action at the current moment; the reward function is used to calculate the reward of the agent's current action based on the action at the current moment; the current network state, task action, immediate reward and the network state at the next moment are stored as a four-tuple in the priority experience replay array; the policy network and value network are trained using the data in the preferred experience replay array to update the Q function; the loss function of the model is calculated and the loss function is optimized using the cumulative discounted reward. When the cumulative discounted reward is maximized, the loss function converges, the model training is completed, and the optimal selection strategy is output; The formula for calculating discount rewards is: Among them, т represents the agent, γ i-t represents the discount rate, r t represents the reward obtained by the agent at time t, s t represents the state of the agent at time t, π(s t ) indicates that the agent is in s t The policy function under The formula for updating the Q function is: Among them, s t+1 represents the state of the agent at time t+1, a t+1 represents the action of the agent at time t+1, Q(s t ,a t ) represents the action value function of the agent at time t, α represents the learning rate, r t (s t ,a t ) indicates that the agent is in s t ,a t The reward obtained under , γ represents the discount rate, a represents the action set, Q t represents the action value function of the agent at time t; The loss function of the model is expressed as: in, Expresses expectation, r t represents the reward obtained by the agent at time t, γ represents the reward discount rate obtained by the agent, Q represents the action value function, θ represents the weight parameter of the target network, and θ ′ Represents the weight parameters of the evaluation network; Step 6: Use the channel contention access algorithm to determine whether the node has reached the maximum number of retransmissions. If so, the node has successfully accessed the channel and executes step 7. Otherwise, return to step 4. Step 7: Determine whether the node that successfully accesses the channel receives the ACK information. If it does, proceed to step 9; otherwise, proceed to step 8. Step 8: Execute the ACK feedback recovery mechanism for the node that successfully accesses the channel and resend the ACK information to the node. If the information is received, execute step 9. Otherwise, the current round of data transmission fails and return to step 6. Step 9: Store the complete experience in the complete experience pool, randomly sample small batches of experience, find the minimum value of the loss function through gradient descent, and update the evaluation network parameter θ according to the minimum value of the loss function; Step 10: Determine whether the current network has reached the target network parameter update period. If so, execute step 11; otherwise, return to step 4. Step 11: Update the target network parameters, θ′=θ, and jump to step 4; Step 12: Number of training rounds N e Add 1 and return to step 2.

Images