A routing optimization method for IoT based on deep reinforcement learning

Abstract

The present invention belongs to the technical field of wireless network communications, and particularly relates to an Internet of Things (IoT) routing optimization method based on deep reinforcement learning. The method comprises: networking a sink node and ordinary sensor nodes to construct a DODAG; entering a data transmission phase, in which the sensor nodes perform data packet sending and receiving operations; after the data transmission phase ends, waking up a Trickle timer to reset a DIO interval; entering a routing optimization phase, in which all nodes broadcast DIO control messages according to the DIO interval; sensor nodes download the latest model parameters from the sink node; routing is selected and experience information is generated based on the latest model parameters and DIO control messages from neighboring sensor nodes; the sink node trains a Dueling DQN model based on experience and updates the model parameters; and after the routing optimization phase ends, freezing the Trickle timer. The present invention can improve various network performances, such as load balancing, life cycle, and reliability.

Description

Technical Field

[0001] The present invention belongs to the technical field of wireless network communications, and in particular relates to an Internet of Things routing optimization method based on deep reinforcement learning. Background Art

[0002] The Internet of Things (IoT) is a system of interconnected computing devices, machinery, digital devices, and other devices. It is a network in which objects communicate with each other via the internet. Currently, IoT is widely used in scenarios such as environmental monitoring, smart grids, home automation, and industrial automation. Some devices used in the IoT exhibit high losses, low power consumption, and limited resources. Therefore, the Internet Engineering Task Force (IETF) has proposed the Routing Protocol for Low-Power and Lossy Networks (RPL) specifically for networks composed of such devices. While RPL is well-suited to lossy links and node energy constraints, with the rapid growth of IoT devices and increasing network complexity, RPL faces challenges such as reduced reliability, increased energy consumption, and increased latency caused by congestion due to heavy loads, which can severely degrade overall network performance and lifecycle. Therefore, optimizing RPL routing for different application scenarios has become an important research topic.

[0003] Many current RPL protocol optimization algorithms based on fixed policies can effectively select the next hop among a set of candidate nodes. However, most existing traditional RPL routing optimization methods select specific routing metrics and design specific objective functions based on the network application scenario. In complex and changing network environments, these methods struggle to autonomously make intelligent routing decisions based on real-time network status and application requirements. Objective functions based on weighted combinations of multiple metrics have improved RPL protocol performance to some extent, but the varying magnitudes of the metrics make them difficult to align. Furthermore, fixed-policy optimization methods lack intelligence and are unable to effectively interact with the environment in dynamic network scenarios, providing more flexible routing choices for nodes. Fuzzy logic-based routing optimization methods achieve good fits for various routing metrics, but the fuzzy rules are often based on statistical data or are manually defined, resulting in a degree of subjectivity. Therefore, more intelligent routing selection solutions are needed, and some researchers have introduced Q-Learning techniques from reinforcement learning for routing optimization. The routing optimization method based on Q-Learning is more effective than traditional optimization algorithms. However, the QL algorithm requires maintaining a Q table. In high-density scenarios, the Q table will be very large, and both search and storage will consume a lot of time and space. Summary of the Invention

[0004] In view of the shortcomings of the existing technology, the present invention proposes an IoT routing optimization method based on deep reinforcement learning, which includes:

[0005] S1: The sink node is used as the root node, and the common sensor nodes are used as descendant nodes to form a tree structure to build a DODAG.

[0006] S2: Entering the data transmission phase, the sensor node performs data packet sending and receiving operations. After the data transmission phase ends, the Trickle timer is awakened to reset the DIO interval.

[0007] S3: Entering the routing optimization phase, all nodes broadcast DIO control messages according to the DIO interval;

[0008] S4: The sensor node downloads the latest model parameters from the sink node; performs routing selection and generates experience information based on the latest model parameters and the DIO control message of the neighboring sensor node; and uploads the experience information to the sink node;

[0009] S5: The sink node trains the Dueling DQN model based on experience and updates the model parameters;

[0010] S6: After the route optimization phase ends, the Trickle timer is frozen;

[0011] S7: Return to step S2 and perform the next routing cycle until the network stops running.

[0012] Preferably, the minimum value of the DIO interval is the duration of the route optimization phase.

[0013] Preferably, the process of performing route selection and generating experience information includes:

[0014] S41: Obtain routing metric information according to the DIO control message of the neighboring sensor node and construct a state matrix according to the routing metric information;

[0015] S42: Calculate action a based on the latest model parameters and state matrix, perform routing selection based on action a and switch the parent node;

[0016] S43: Calculate the reward value based on action a and the routing metric information of the parent node;

[0017] S44: Generate experience information based on the state matrix, action a and reward value.

[0018] Furthermore, the process of constructing the state matrix includes: obtaining the routing metrics of neighboring sensor nodes based on DIO control messages, including ETX, HC, RE, and QLR; finding the k nodes with the highest RE values ​​among the candidate parent nodes based on the routing metrics to generate the state matrix; if the number of candidate parent nodes is less than k, all candidate nodes are filled into the state matrix, and the remaining entries are filled with 0; wherein the matrix size is 4 rows and k columns.

[0019] Furthermore, the action a is calculated by inputting the state matrix into the local routing module, indicating that in the data transmission phase, the node corresponding to the value a in the state matrix is ​​selected as the parent node for data forwarding; wherein a∈[0,k-1], k represents the number of columns in the state matrix.

[0020] Furthermore, the formula for calculating the reward value is:

[0021]

[0022] Among them, r represents the reward value, Rank(N i ) represents the rank value of node i; a represents the action, k represents the number of state matrix columns, and m represents the number of candidate parent nodes of the node.

[0023] Furthermore, the calculation formula of rank value is:

[0024] Rank(N i )=Rank(p)+ω1QU(p)+ω2ECR(p)

[0025] Among them, Rank(N i ) represents the rank value of node i, Rank(p) represents the rank value of the parent node p of node i, QU(p) represents the queue utilization of the parent node p, ECR(p) represents the energy consumption rate of the parent node, ω1 represents the first weight, and ω2 represents the second weight.

[0026] Preferably, the sink node is deployed with a Dueling DQN model, which includes an estimation network and a target network, and the estimation network and the target network are neural networks with the same network structure.

[0027] Preferably, the common sensor node is deployed with a local routing decision module, and the local routing decision module has the same structure as a single neural network in the sink node.

[0028] The beneficial effects of the present invention are as follows: the present invention uses deep reinforcement learning technology to perform routing optimization of RPL, and uses neural networks to fit Q tables or directly fit strategies to solve the problem that the state space action space in shallow reinforcement learning is too large and difficult to converge. Compared with the fully distributed multi-agent reinforcement learning method, the CTDE architecture adopted in the present invention can fully collect and train more comprehensive learnable experience in the entire network. Compared with Q-Learning, the duelingDQN model used in the present invention has a better fitting ability for the Q function. The present invention selects ETX, HC, QU, ECR, QLR and other routing metrics for load balance and energy balance to construct the state space and reward function, which can better seek more intelligent and effective routing methods in dense scene wireless networks to improve the network's load balance, life cycle, reliability and other performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is a flow chart of the IoT routing optimization method based on deep reinforcement learning in the present invention;

[0030] Figure 2 This is an architecture diagram of the IoT routing optimization method based on deep reinforcement learning in the present invention;

[0031] Figure 3 Schematic diagram of the superframe period structure in the present invention. DETAILED DESCRIPTION

[0032] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0033] This paper proposes an IoT routing optimization method based on deep reinforcement learning. Figure 1 As shown, the method includes the following contents:

[0034] S1: The sink node is used as the root node and the common sensor nodes are used as descendant nodes to form a tree structure to build a DODAG.

[0035] The network used in the present invention is a tree-shaped network topology, which consists of a root node (a sink node with sufficient computing resources, storage resources and energy) and N descendant nodes (ordinary sensor nodes).

[0036] like Figure 2As shown, sensor nodes are usually resource-constrained devices. In order to implement a lightweight local routing optimization module on the sensor node, reduce the computational cost of the model and improve the convergence speed of the deep reinforcement learning (DRL) model, the present invention adopts a centralized training decentralized execution (CTDE) architecture to decouple the model training and reasoning processes. The routing optimization method proposed in the present invention adopts the dueling DQN (Dueling Deep Q Network) model that introduces the CTDE architecture. Specifically, in the Internet of Things network scenario, the sink node deploys a complete dueling DQN model and collects asynchronous experience from the sensor nodes, learns routing strategies from it and updates network parameters; the sensor node only needs to deploy a local routing optimization module for routing selection and intelligently selects the optimal parent node based on the perceived local environment state. Among them, the Dueling DQN model includes an estimation network and a target network. The estimation network and the target network are neural networks with the same network structure, and the local routing decision module has the same structure as the single neural network in the sink node.

[0037] In order to further decouple data transmission and routing maintenance processes, such as Figure 3 As shown, the present invention divides the entire network operation cycle into a network initialization phase and multiple consecutive superframe periods. During the network initialization phase, each node transmits control messages such as DODAG Information Object (DIO) and DODAG Destination Advertisement Object (DAO) to establish a network. When entering a superframe period, the routing optimization method of the present invention begins to operate.

[0038] S2: Entering the data transmission phase, the sensor node performs data packet sending and receiving operations; after the data transmission phase ends, the Trickle timer is awakened to reset the DIO interval.

[0039] During the data transmission phase, the node is responsible for forwarding the generated and received data packets. After the data transmission phase is completed, the Trickle timer is woken up to reset the DIO interval to the minimum value I min The present invention sets the minimum value of the DIO interval I min is the duration of the routing optimization phase T r . Reset the DIO transmission interval to I in each route optimization phase minThis ensures that each node can broadcast its own DIO control message to surrounding neighbor nodes at least once within the duration of each superframe. Based on this, each sensor node can periodically update its local state based on the DIO control messages received from neighbor nodes.

[0040] S3: Entering the routing optimization phase, all nodes broadcast DIO control messages according to the DIO interval.

[0041] In each routing optimization phase, sensor nodes regularly broadcast and receive DIO control messages, that is, the frequency of broadcasting and receiving DIO control messages is the DIO interval set by the Trickle timer; each node uses the options field in the DIO control message to broadcast its own comprehensive routing metric to neighboring nodes.

[0042] S4: The sensor node downloads the latest model parameters from the sink node; performs routing selection and generates experience information based on the latest model parameters and the DIO control message of the neighboring sensor node; and uploads the experience information to the sink node.

[0043] The routing optimization problem is defined as a four-tuple<S,A,P,R> Represents a Markov decision process (MDP). S represents the state space, A represents the action space, P is the state transition probability, and R represents the reward function.

[0044] The process by which a sensor node selects a route and generates experience information based on the latest model parameters and DIO control messages from neighboring sensor nodes includes:

[0045] S41: Obtain routing metric information according to the DIO control message of the neighboring sensor node and construct a state matrix according to the routing metric information.

[0046] In the present invention, the state space S is defined as S={s1,s2,...,s t}, where each s i∈S is a state. A node obtains routing metric information from neighboring sensor nodes based on received DIO control messages. Routing metrics include expected transmission count (ETX), hop count (HC), residual energy (RE), queue loss ratio (QLR), queue utilization (QU), and energy consumption rate (ECR). Based on the ETX, HC, RE, and QLR in the routing metrics, the k nodes with the highest RE values ​​are found among the candidate parent nodes to generate a state matrix. If the number of candidate parent nodes is less than k, all candidate nodes are added to the state matrix, and the remaining entries are padded with zeros. The state matrix is ​​4 rows and k columns long and contains k entries, each of which represents the routing metric information for a candidate parent node.

[0047] The node action space consists of a finite set of actions, defined as A = {0, 1, 2, ..., k-1}. Each action a is calculated from the state matrix input by the local routing module. During the data transmission phase, the node corresponding to the value a in the state matrix is ​​selected as the parent node for data forwarding. Here, a∈[0, k-1], where k represents the number of columns in the state matrix, is matched to the corresponding node in the state matrix. If the value a outputs a zero-padded entry, the node will not switch parents during this routing optimization phase and will be assigned a negative reward to avoid outputting a value corresponding to a zero-padded entry.

[0048] S42: Calculate the action according to the latest model parameters and state matrix, perform routing selection and switch the parent node according to the action.

[0049] All sensor nodes can construct corresponding local states s related to their available candidate parent nodes. The local routing optimization module is a policy network that can generate the optimal action a based only on the locally observed states s.

[0050] The sensor nodes will update the strategic network parameters θ by downloading the latest network parameters from the estimated network of the sink node in each routing optimization phase, which can be expressed as:

[0051] θ i ←θ

[0052] And calculate the action a based on the latest model parameters and state matrix:

[0053] a t =π(s t θ i )

[0054] In this invention, the ∈-greedy strategy is adopted to explore more possibilities. Regardless of the Q value calculated by the strategy network, random actions will be selected according to the probability of ∈(0≤∈≤1).

[0055] After selecting an action, perform routing selection and switch the parent node according to the action.

[0056] S43: Calculate the reward value based on the action and the routing metric information of the parent node.

[0057] The formula for calculating the reward value is:

[0058]

[0059] Among them, r represents the reward value, Rank(N i ) represents the rank value of node i; a represents the action, and its value range is [0, k-1]; k represents the number of columns in the state matrix.

[0060] When a node selects a specific node based on its action, its reward is allocated according to its rank value, and a negative reward is allocated when a 0 filler item is selected.

[0061] The formula for calculating the rank value is:

[0062] Rank(N i )=Rank(p)+ω1QU(p)+ω2ECR(p)

[0063] Among them, Rank(N i ) represents the rank value of node i, Rank(p) represents the rank value of node i’s parent node p, QU(p) represents the queue utilization of parent node p, i.e., the ratio of the number of packets in the node buffer queue to the total size of the buffer queue; ω1 represents the first weight, ω2 represents the second weight, ω1, ω2∈[0,1], and their values ​​can be determined according to the specific network performance requirements; ECR(p) represents the energy consumption rate of the parent node, which is defined as:

[0064]

[0065] S44: Generate experience information based on the state matrix, action and reward value.

[0066] In the routing optimization phase, sensor nodes generate experience t ,a t ,r t ,s t+1 >, and upload it to the prioritized experience replay (PER) buffer, and the model samples a small batch of experience from the PER buffer for model training. ​

[0067] S5: The sink node trains the Dueling DQN model based on experience and updates the model parameters.

[0068] The present invention adopts the Dueling DQN model and introduces the Prioritized Experience Replay (PER) mechanism to improve learning efficiency. The estimated network Q(s,a;θ) and the target network Q(s,a;θ) in the Dueling DQN model deployed at the sink node are - ) share the same neural network structure but different parameters. The model is trained based on empirical information, where the estimation network is used to approximate the Q-value function, while the target network is used to calculate the target Q-value.

[0069] The Dueling DQN model used in this paper uses two convolutional layers as input layers to effectively extract feature information from the node state matrix. The neural network related parameter settings are shown in Table 1:

[0070] Table 1 Neural network related parameters

[0071] Number of convolutional layers 2 Number of hidden layers 3 Sampling size 32 Learning rate α 0.01 Reward Discounts¶ 0.9 Target network update steps C 5 Greedy strategy probability ∈ 0.9 Experience pool capacity 1000 Loss Function Mean squared loss function

[0072] The sink node samples a small batch of experiences from the PER buffer for model training and network parameter updates. The PER mechanism breaks uniform sampling and assigns greater sampling weight to experiences with high learning efficiency, thereby improving model learning efficiency and accelerating model convergence. Specifically, the present invention adopts a proportional priority PER mechanism, that is, the sampling probability is proportional to the temporal-difference (TD) error calculated by the model. The TD error is calculated as follows:

[0073]

[0074] Where r is the reward value and γ is the discount factor. PER performs non-uniform sampling on the experience samples in the buffer, and the weight of each experience is the absolute value of the TD error. The sampling probability of the proportional priority PER mechanism for experience j is:

[0075] p j ∝|δ j |+η

[0076] Among them, η is a small positive number that prevents the sampling probability from approaching zero and is used to ensure that all samples are sampled with non-zero probability. The purpose of model training is to minimize the loss calculated between the target network and the estimation network. The loss function is defined as:

[0077]

[0078] Among them, l represents the size of the sampled experience batch, w jRepresents weighted importance sampling, which is used to correct the calculation bias caused by the fact that some high-weight experiences may be sampled more frequently. The parameter update of the Dueling DQN model is back-propagated by the calculated loss function:

[0079]

[0080] Where α is the learning rate. The parameters of the estimated network are continuously updated in each cycle, while the parameters of the target network are updated once every C training steps. That is, the target network copies the parameters of the estimated network once every C cycles:

[0081] θ - ←θ

[0082] S6: After the route optimization phase is completed, the Trickle timer is frozen.

[0083] Sensor nodes use the Trickle timer to send DIO control messages and adaptively control their transmission frequency to maintain the DODAG. In unstable network conditions, such as when nodes join or leave or routing loops occur, nodes increase the frequency of DIO message transmission to update information as quickly as possible to ensure rapid network stabilization. Once the network stabilizes, nodes gradually increase the DIO message transmission interval to reduce control overhead. The Trickle timer is frozen during the data transmission phase to decouple data transmission and routing maintenance, reducing control overhead and conflicts.

[0084] S7: Return to step S2 and perform the next routing cycle until the network stops running.

[0085] In summary, the present invention models the RPL routing problem as an MDP problem, utilizes a trained asynchronous DRL model for routing, and adopts a centralized training distributed execution (CTDE) architecture. A sink node deploying a dueling DQN collects asynchronous experience information uploaded by network nodes. Combined with a priority experience replay mechanism, uniform sampling is broken, and a larger sampling weight is allocated to experience with high learning efficiency, thereby further improving model training efficiency. Ordinary sensor nodes only need to deploy a local routing decision module. This module has the same structure as the single neural network in the sink node and is used to output the optimal action a based on the local state s observed by the node, thereby achieving intelligent and efficient routing selection. A method for dividing the superframe period is proposed to decouple the data transmission and routing update phases to improve the interaction efficiency between the DRL agent and the environment. At the same time, to better serve the division of the superframe period, the present invention also proposes an improved Trickle mechanism to control the sending time of DIO control messages to avoid information loss due to unnecessary collisions.

[0086] The present invention uses deep reinforcement learning technology to perform RPL routing optimization, and uses neural networks to fit the Q table or to solve the problem that the state space action space is too large and difficult to converge in shallow reinforcement learning. Compared with the fully distributed multi-agent reinforcement learning method, the CTDE architecture used in the present invention can fully collect and train more comprehensive learnable experience in the entire network. And compared with Q-Learning, the dueling DQN model used in the present invention has a better fitting ability for the Q function. Furthermore, the present invention selects ETX, QU, ECR, HC, QLR and other routing metrics for load balance and energy balance to construct the state space and reward function, which can better seek more intelligent and effective routing methods in dense scene wireless networks to improve the network's load balance, life cycle, reliability and other performance.

[0087] The above embodiments further illustrate the purpose, technical solutions and advantages of the present invention in detail. It should be understood that the above embodiments are only preferred implementation plans of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made to the present invention within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for IoT routing optimization based on deep reinforcement learning, characterized in that: include: S1: The sink node is used as the root node, and the common sensor nodes are used as descendant nodes to form a tree structure to build a DODAG. S2: Entering the data transmission phase, the sensor node performs data packet sending and receiving operations. After the data transmission phase ends, the Trickle timer is awakened to reset the DIO interval. S3: Entering the routing optimization phase, all nodes broadcast DIO control messages according to the DIO interval; S4: The sensor node downloads the latest model parameters from the sink node; it performs routing selection and generates experience information based on the latest model parameters and the DIO control messages of neighboring sensor nodes; Upload the experience information to the aggregation node; The process of performing route selection and generating experience information includes: S41: Obtain routing metric information based on the DIO control message of the neighboring sensor node and construct a state matrix based on the routing metric information; the process of constructing the state matrix includes: obtaining the routing metrics of the neighboring sensor node based on the DIO control message, including ETX, HC, RE, and QLR; finding k nodes with the highest RE value among the candidate parent nodes based on the routing metrics to generate the state matrix; if the number of candidate parent nodes is less than k, then fill all candidate nodes into the state matrix and fill the remaining entries with 0; wherein the matrix size is 4 rows and k columns; S42: Calculate action a based on the latest model parameters and state matrix, perform routing selection and switch parent nodes based on action a; the action a is calculated by inputting the state matrix into the local routing module, indicating that the node corresponding to the value a in the state matrix is ​​selected as the parent node for data forwarding during the data transmission phase; where a∈ [0,k-1], k represents the number of columns in the state matrix; S43: Calculate the reward value based on action a and the routing metric information of the parent node; the formula for calculating the reward value is: Among them, r represents the reward value, Rank(N i ) represents the rank value of node i; m represents the number of candidate parent nodes of the node; The formula for calculating the rank value is: Rank(N i )=Rank(p)+ω1QU(p)+ω2ECR(p) Where Rank(p) represents the rank value of the parent node p of node i, QU(p) represents the queue utilization of the parent node p, ECR(p) represents the energy consumption rate of the parent node, ω1 represents the first weight, and ω2 represents the second weight; S44: Generate experience information based on the state matrix, action a and reward value; S5: The sink node trains the Dueling DQN model based on experience and updates the model parameters; S6: After the route optimization phase ends, the Trickle timer is frozen; S7: Return to step S2 and perform the next routing cycle until the network stops running.

2. The method for optimizing Internet of Things routing based on deep reinforcement learning according to claim 1, wherein: The minimum value of the DIO interval is the duration of the route optimization phase.

3. The method for optimizing Internet of Things routing based on deep reinforcement learning according to claim 1, wherein: The sink node is deployed with a Dueling DQN model, which includes an estimation network and a target network. The estimation network and the target network are neural networks with the same network structure.

4. The method for optimizing Internet of Things routing based on deep reinforcement learning according to claim 1, wherein: The common sensor nodes are deployed with a local routing decision module, which has the same structure as a single neural network in the sink node.

Images