Energy-saving small-world wireless Mesh network construction method based on reinforcement learning
Through reinforcement learning algorithm dynamically replaces traditional links with LoRa links in wireless Mesh networks, optimizes routing, and solves the problems of low transmission rate and unbalanced load in traditional technologies, and realizes efficient data transmission and energy consumption management. Extends the network life of IoT devices.
Patent Information
- Application Number
- CN202411542370.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-31
- Publication Date
- 2024-12-03
AI Technical Summary
Traditional wireless communication technology cannot meet the long-distance and low-power data transmission requirements in large-scale networks or drone networks. Although LoRa technology has low power consumption and long transmission distance, its low data transmission rate leads to increased communication delay and Unbalanced load affects network life.
Adopt the energy-saving small-world wireless Mesh network construction method based on reinforcement learning. Through the Q learning framework, the weight between traditional links and LoRa links is learned, dynamically select and replace traditional links with LoRa links, optimize network routing, and reduce average paths. Length and energy consumption.
It improves the network data transmission efficiency and energy efficiency, extends the network life, reduces maintenance costs and service interruption risks, and achieves higher network reliability and packet delivery rates.
Smart Images

Figure CN119071890A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of intelligent optimization algorithms and wireless communication technology, and relates to a method for constructing an energy-saving small-world wireless Mesh network based on reinforcement learning. Background Art
[0002] With the rapid development of the Internet of Things (IoT), traditional wireless communication technologies can no longer meet the needs of social development. Although communication protocols such as Zigbee and Wi-Fi have certain advantages in power consumption and latency, their applications in large-scale networks or drone networks are limited due to their short communication distance. In order to achieve long-distance, low-power data transmission, Low-Power Wide-Area Network (LPWAN) technology has been widely used. Among them, LoRa technology has gradually become an important means of connecting IoT devices due to its low power consumption, long transmission distance, and strong anti-interference ability. However, due to the relatively low data transmission rate of LoRa technology, relying solely on LoRa technology may cause problems such as increased communication latency or unbalanced load, resulting in accelerated energy consumption of some nodes, which in turn affects the life of the entire network.
[0003] Small-World Network (SWN) is an efficient communication network with two major characteristics: high clustering and low characteristic path length. Small-world networks are connected by a small number of long links, which enables the network to achieve a shorter global average path length while maintaining local dense connections. Reinforcement learning, as an intelligent optimization algorithm, can dynamically adjust routing decisions according to environmental changes, and is particularly suitable for network optimization in highly dynamic environments. Applying reinforcement learning to link selection and network optimization can effectively improve the load balancing and energy efficiency of the network. In large-scale Internet of Things, intelligent algorithms such as Q-learning are used to optimize node routing selection, dynamically adjust load distribution between nodes, and introduce small-world characteristics into the Internet of Things, which helps to reduce the energy consumption of Internet of Things devices and extend the network life.
[0004] Wireless Mesh Network (WMN) is one of the key technologies for building the next generation of broadband wireless access networks. It has been widely used in many fields due to its advantages such as low cost, self-healing, self-configuration, strong fault tolerance, good scalability, convenient and fast deployment, and wide coverage. For example, intelligent traffic management in smart cities, precision agriculture monitoring of agricultural Internet of Things, equipment status monitoring and control in industrial automation, real-time data transmission of logistics tracking systems, real-time data collection of environmental monitoring, and device interconnection and management in smart homes. These applications not only improve data transmission efficiency, but also optimize resource utilization, and promote the intelligent and sustainable development of various industries. However, in terms of routing selection, due to the high dynamics of IoT devices, the traditional static routing algorithm of wireless mesh networks cannot adapt to the frequently changing network topology, making the management and maintenance of the network more complicated. Summary of the invention
[0005] The technical problem to be solved by the present invention is to provide a method for constructing an energy-saving small-world wireless Mesh network based on reinforcement learning, which can extend the network life and improve the data transmission efficiency of the network.
[0006] The IoT devices using the present invention are randomly distributed in km 2 in the region, where and are the length and width of the area respectively. The location of the gateway is fixed at the center of the entire area. All IoT devices in the network have the same initial energy, and the gateway is always powered on. Each device in the network knows its location, neighbor list and distance, remaining energy, and queue size. Depending on the device specifications and network specifications, each device uses the LoRaWAN protocol to send data packets to the gateway directly or through multiple hops. The gateway uses the Transmission Convergence Protocol / Internet Protocol (TCP / IP) to send the received data packets to the network layer server, and finally the data packets are sent to the application layer server.
[0007] The small-world wireless mesh network graph proposed by the present invention can be represented by ,in is a collection of N IoT devices, Represents a set of M gateways deployed in the network. represents the set of all possible traditional links, that is, , represents the Euclidean distance between the a-th IoT device and the b-th IoT device, is the maximum range of a conventional link. Similarly, It represents the set of all possible LoRa links introduced in the network, where represents the minimum distance of the LoRa link, and > .gather This includes the legacy links that have not been replaced and the additional LoRa links that have been introduced. In particular, For a conventional wireless mesh network, it is an empty set.
[0008] Since this work needs to introduce small-world features by replacing a part of traditional links with LoRa links, a reinforcement learning method is proposed to learn all traditional links and LoRa link pairs to add weights. The energy-saving small-world wireless Mesh network construction method based on reinforcement learning of the present invention is characterized by comprising the following steps:
[0009] Step 1: Connect IoT devices and central area gateways with traditional links to create a wireless Mesh network;
[0010] Step 2: Get the distance (Include All possible traditional links within ) and distances greater than (Include ) are stored in space and space middle;
[0011] Step 3: Initialize the Q matrix;
[0012] Step 4: Iterate and train the Q matrix;
[0013] Step 5: Sort the state-action pairs in descending order of the Q values in the trained Q matrix;
[0014] Step 6: Create a small-world wireless mesh network based on the Q matrix, select the state-action pairs with the highest Q value in the Q matrix in turn, and replace the traditional links with LoRa links;
[0015] Step 7: Calculate the life cycle, remaining energy and packet delivery rate of the small-world wireless mesh network.
[0016] When creating a small-world wireless mesh network based on the Q-learning framework, the gateway is defined as an intelligent agent, the replaced traditional link is considered as a state, the LoRa link is considered as an action, the reward represents the network's evaluation of the action performed by the intelligent agent in the previous round, and a weighted function is used to balance the network energy efficiency.
[0017] Since the correspondence between the traditional link and the LoRa link is regarded as a state-action pair, the dimension of the initialization Q matrix is For the training of Q matrix, the present invention converts the IoT device ( ), traditional link ( )、LoRa link( )、Exploration Rate( ), learning rate ( )、Discount rate( ), the maximum range of traditional links ( ) and the minimum distance of the LoRa link ( ) as input and output a small-world wireless mesh network. In each round, the agent selects the first state (traditional link) and uses the working principle of Q learning to use the exploration rate ( ) to decide the corresponding actions (LoRa link) for exploration and development. First, a random number is selected in [0,1] ,if , the agent will choose to perform a random action (exploration), otherwise the agent will choose the best action (exploitation) based on the Q value. Then replace the selected traditional link with the LoRa link to construct a new network. For the new network, calculate the average path length of the IoT devices connected to the selected traditional link ( ), average clustering coefficient ( ) and node degree ( ), and the remaining energy of IoT devices connected to the LoRa link ( ). Then, the reward or penalty is calculated according to formula (1):
[0018] (1)
[0019] in, and They are the node degrees of IoT devices connected to the current traditional links, and is the remaining energy of the IoT devices connected to the selected LoRa link. The main reason for considering the node degree of IoT devices connected to the traditional link is to avoid the IoT devices from being disconnected from the network when creating a SW-WMN. In addition, the reason for selecting the energy level of IoT devices connected to the LoRa link is to improve the network lifetime. According to the reward or penalty, the state-action pair can be updated using formula (2): Q value:
[0020] (2)
[0021] Among them, the learning rate coefficient \eta \in \left [ {0,1} \right ] , used to control the learning speed. The faster the learning speed, the faster the Q table converges and the greater the possibility of oscillation. Discount factor \mu \in \left [ {0,1} \right ] , used to characterize the foresight of the learning system. The larger the value, the more the system pays attention to the impact of future states. Represents the reward for the target action at the current state. The key point to note is that the target action is only used to update the Q value of the current state, it may not be the final target action.
[0022] The use and The small-world network parameters verify the creation of SW-WMN. In this work, we use equations (3) and (4) to verify the formation of SW-WMN:
[0023] > (3)
[0024] < (4)
[0025] in, and denote the average clustering coefficient and average path length after replacing n traditional links with LoRa links, respectively. If either condition fails, a SW-WMN is formed. Once the SW-WMN is formed, we are able to evaluate the network’s lifetime, residual energy, and packet delivery rate to verify the performance improvement of the proposed method.
[0026] The present invention is compared with the LEACH protocol, the improved LEACH protocol, the single-hop direct data transmission and the traditional multi-hop data transmission, and its remarkable effects in terms of network life cycle, residual energy and data packet delivery rate are fully verified. The present invention proposes a method for constructing an energy-saving small-world wireless Mesh network based on reinforcement learning. The method combines the reinforcement learning algorithm with the low-power wide area network technology, and realizes the low average path length of the network by introducing the small-world feature, reducing the number of intermediate hops required for data transmission, which not only improves the data transmission efficiency of the IoT device, but also saves the energy consumption of the device. A network with high residual energy has a positive impact on the network life, which in turn can improve the network reliability and reduce the maintenance cost. Compared with other methods, the proposed method has more active nodes, which means that it can run longer with lower risks of service interruption and data loss. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments.
[0028] Figure 1 It is a schematic diagram of the Internet of Things topology to which the present invention is applied.
[0029] Figure 2 It is a node distribution schematic diagram of the present invention.
[0030] Figure 3 This is a flow chart of creating a small-world wireless Mesh network based on Q learning of the present invention.
[0031] Figure 4 This is a comparison chart of the number of dead nodes changing with the number of data transmission rounds under different communication algorithms.
[0032] Figure 5 This is a comparison chart of how the network's remaining energy changes with the number of data transmission rounds under different communication algorithms.
[0033] Figure 6 This is a comparison chart of packet delivery rate changes with data transmission rounds under different communication algorithms. DETAILED DESCRIPTION
[0034] like Figure 1 , 2 As shown, the IoT devices applying the present invention are randomly distributed in km 2 in the region, where and are the length and width of the area respectively. The location of the gateway is fixed at the center of the entire area. All IoT devices in the network have the same initial energy and the gateway is always powered on. Each device in the network knows its location, neighbor list and distance, remaining energy and queue size. Depending on the device specifications and network specifications, each device sends data packets to the gateway using the LoRaWAN protocol directly or through multiple hops. Figure 1 The black dotted line in the figure represents the traditional link for data transmission between devices, and the red solid line represents the LoRa link for data transmission between devices. The gateway uses the Transmission Convergence Protocol / Internet Protocol (TCP / IP) to send the received data packets to the network layer server, and finally the data packets are sent to the application layer server.
[0035] The small-world wireless mesh network graph proposed by the present invention can be represented by ,in is a collection of N IoT devices, Represents a set of M gateways deployed in the network. represents the set of all possible traditional links, that is, , represents the Euclidean distance between the a-th IoT device and the b-th IoT device, is the maximum range of a conventional link. Similarly, It represents the set of all possible LoRa links introduced in the network, where represents the minimum distance of the LoRa link, and > .gather This includes the legacy links that have not been replaced and the additional LoRa links that have been introduced. In particular, For a conventional wireless mesh network, it is an empty set.
[0036] Since this work needs to introduce small-world features by replacing a part of traditional links with LoRa links, a reinforcement learning method is proposed to learn all the traditional links and LoRa link pairs to add weights. Figure 3 As shown, the energy-saving small-world wireless Mesh network construction method based on reinforcement learning of the present invention is as follows:
[0037] Step 1: Connect IoT devices and central area gateways with traditional links to create a wireless Mesh network;
[0038] Step 2: Get the distance (Include All possible traditional links within ) and distances greater than (Include ) are stored in space and space middle;
[0039] Step 3: Initialize the Q matrix;
[0040] Step 4: Iterate and train the Q matrix;
[0041] Step 5: Sort the state-action pairs in descending order of the Q values in the trained Q matrix;
[0042] Step 6: Create a small-world wireless mesh network based on the Q matrix, select the state-action pairs with the highest Q value in the Q matrix in turn, and replace the traditional links with LoRa links;
[0043] Step 7: Calculate the life cycle, remaining energy and packet delivery rate of the small-world wireless mesh network.
[0044] In this implementation, when creating a small-world wireless mesh network based on the Q-learning framework, some concepts are defined as follows:
[0045] (1) Agent: This work regards the gateway as an agent that takes actions for each state.
[0046] (2) State Space: The traditional links that should be replaced are considered as states. That is the state space.
[0047] (3) Action Space: Since the traditional link is replaced by the LoRa link, the LoRa link is regarded as an action. This is the action space.
[0048] (4) Reward: Reward represents the network’s evaluation of the actions performed by the agent in the previous round. A weighted function is used to balance the network’s energy efficiency.
[0049] In step 3, since the correspondence between the traditional link and the LoRa link is regarded as a state-action pair, the dimension of the initial Q matrix is .
[0050] In step 4, for the training of the Q matrix, this embodiment converts the IoT device ( ), traditional link ( )、LoRa link( )、Exploration Rate( ), learning rate ( )、Discount rate( ), the maximum range of traditional links ( ) and the minimum distance of the LoRa link ( ) as input and output a small-world wireless mesh network. In each round, the agent selects the first state (traditional link) and uses the working principle of Q learning to use the exploration rate ( ) to decide the corresponding actions (LoRa link) for exploration and development. First, a random number is selected in [0,1] ,if , the agent will choose to perform a random action (exploration), otherwise the agent will choose the best action (exploitation) based on the Q value. Then replace the selected traditional link with the LoRa link to construct a new network. For the new network, calculate the average path length of the IoT devices connected to the selected traditional link ( ), average clustering coefficient ( ) and node degree ( ), and the remaining energy of IoT devices connected to the LoRa link ( ). Then, the reward or penalty is calculated according to formula (1):
[0051] (1)
[0052] in, and They are the node degrees of IoT devices connected to the current traditional links, and is the remaining energy of the IoT devices connected to the selected LoRa link. The main reason for considering the node degree of IoT devices connected to the traditional link is to avoid the IoT devices from being disconnected from the network when creating a SW-WMN. In addition, the reason for selecting the energy level of IoT devices connected to the LoRa link is to improve the network lifetime. According to the reward or penalty, the state-action pair can be updated using formula (2): Q value:
[0053] (2)
[0054] Among them, the learning rate coefficient \eta \in \left [ {0,1} \right ] , used to control the learning speed. The faster the learning speed, the faster the Q table converges and the greater the possibility of oscillation. Discount factor \mu \in \left [ {0,1} \right ] , used to characterize the foresight of the learning system. The larger the value, the more the system pays attention to the impact of future states. Represents the reward for the target action at the current state. The key point to note is that the target action is only used to update the Q value of the current state, it may not be the final target action.
[0055] In step six, use and The small-world network parameters verify the creation of SW-WMN. In this work, we use equations (3) and (4) to verify the formation of SW-WMN:
[0056] > (3)
[0057] < (4)
[0058] in, and denote the average clustering coefficient and average path length after replacing n traditional links with LoRa links, respectively. If either condition fails, a SW-WMN is formed. Once the SW-WMN is formed, we are able to evaluate the network’s lifetime, residual energy, and packet delivery rate to verify the performance improvement of the proposed method.
[0059] In order to fully verify the performance of the proposed method, we use MATLAB software to conduct simulation experiments and compare it with other standard methods, such as Low Energy Adaptive Clustering Hierarchical Protocol (LEACH), Modified Low Energy Adaptive Clustering Hierarchical Protocol (Modified LEACH), single-hop direct data transmission, and traditional multi-hop data transmission. The effectiveness of different methods is evaluated from the perspective of energy efficiency. The simulation results are as follows:
[0060] In the experiment, 300 IoT devices are randomly distributed in km 2 The IoT devices are placed in the same area and a gateway is placed in the center of the area. All IoT devices have the same initial energy of 72000J, and the gateway is always plugged into an uninterruptible power supply. During each iteration, each IoT device can transmit data directly to the gateway or transmit data through a multi-hop method using the LoRaWAN protocol. The data packet size for transmission is 300 bits, and the maximum range of the traditional link is m, the minimum distance of the LoRa link m, the number of iterations of data transmission is set to 20000 times.
[0061] (1) Network life cycle
[0062] The number of iterations that the network can successfully transmit data to the gateway is used as a measure of the network lifespan. In this case, the lifespan of the network is directly proportional to the number of active IoT devices and inversely proportional to the number of dead IoT devices. Figure 4 It can be seen that the number of dead nodes quickly reaches the maximum value for the networks using single-hop direct data transmission, LEACH protocol, and improved LEACH protocol. From 9000 to 20000 iterations, the mortality rate of IoT devices reaches 99% for single-hop direct data transmission, 98% for LEACH protocol transmission, and 98.3% for improved LEACH protocol transmission. In addition, the mortality rate of the proposed method is slightly better at 59.3% compared to 82.7% for the traditional multi-hop transmission strategy. This improvement in performance can be attributed to the introduction of small-world characteristics in the proposed method, which leads to a longer network lifetime compared to other routing techniques even after a large number of data transmission iterations.
[0063] (2) Residual energy
[0064] The residual energy of the network is defined as the cumulative sum of the residual energies of all IoT devices in a given iteration. Figure 5As shown in the figure, for single-hop direct data transmission, LEACH protocol and improved LEACH protocol, the residual energy of the network is significantly reduced, while the traditional multi-hop algorithm and the proposed method retain 13.8% and 38% of the residual energy respectively. The present invention achieves a low average path length of the network by introducing the small-world feature and reduces the number of intermediate hops required for data transmission, which not only improves the data transmission efficiency of IoT devices, but also saves the energy consumption of the devices. A network with high residual energy has a positive impact on the network life, which in turn can improve network reliability and reduce maintenance costs.
[0065] (3) Packet delivery rate
[0066] The packet delivery rate is defined as the ratio of the sum of packets successfully received at the gateway to the sum of packets from all IoT devices. Figure 6 As shown in the figure, for single-hop direct data transmission, LEACH protocol and improved LEACH protocol, the packet delivery rate of the overall network drops rapidly, while the packet delivery rates of the traditional multi-hop algorithm and the proposed method can be maintained at 17.3% and 40.7% respectively. Since single-hop direct data transmission, LEACH protocol and improved LEACH protocol have higher requirements on network energy consumption, as the entire network energy is exhausted, many IoT devices fail to send their data packets to the gateway. It can be seen that the packet delivery rate is positively correlated with the network lifetime. The proposed method has more active nodes, which means that it can run longer with lower risks of service interruption and data loss.
[0067] The above are only specific embodiments of the present invention, and the protection scope of the present invention is not limited thereto. Any modification, equivalent replacement or improvement made within the technical concept of the present invention shall be included in the protection scope of the claims of the present invention.
Claims
1. A method for building an energy-saving small-world wireless Mesh network based on reinforcement learning. It is characterized in that IoT devices are randomly distributed in km 2 in the region, where and are the length and width of the area respectively. The location of the gateway is fixed at the center of the entire area. All IoT devices in the network have the same initial energy, and the gateway is always powered on. Each device in the network knows its location, neighbor list and distance, remaining energy, and queue size. Depending on the device specifications and network specifications, each device sends data packets to the gateway directly or through multiple hops using the LoRaWAN protocol. The gateway sends the received data packets to the network layer server using Transmission Convergence Protocol / Internet Protocol (TCP / IP), and finally the data packets are sent to the application layer server.
2. According to claim 1, a method for constructing an energy-saving small-world wireless Mesh network based on reinforcement learning, It is characterized in that Small-world wireless mesh network graphs can be represented ,in is a collection of N IoT devices, Represents a set of M gateways deployed in the network. represents the set of all possible traditional links, that is, , represents the Euclidean distance between the a-th IoT device and the b-th IoT device, is the maximum range of a conventional link. Similarly, It represents the set of all possible LoRa links introduced in the network, where represents the minimum distance of the LoRa link, and > .gather This includes the legacy links that have not been replaced and the additional LoRa links that have been introduced. In particular, For a conventional wireless mesh network, it is an empty set.
3. According to the energy-saving small-world wireless mesh network construction method based on reinforcement learning in claim 1, since this work needs to introduce small-world features by replacing a part of traditional links with LoRa links, a reinforcement learning method is proposed to learn all traditional links and LoRa link pairs to add weights. Features The steps include: Step 1: Connect IoT devices and central area gateways with traditional links to create a wireless Mesh network; Step 2: Get the distance (Include All possible traditional links within ) and distances greater than (Include ) are stored in space and space middle; Step 3: Initialize the Q matrix; Step 4: Iterate and train the Q matrix; Step 5: Sort the state-action pairs in descending order of the Q values in the trained Q matrix; Step 6: Create a small-world wireless mesh network based on the Q matrix, select the state-action pairs with the highest Q value in the Q matrix in turn, and replace the traditional links with LoRa links; Step 7: Calculate the life cycle, remaining energy and packet delivery rate of the small-world wireless mesh network.
4. According to claim 3, a method for constructing an energy-saving small-world wireless Mesh network based on reinforcement learning, It is characterized in that When creating a small-world wireless mesh network based on the Q-learning framework, the gateway is defined as an intelligent agent, the replaced traditional link is considered as a state, the LoRa link is considered as an action, the reward represents the network's evaluation of the action performed by the intelligent agent in the previous round, and a weighted function is used to balance the network energy efficiency.
5. According to claim 3, a method for constructing an energy-saving small-world wireless Mesh network based on reinforcement learning, It is characterized in that In step 3, since the correspondence between the traditional link and the LoRa link is regarded as a state-action pair, the dimension of the initialization Q matrix is In step 4, for the training of the Q matrix, the present invention converts the IoT device ( ), traditional link ( )、LoRa link( )、Exploration Rate( ), learning rate ( )、Discount rate( ), the maximum range of traditional links ( ) and the minimum distance of the LoRa link ( ) as input and output a small-world wireless mesh network. In each round, the agent selects the first state (traditional link) and uses the working principle of Q learning to use the exploration rate ( ) to decide the corresponding actions (LoRa link) for exploration and development. First, a random number is selected in [0,1] ,if , the agent will choose to perform a random action (exploration), otherwise the agent will choose the best action (exploitation) according to the Q value. Then replace the selected traditional link with the LoRa link to construct a new network.
6. According to claim 5, a method for constructing an energy-saving small-world wireless Mesh network based on reinforcement learning, It is characterized in that For the new network, calculate the average path length of IoT devices connected to the selected legacy links ( ), average clustering coefficient ( ) and node degree ( ), and the remaining energy of IoT devices connected to the LoRa link ( ). Then, the reward or penalty is calculated according to formula (1): (1) in, and They are the node degrees of IoT devices connected to the current traditional links, and is the remaining energy of the IoT devices connected to the selected LoRa link. The main reason for considering the node degree of IoT devices connected to the traditional link is to avoid the IoT devices from being disconnected from the network when creating a SW-WMN. In addition, the reason for selecting the energy level of IoT devices connected to the LoRa link is to improve the network lifetime.
7. According to claim 6, a method for constructing an energy-saving small-world wireless Mesh network based on reinforcement learning, It is characterized in that According to the reward or penalty, the state-action pair can be updated using formula (2): Q value: (2) Among them, the learning rate coefficient \eta \in \left [ {0,1} \right ] , used to control the learning speed. The faster the learning speed, the faster the Q table converges and the greater the possibility of oscillation. Discount factor \mu \in \left [ {0,1} \right ] , used to characterize the foresight of the learning system. The larger the value, the more the system pays attention to the impact of future states. Represents the reward for the target action at the current state. The key point to note is that the target action is only used to update the Q value of the current state, it may not be the final target action.
8. According to claim 3, a method for constructing an energy-saving small-world wireless Mesh network based on reinforcement learning, It is characterized in that In step 6, use and The small-world network parameters verify the creation of SW-WMN. In this work, we use equations (3) and (4) to verify the formation of SW-WMN: > (3) < (4) in, and denote the average clustering coefficient and average path length after replacing n traditional links with LoRa links, respectively. If either condition fails, a SW-WMN is formed. Once the SW-WMN is formed, we are able to evaluate the network’s lifetime, residual energy, and packet delivery rate to verify the performance improvement of the proposed method.
