High-energy-efficiency multi-hop and power control combined method based on reinforcement learning

Abstract

The invention discloses a reinforcement learning-based high-energy-efficiency multi-hop and power control combined method, and belongs to the field of power transmission line Internet of Things. The method specifically comprises the following steps: establishing a link type wireless sensor network architecture, respectively acquiring own and neighbor observation states st by each sensing node in a current time slot, and then selecting an action at comprising a next hop node and a sending power level; meanwhile, interaction with the environment is executed, and a global utility function rt is obtained; at the end of the current time slot, the quaternion group lt is reserved; c, t, t, st, at, rt, st + 1gt; ; and then, in the current round, the network uses N tetrads in each time slot of memory as a training set, a multi-hop and power control model is trained, parameters of the model are updated and then deployed on sensor nodes, and each sensing node sends a data packet to the selected next-hop node at the selected transmitting power. According to the invention, the transmission path and the transmitting power can be adaptively selected according to the dynamic change of the environment, and intelligent scheduling and resource allocation of system energy efficiency are improved.

Description

Technical Field

[0001] The present invention belongs to the field of power transmission line Internet of Things, and specifically is a high energy-efficient multi-hop and power control joint method based on reinforcement learning. Background Art

[0002] In today's rapidly developing digital age, the Internet of Things (IoT) is rapidly becoming a core component of people's daily lives and industrial automation. Its devices have brought great convenience to people's lives and work through real-time monitoring and data exchange. However, with the exponential growth of IoT devices, the demand for energy is also rising sharply. These devices are often deployed in places where it is difficult to access traditional energy supplies, and many rely on battery power. Therefore, improving the energy efficiency of the Internet of Things is not only to reduce operating costs and maintenance difficulties, but also the key to ensure the long-term stable operation of these devices.

[0003] In the perceptual Internet of Things, intelligent scheduling and resource allocation are one of the key technologies to improve energy efficiency, especially in a multi-hop network structure where data needs to be transmitted to the destination through multiple nodes. Multi-hop selection and power control are particularly important.

[0004] Multi-hop Selection: In a multi-hop network, the transmission path of a data packet is not directly from one node to another, but forwarded through multiple intermediate nodes. The intelligent multi-hop selection algorithm can select the optimal transmission path based on factors such as the current network status, the remaining energy of the node, the communication distance, and the degree of congestion; this can reduce energy waste and data transmission delays caused by improper path selection.

[0005] Power Control: It can dynamically adjust the transmit power of each node according to the communication distance and link quality. When the distance between nodes is close or the channel conditions are good, the transmit power can be reduced to reduce energy consumption. At the same time, power control can also reduce signal interference and improve the overall performance of the network.

[0006] By combining multi-hop selection and power control, the perception of the Internet of Things can achieve more efficient energy utilization and data transmission. For example, in scenarios such as smart grids, environmental monitoring or industrial automation, it can help IoT devices flexibly adjust communication strategies according to actual needs and environmental changes, extend battery life, reduce maintenance costs, and ensure the reliability and real-time nature of data transmission.

[0007] In addition, the perceptual Internet of Things can also be combined with machine learning algorithms to achieve adaptive scheduling and resource allocation, further improving the energy efficiency and intelligence level of the perceptual Internet of Things. Summary of the invention

[0008] In order to solve the problem of severe signal fading and heavy burden on single nodes caused by long-distance transmission of power line networks with multiple sensing nodes, the present invention proposes a high-energy-efficiency multi-hop and power control joint method based on reinforcement learning, which can adaptively select transmission paths and transmission powers according to dynamic changes in the environment, and improve the intelligent scheduling and resource allocation of system energy efficiency.

[0009] The high energy-efficient multi-hop and power control joint method based on reinforcement learning has the following specific steps:

[0010] Step 1: In the transmission line IoT environment, a link-type wireless sensor network architecture including N sensing nodes and 1 aggregation node is constructed; N is a positive integer;

[0011] The sensing node periodically senses the health status of the transmission tower and delivers the sensed data to the sink node;

[0012] The aggregation node collects the health status data of the transmission tower and determines whether the transmission tower is operating normally and whether manual maintenance is required.

[0013] Step 2: When learning a link-type wireless sensor network, at the beginning of the current time slot, each sensor node obtains its own state and the observed state s of its neighboring sensor nodes. t :

[0014] s t = {B(t), B1(t), ..., B m (t),…,B M (t), E(t), E1(t), …, E m (t),…,E M (t)}

[0015] Where B(t) is the data cache state observed by the sensing node agent, E(t) is the observed remaining energy cache state, and B m (t) is the data cache information of neighbor node m obtained after the sensing node communicates with the neighbor node, E m (t) is the remaining energy state information of neighbor node m obtained after the sensing node communicates with the neighbor node.

[0016] Step 3: After each sensing node collects sensing data or receives data sent by a neighboring sensing node in the current time slot, it will t Select action a that includes the next hop node and the transmit power level t , and execute the action;

[0017] a t ~π θ (s t ), π θ(s t )=softmax(θ T φ(s t ))

[0018] φ(s t ) is the observed state s t , θ is the parameter of the policy network, which is subsequently updated according to the environmental feedback reward.

[0019] Step 4: At the same time, the action decisions obtained by each sensor node are executed simultaneously to interact with the environment to obtain the global utility function r t ;

[0020]

[0021] are weight coefficients, r e (t) is the minimum remaining energy reward, r b (t) is the packet retention penalty.

[0022] Step 5: At the end of the current time slot, the observation state s of each sensing node is t , action a t , the global utility function r(t) and the observed state s of the next time slot t+1 , forming the quadruple corresponding to each node t , a t , r t ,s t+1 >, return to step 2 and continue to execute the next time slot.

[0023] Step 6: After all time slots of the current round are executed, the link-type wireless sensor network memorizes the N quadruplets in each time slot as a training set, trains the multi-hop and power control model, and updates the parameters of the model;

[0024] The specific training process is:

[0025] First, a batch is randomly sampled from the link-based wireless sensor network memory cache

[0026] The batch Contains several pairs of quadruple data t , a t , r t ,s t+1 >, the specific number of selected logarithms is set manually according to actual conditions.

[0027] For batches For each pair of quadruplets in , calculate their respective time difference target TD_Target and TD error δ t :​​

[0028] TD_Target = r t +γV φ (s t+1 )×(1-d t+1 )

[0029] δ t =TD_Target-V φ (s t )

[0030] where s t+1 is the state at the next moment, d t+1 is the sign of whether the episode is over, γ is the discount factor, V φ (s t ) is the value function estimate of the current state;

[0031] Then, for the batch For each pair of quadruple in , the GAE method is used to calculate the advantage function of each state-action pair:

[0032]

[0033] where λ is the decay parameter of GAE.

[0034] Next, use the advantage function to calculate the strategy loss L PPO (θ) and the value of network loss L Value (φ):

[0035]

[0036] in, Indicates in batch Medium sampling (s t , a t ) is the expected value of L. CLIP (a t |s t ) represents a given state s t Take action a t The truncated loss function of π θ (a t |s t ) is the state s given by the current policy parameter θ t Take action a t The probability of π old (a t |s t ) represents the old policy parameter θ old Next, given the state s t Take action a t∈ is a preset small positive number, indicating the strictness of truncation.

[0037] Finally, using the policy loss L PPO (θ) and the value of network loss L Value (φ) Perform gradient descent to update the policy network parameters θ and value network parameters φ until convergence to obtain the trained multi-hop and power control model.

[0038]

[0039] Among them, α and β represent the learning rates of the two networks respectively.

[0040] Step 7: Deploy the trained multi-hop and power control model on the sensor nodes of the link-type wireless sensor network. The sensor nodes input the acquired information into the multi-hop and power control model and output the next hop node and the transmission power.

[0041] The information obtained by each sensing node includes: data exchanged with neighboring nodes when the data packet cache is not empty, energy cache information, channel status, electromagnetic interference of transmission lines, its own remaining energy, its own data cache, neighboring node remaining energy and neighboring data cache, etc.

[0042] Step 8: Each sensing node sends a data packet to the selected next hop node with the selected transmission power.

[0043] The advantages of the present invention are:

[0044] 1. A joint method of high-efficiency multi-hop and power control based on reinforcement learning can help nodes to transmit data with high energy efficiency, thereby increasing the network life and improving the overall throughput of the system.

[0045] 2. A high-energy-efficiency multi-hop and power control joint method based on reinforcement learning, which trains each node to learn the optimal next-hop selection and power control strategy under different energy and data packet backlogs. Compared with traditional multi-hop and power control methods, the present invention takes into account the characteristics of long-distance transmission and electromagnetic interference in the transmission line scenario, and jointly optimizes the next-hop node selection and power control. It can intelligently and dynamically adjust the transmission path and power allocation according to the real-time network status and environmental changes, thereby maximizing the energy efficiency and transmission reliability of the entire network, ensuring that key monitoring data can be stably and accurately transmitted to the sink node, which is crucial to ensuring the safe operation of ultra-high voltage transmission lines. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a flow chart of a high energy-efficient multi-hop and power control joint method based on reinforcement learning of the present invention;

[0047] Figure 2 It is a schematic diagram of the architecture of a link-type wireless sensor network in a power transmission line environment constructed by the present invention;

[0048] Figure 3 It is a comparison diagram of the minimum residual energy of the present invention and the multi-hop selection scheme in networks with different numbers of nodes;

[0049] Figure 4 It is a comparison diagram of the minimum residual energy of the present invention in networks with different transmission tower spans and that of the multi-hop selection scheme alone. DETAILED DESCRIPTION

[0050] In order to facilitate those skilled in the art to understand and implement the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. Obviously, the described embodiments are only partial embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work should fall within the scope of protection of the present invention.

[0051] In the prior art, sensing nodes communicate within the range of neighboring sensing nodes in a frequency division multiplexing manner, and use the CSMA / CA protocol to avoid collisions when sending data; at the same time, the sensing nodes collect energy from slowly time-varying solar energy to supplement transmission energy consumption. However, in the scenario of ultra-high voltage transmission lines, the distance between poles and towers of ultra-high voltage lines may be hundreds of meters or even more than one thousand meters. This long-distance transmission will bring a great transmission burden to a single node. At the same time, the dynamic changes in channel states, electromagnetic interference in the transmission line environment, and dynamic fluctuations in energy collection all make the transmission power required to ensure the same communication quality at the receiving node different. The introduction of multi-hop affects the required transmission power by affecting the transmission distance and the size of the transmitted data. The unchanging transmission power may cause energy waste or transmission failure.

[0052] Therefore, it is necessary to design an intelligent scheduling and resource allocation method that can adaptively select transmission paths and transmission powers according to dynamic changes in the environment and improve system energy efficiency.

[0053] The energy-efficient multi-hop and power control joint method based on reinforcement learning, such as Figure 1 As shown, the specific steps are as follows:

[0054] Step 1: In the transmission line IoT environment, build a link-type wireless sensor network architecture including sensing nodes and aggregation nodes;

[0055] like Figure 2As shown in the figure, the link-type wireless sensor network in the transmission line environment includes two types of nodes: sensor nodes and sink nodes; the sensor nodes periodically sense the health status of the transmission towers and deliver the sensed data to the sink nodes; the sink nodes collect the health status data of the transmission towers and determine whether the transmission towers are operating normally and whether manual maintenance is required.

[0056] Step 2: When learning a link-type wireless sensor network, at the beginning of the current time slot, each sensor node obtains its own state and the observed state s of its neighboring sensor nodes. t :

[0057] s t = {B(t), B1(t), ..., B m (t),…,B M (t), E(t), E1(t), …, E m (t),…,E M (t)}

[0058] Where B(t) is the data cache state observed by the node agent, E(t) is the observed remaining energy cache state, and B m (t) is the data cache information of neighbor node m obtained after the node communicates with the neighbor node, E m (t) is the remaining energy state information of neighbor node m obtained after the node communicates with the neighbor node.

[0059] The time measurement in the training phase is divided into rounds, cycles and time slots according to their length;

[0060] Step 3: Each node selects the next hop and the transmission power level when the data buffer is not empty;

[0061] That is, after each sensing node collects sensing data in the current time slot or receives data sent by the neighboring sensing node, it will t Select action a that includes the next hop node and the transmit power level t , and execute the action;

[0062] a t ~π θ (s t ), π θ (s t )=softmax(θ T φ(s t ))

[0063] φ(s t ) is the observed state s t, θ is the parameter of the policy network, which is subsequently updated according to the environmental feedback reward.

[0064] Step 4: At the same time, the action decisions obtained by each sensor node are executed simultaneously to interact with the environment to obtain the global utility function r t ;

[0065]

[0066] are weight coefficients, r e (t) is the minimum remaining energy reward, r b (t) is the packet retention penalty.

[0067] (1) The minimum residual energy of the system is r e (t) = min(E1, E2, ..., E n , ..., E N ). The remaining energy E of node n n The update at time slot t is E n (t) = E n (t-1)-C n，t P n (t)T+(1-C n，t ) eh，n (t)T, where P n (t) is the transmission power determined by node n at time slot t, P eh，n (t) is the energy harvesting power of the node at time slot t, T is the time slot length, C n，t Indicates whether node n has enough energy to transmit at time slot t. If not, energy harvesting is performed.

[0068] (2) The packet retention penalty is expressed as The penalty is determined by the number of packets that are not transmitted to the sink node and remain at the sensing node after each time slot interaction with the environment.

[0069] Step 5: At the end of the current time slot, the observation state s of each sensing node is t , action a t , the global utility function r(t) and the observed state s of the next time slot t+1 , forming the quadruple corresponding to each node t , a t , r t ,s t+1 >, return to step 2 and continue to execute the next time slot.

[0070] ​Step 6: After all time slots of the current round are executed, the link-type wireless sensor network memorizes the N quadruplets in each time slot as a training set, trains the multi-hop and power control model, and updates the parameters of the model;

[0071] When the network is learning, the perception nodes perform coordinated independent training: the agents share the same policy network, but their experience collection and policy update processes are independent. The specific training process is:

[0072] First, a batch is randomly sampled from the link-based wireless sensor network memory cache

[0073] The batch Contains several pairs of quadruple data t , a t , r t ,s t+1 >, the specific number of selected logarithms is set manually according to actual conditions.

[0074] For batches For each pair of quadruplets in , calculate their respective time difference target TD_Target and TD error δ t :

[0075] TD_Target = r t +γV φ (s t+1 )×(1-d t+1 )

[0076] δ t =TD_Target-V φ (s t )

[0077] where s t+1 is the state at the next moment, d t+1 is the sign of whether the episode is over, γ is the discount factor, V φ (s t ) is the value function estimate of the current state;

[0078] Then, for the batch For each pair of quadruple in , the GAE method is used to calculate the advantage function of each state-action pair:

[0079]

[0080] where λ is the decay parameter of GAE.

[0081] Next, use the advantage function to calculate the strategy loss L PPO (θ) and the value of network loss L Value ​(φ):

[0082]

[0083] in, Indicates in batch Medium sampling (s t , a t ) is the expected value of L. CLIP (a t |s t ) represents a given state s t Take action a t The truncated loss function of π θ (a t |s t ) is the state s given by the current policy parameter θ t Take action a t The probability of π old (a t |s t ) represents the old policy parameter θ old Next, given the state s t Take action a t ∈ is a preset small positive number, indicating the strictness of truncation.

[0084] Finally, using the policy loss L PPO (θ) and the value of network loss L Value (φ) Perform gradient descent to update the policy network parameters θ and value network parameters φ until convergence to obtain the trained multi-hop and power control model.

[0085]

[0086] Among them, α and β represent the learning rates of the two networks respectively.

[0087] Step 7: Deploy the trained multi-hop and power control model on the sensor nodes of the link-type wireless sensor network. The sensor nodes input the acquired information into the multi-hop and power control model and output a next-hop node and a transmission power with low energy consumption and a high probability of successful arrival of the data packet at the sink.

[0088] The information obtained by each sensing node includes: data exchanged with neighboring nodes when the data packet cache is not empty, energy cache information, channel status, electromagnetic interference of transmission lines, its own remaining energy, its own data cache, neighboring node remaining energy and neighboring data cache, etc.

[0089] Step 8: Each sensing node sends a data packet to the selected next hop node with the selected transmission power.

[0090] In the present invention, the learning phase, i.e., the state observation phase, is entered upon initialization: at the beginning of each time slot, all sensing nodes observe the state of the nodes in the local network in which they are located, and obtain the node next hop and transmission power selection strategy in this state through exploratory learning; the node periodically generates sensing data, i.e., the node senses data every few time slots. At the beginning of each time slot, each node exchanges state information with neighboring nodes, including data cache and remaining energy information, and makes next hop node selection and transmission power control decisions for the node in this state through the output of the current strategy network, executes actions, interacts with the transmission line environment to obtain rewards or costs to update the strategy network and the value network, and iterates the training process until the network converges stably.

[0091] The next hop and power control selection execution process is as follows:

[0092] Step ①: (Perception data generation phase) In a link-type network of multiple perception nodes, each node periodically generates a perception data packet, which needs to be transmitted to the sink node with the lowest possible energy consumption.

[0093] Step ②: (Deployment model decision phase) When the data packet cache of each sensing node is not empty, it exchanges data and energy cache information with neighboring nodes, inputs information such as channel status, electromagnetic interference of transmission lines, its own remaining energy, remaining energy of neighboring nodes and data cache into the model obtained by the learning module, and uses the model to select a next-hop node and sending power with low energy consumption and a high probability of successful arrival of the data packet at the sink.

[0094] Step ③: (Execution decision phase) Each sensing node sends a data packet to the selected next-hop node with the selected transmission power.

[0095] like Figure 3 As shown in FIG. 1 , this is a comparison diagram of the minimum remaining energy of the scheme proposed in the present invention and the multi-hop selection scheme alone in networks with different numbers of nodes. As the number of sensing nodes increases, the minimum remaining energy of the scheme proposed in the present invention and the multi-hop selection scheme alone decreases accordingly, but the minimum remaining energy of the scheme proposed in the present invention is always higher than that of the multi-hop selection scheme alone, indicating that the method of the present invention can effectively reduce system energy consumption.

[0096] like Figure 4 As shown in the figure, it is a comparison diagram of the minimum residual energy of the scheme proposed in the present invention with the multi-hop selection scheme in different transmission tower span networks. As the transmission tower span increases, the minimum residual energy of the scheme proposed in the present invention decreases accordingly, but the minimum residual energy is always higher than that of the multi-hop selection scheme, indicating that the method of the present invention can effectively reduce system energy consumption.

Claims

1. A joint method of energy-efficient multi-hop and power control based on reinforcement learning, It is characterized in that The specific steps are as follows: Step 1: In the transmission line IoT environment, a link-type wireless sensor network architecture consisting of N sensing nodes and 1 aggregation node is constructed; N is a positive integer; Step 2: When learning a link-type wireless sensor network, at the beginning of the current time slot, each sensor node obtains its own state and the observed state s of its neighboring sensor nodes. t : Step 3: After each sensing node collects sensing data or receives data sent by a neighboring sensing node in the current time slot, it will t Select action a that includes the next hop node and the transmit power level t , and execute the action; a t ~π θ (s t ),p θ (s t )=softmax(θ T φ(s t )) φ(s t ) is the observed state s t , θ is the parameter of the policy network, which is subsequently updated according to the environmental feedback reward; Step 4: At the same time, the action decisions obtained by each sensor node are executed simultaneously to interact with the environment to obtain the global utility function r t ; are weight coefficients, r e (t) is the minimum remaining energy reward, r b (t) is the penalty for data packet retention; Step 5: At the end of the current time slot, the observation state s of each sensing node is t , action a t , the global utility function r(t) and the observed state s of the next time slot t+1 , forming the quadruple corresponding to each node t ,a t ,r t ,s t+1 >, return to step 2 and continue to execute the next time slot;​ Step 6: After all time slots of the current round are executed, the link-type wireless sensor network memorizes the N quadruplets in each time slot as a training set, trains the multi-hop and power control model, and updates the parameters of the model; The specific training process is: First, a batch is randomly sampled from the link-based wireless sensor network memory cache The batch Contains several pairs of quadruple data t ,a t ,r t ,s t+1 >, the specific number of selected logarithms is set manually according to the actual situation;​ For each pair of quads in batch B, calculate their respective temporal difference targets TD_Target and TD errors δ t : TD_Target=r t +γV φ (s t+1 )×(1-d t+1 ) δ t =TD_Target-V φ (s t ) where s t+1 is the state at the next moment, d t+1 is the sign of whether the episode is over, γ is the discount factor, V φ (s t ) is the value function estimate of the current state; Then, for the batch For each pair of quadruple in , the GAE method is used to calculate the advantage function of each state-action pair: Where λ is the attenuation parameter of GAE; Next, use the advantage function to calculate the strategy loss L PPO (θ) and the value of network loss L Value (φ): in, Indicates in batch Medium sampling (s t ,a t )’s expected value; L CLIP (a t |s t ) represents a given state s t Take action a t The truncation loss function of θ (a t |s t ) is the state s given by the current policy parameter θ t Take action a t The probability of old (a t |s t ) represents the old policy parameter θ old Next, given the state s t Take action a t The probability of; ∈ is a preset small positive number, indicating the strictness of truncation; Finally, using the policy loss L PPO (θ) and the value of network loss L Value (φ) Perform gradient descent to update the policy network parameters θ and value network parameters φ until convergence to obtain the trained multi-hop and power control model; Among them, α and β represent the learning rates of the two networks respectively; Step 7: Deploy the trained multi-hop and power control model on the sensor nodes of the link-type wireless sensor network. The sensor nodes input the acquired information into the multi-hop and power control model and output the next hop node and the transmission power. Step 8: Each sensing node sends a data packet to the selected next hop node with the selected transmission power.

2. A high energy efficiency multi-hop and power control joint method based on reinforcement learning as claimed in claim 1, It is characterized in that In the step 1, the sensing node periodically senses the health status of the transmission tower and delivers the sensed data to the aggregation node; after collecting the health status data of the transmission tower, the aggregation node determines whether the transmission tower is operating normally and whether manual maintenance is required.

3. A high energy efficiency multi-hop and power control joint method based on reinforcement learning as claimed in claim 1, It is characterized in that In step 2, the observed state s t The formula is: s t ={B(t),B 1 (t),…,B m (t),…,B M (t),E(t),E 1 (t),...,E m (t),…,E M (t)} Where B(t) is the data cache state observed by the sensing node agent, E(t) is the observed remaining energy cache state, and B m (t) is the data cache information of neighbor node m obtained after the sensing node communicates with the neighbor node, E m (t) is the remaining energy state information of neighbor node m obtained after the sensing node communicates with the neighbor node.

4. The high energy efficiency multi-hop and power control joint method based on reinforcement learning as claimed in claim 1, It is characterized in that In step 4, the minimum remaining energy of the system is: r e (t)=min(E 1 ,E 2 ,…,E n ,…,E N ) E n is the residual energy of node n, and the update formula in time slot t is: E n (t)=E n (t-1)-C n,t P n (t)T+(1-C n,t )P eh,n (t)T Where P n (t) is the transmission power determined by node n at time slot t, P eh,n (t) is the energy harvesting power of the node at time slot t, T is the time slot length, C n,t Indicates whether node n has enough energy to transmit at time slot t. If not, energy harvesting is performed. The packet retention penalty is expressed as: The penalty is determined by the number of packets that are not transmitted to the sink node and remain at the sensing node after each time slot interaction with the environment.

5. The high energy efficiency multi-hop and power control joint method based on reinforcement learning as claimed in claim 1, It is characterized in that In step seven, the information obtained by each sensing node includes: data exchanged with neighboring nodes when the data packet cache is not empty, energy cache information, channel status, electromagnetic interference of transmission lines, its own remaining energy, its own data cache, neighboring node remaining energy and neighboring data cache.

Images