Distributed WSN energy-efficient routing optimization method based on multi-agent reinforcement learning

Abstract

This invention discloses a distributed WSN energy-efficiency routing optimization method based on multi-agent reinforcement learning. Agents with decision-making modules are distributedly deployed at each sensor node to construct a routing collaborative decision-making system. All agents make decisions synchronously, and each agent's decision module periodically calculates a routing strategy vector based on its local observation value vector and position vector. The decision modules are trained using a multi-agent reinforcement learning algorithm and then deployed in the wireless sensor network. After deployment, the routing strategy is periodically updated using the routing collaborative decision-making system. This method comprehensively considers energy consumption and energy balance, achieving loop-free forwarding path planning for data packets in a fully distributed manner, thereby improving network lifespan.

Description

Technical Field

[0001] The present invention belongs to the field of communication technology, and more specifically, relates to a distributed WSN energy efficiency routing optimization method based on multi-agent reinforcement learning. Background Art

[0002] Wireless Sensor Network (WSN) is the foundation of the Internet of Things. It can realize real-time monitoring of environmental conditions through wireless communication between sensors. It has received widespread attention in recent years and has been widely used in industries such as industry, agriculture, and military.

[0003] WSNs typically deploy battery-powered sensor nodes in a distributed fashion in unattended environments to continuously monitor the surroundings. Whenever sensors collect critical data, the data needs to be packaged and sent to a sink node, acting as a gateway, for further processing and analysis. Due to the limited resources of sensor nodes, improving network energy efficiency and extending network lifespan has always been a key challenge in WSNs.

[0004] Research shows that WSN energy consumption is primarily concentrated in the communication phase. Therefore, designing an effective routing solution that plans energy-efficient forwarding paths for data packets to extend network lifespan is an effective approach. On the one hand, improperly planned transmission paths can cause data packets to be forwarded multiple times within the network, resulting in additional energy consumption. On the other hand, transmission path planning also needs to consider the energy balance of nodes within the network, minimizing the risk of low-energy nodes carrying excessive forwarding loads, which could lead to premature energy depletion. This, in turn, leaves a large number of nodes with surplus energy, resulting in incomplete energy utilization.

[0005] Given the distributed nature of WSN nodes, to avoid the significant energy and time overhead associated with control signaling, routing algorithms must make decisions based on limited local information when nodes need to forward data. The lack of global information can lead to routing decisions falling into local optima. Furthermore, this distributed decision-making approach based on incomplete information can easily lead to routing loops, resulting in significant energy loss for nodes along the loop. Summary of the Invention

[0006] The purpose of the present invention is to overcome the shortcomings of the existing technology and provide a distributed WSN energy efficiency routing optimization method based on multi-agent reinforcement learning, which comprehensively considers energy consumption and energy balance, realizes the joint planning of loop-free forwarding paths of data packets in a fully distributed manner, and improves network survival time.

[0007] To achieve the above-mentioned object of the invention, the distributed WSN energy efficiency routing optimization method based on multi-agent reinforcement learning of the present invention comprises the following steps:

[0008] S1: For the wireless sensor network to be deployed, the sink node connected to the power supply is n 0 , the sensor nodes powered by batteries are n i , i=1,2,…,A, A represents the number of sensor nodes powered by batteries; for each sensor node n i , taking the other node sets within its communication range as its neighbor node set

[0009] Each sensor node continuously collects data and packages it to the sink node when key data is collected. The data packet carries a historical path field to record the nodes that have been passed.

[0010] At each sensor node n i Deploy an agent a equipped with a decision module i ,periodically make routing policy vector decisions, and distributely plan the forwarding path for each data packet based on the routing policy vector;

[0011] S2: Construct a routing collaborative decision system, including A decision modules, which are deployed in each sensor node. i The decision interval of the decision module is τ seconds, that is, the routing strategy is updated every τ seconds to generate a probability-based routing strategy vector in Indicates the sensor node n after the routing strategy is updated i Select node n j The probability of being the next forwarding node is j = 1, ..., A, and t represents the time when the current routing decision is made. In the interval between two decisions, whenever a data packet arrives at the node, the next hop forwarding node is selected based on the current routing strategy vector sampling. The decision module includes a neural network and a correction module, where:

[0012] The input of the neural network is the local observation vector and position vector Pos i , processed to obtain the initial probability vector And sent to the correction module, where Indicates the sensor node n in this preliminary routing decision i Select node n j The probability of being the next hop forwarding node;

[0013] Local observation vector The determination method is: Get the sensor node n i The amount of data collected from the environment in the past B1 decision intervals Where b1=1,2,…,B1; Get sensor node n i The amount of data forwarded by the relay node to other sensor nodes in the past B2 decision intervals Where b2=1,2,…,B2; Get sensor node n i The remaining energy W at the current moment i,t ; According to the amount of data Data volume and the remaining energy W i,t Normalize each data according to the theoretical maximum value and get the normalized data volume Data volume and remaining energy

[0014]

[0015]

[0016]

[0017] in, and Respectively represent the amount of data Data volume and the remaining energy W i,t The theoretical maximum value of

[0018] Then the normalized data are spliced ​​together to obtain the local observation value vector at time t

[0019] Position vector Pos i The determination method is as follows: establish a two-dimensional rectangular coordinate system with the sink node as the origin, and the sensor node n i Get the coordinate position of the self in this coordinate system Represent the horizontal and vertical coordinates respectively; then get the sensor node n i The distance between the sink node and i , filter out A distances dis i The maximum value max_dis in the coordinates is normalized to obtain the position vector

[0020] The correction module is used to adjust the sensor node n i The set of neighbor nodes For the initial probability vector Make corrections and convert the corrected probability vector As a routing policy vector, the specific method is as follows:

[0021] Based on neighbor set Generate neighbor mask vector The amount Represents node n j For node n i neighbors, the two can communicate with each other; if node n j Not at node n i Within the neighborhood of In addition, setting

[0022] Then calculate the Hadamard product of the preliminary probability vector and the neighbor mask vector to obtain the routing strategy vector Finally, the routing strategy vector is normalized. The specific calculation formula for normalization of each probability component is:

[0023]

[0024] S3: Using multi-agent reinforcement learning algorithm to simulate the routing collaborative decision-making system in a simulation environment i The decision module is trained;

[0025] S4: After the training is completed, the wireless sensor network is deployed. The specific method is as follows:

[0026] S4.1: Write neighbor node information into the configuration file of each sensor node according to the predefined deployment location;

[0027] S4.2: Deploy each sensor node in the actual environment according to the predefined deployment location;

[0028] S4.3: Each sensor node n i Set the counter for the amount of data collected and forwarded data volume counters And initialize its value to 0, where the data volume counter is collected Used to count the number of sensor nodes n in a single decision interval i The amount of data sent that you collected from the environment, and the forwarded data volume counter Used to count the number of sensor nodes n in a single decision interval i The amount of data forwarded by the relay node from other sensor nodes;

[0029] S4.4: The intelligent agent deployed at each sensor node generates an initial routing strategy vector for each sensor node through the decision module;

[0030] S5: Each sensor node continuously monitors the environment and receives data sent by other sensor nodes. Whenever a data packet is generated or arrives, it samples the next-hop forwarding node based on the packet's historical path field and the node's current routing strategy vector, and sends the packet to the next-hop forwarding node. The specific method for sampling the next-hop forwarding node is as follows:

[0031] S5.1: Let the sequence number of the current data packet generated in the system be k and the size be l k , the current node reached is n i , generate the forwarding mask vector based on the historical path field in Indicates that the kth data packet arrives at node n i When the node n is not passed j ; Indicates that the kth data packet arrives at node n i When node n is passed j ;

[0032] S5.2: Calculate node n i Current routing policy vector The forwarding mask vector of the current packet The Hadamard product of The route sampling vector masks the nodes that the packet has passed through in the routing strategy vector, avoiding routing loops in the sampled forwarding path;

[0033] S5.3: Routing sampling vector Normalization is performed according to the following formula:

[0034]

[0035] S5.4: Agent a i Generate a random floating point number in the range of (0,1] and determine whether it falls within the normalized routing sampling vector The interval of the probability cumulative distribution function, the node corresponding to the probability value of the interval Acts as the next hop forwarding node for the data packet;

[0036] S5.5: If the current historical path field is empty, it means that the data packet is from node n. i Generate, then let the amount of collected data Otherwise, it means that the data packet is generated by other nodes, then the forwarding data volume

[0037] S5.6: Set node n i The id is written into the historical path field of the data packet, indicating that it has passed through node n i ;

[0038] S5.7: Node n i Send the data packet to the selected next-hop forwarding node

[0039] S6: Check the remaining energy of each node to determine whether any sensor node has exhausted its energy. If so, the network is considered paralyzed and the routing decision ends. Otherwise, proceed to step S7.

[0040] S7: Determine whether τ seconds have passed since the last decision. If so, proceed to step S8; otherwise, return to step S5.

[0041] S8: Each agent makes a synchronous decision and updates its routing strategy vector. The specific method is as follows:

[0042] S8.1: Each agent a i Get the corresponding sensor node n respectively i The amount of collected data and forwarding data volume

[0043] S8.2: Data volume counter will be collected and forwarded data volume counters Set to zero;

[0044] S8.3: Each agent a i Get their current local observation vectors respectively and position vector Pos i And input the decision module to obtain the routing strategy vector Return to step S5.

[0045] The present invention is based on a distributed WSN energy efficiency routing optimization method based on multi-agent reinforcement learning. The invention deploys agents with decision modules at each sensor node in a distributed manner to build a routing collaborative decision system. All agents adopt a synchronous decision-making method. The decision module of each agent periodically calculates a routing strategy vector locally based on the local observation value vector and position vector. The decision module is trained using a multi-agent reinforcement learning algorithm and then deployed in the wireless sensor network. After the deployment is completed, the routing collaborative decision system is periodically used to update the routing plan.

[0046] The present invention has the following beneficial effects:

[0047] 1) This invention realizes distributed joint routing planning for wireless sensor networks, which has the advantage of multi-sensor collaboration compared with traditional distributed routing algorithms;

[0048] 2) The present invention sets a historical path field in the data packet to record information about the nodes that have been passed. When the data packet arrives at a node and needs to be forwarded, a mask vector is extracted based on the historical path field to mask the nodes that the data packet has passed in the current node routing strategy vector, thereby avoiding routing loops.

[0049] 3) The present invention uses a probability-based routing decision vector, which can spatially disperse the energy consumption of nodes compared to traditional solutions using deterministic strategies, thereby achieving better energy balance;

[0050] 4) This invention uses multi-agent reinforcement learning technology to solve routing problems. Compared with the traditional distributed routing algorithm that makes greedy or heuristic decisions, it can more accurately plan forwarding paths for data packets.

[0051] 5) The training process of the present invention uses global information and designs a reward function that comprehensively considers energy consumption and energy balance. Compared with traditional schemes that only use local information, it can better guide the training of the decision module; at the same time, a corresponding network update scheme is designed to ensure the rationality and effectiveness of training. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 This is a flowchart of a specific implementation of the distributed WSN energy efficiency routing optimization method based on multi-agent reinforcement learning of the present invention;

[0053] Figure 2 is a predefined structural diagram of the wireless sensor network in this embodiment;

[0054] Figure 3 It is a structural diagram of the decision module in the present invention;

[0055] Figure 4 is a structural diagram of the neural network in this embodiment;

[0056] Figure 5 is the structural diagram of the critic network in this embodiment;

[0057] Figure 6 is a flow chart of wireless sensor network deployment in the present invention;

[0058] Figure 7 This is a flow chart of sampling and selecting the next hop forwarding node after the wireless sensor network is deployed in the present invention;

[0059] Figure 8 This is a flow chart of updating the routing strategy vector in the present invention. DETAILED DESCRIPTION

[0060] The following describes the specific embodiments of the present invention in conjunction with the accompanying drawings so that those skilled in the art can better understand the present invention. It should be noted that in the following description, when detailed descriptions of known functions and designs may dilute the main content of the present invention, such descriptions will be omitted here.

[0061] Example

[0062] Figure 1 This is a flow chart of a specific implementation of the distributed WSN energy efficiency routing optimization method based on multi-agent reinforcement learning of the present invention. Figure 1 As shown, the specific steps of the distributed WSN energy efficiency routing optimization method based on multi-agent reinforcement learning of the present invention include:

[0063] S101: Wireless sensor network predefined:

[0064] For the wireless sensor network to be deployed, the sink node connected to the power supply is n 0 , the sensor nodes powered by batteries are n i , i=1,2,…,A, A represents the number of sensor nodes powered by batteries; for each sensor node n i , taking the other node sets within its communication range as its neighbor node set

[0065] Each sensor node continuously collects data and packages it to the sink node when critical data is collected. The data packet carries a historical path field that records the nodes that have been passed, which is used to avoid routing loops when forwarding path sampling.

[0066] At each sensor node n i Deploy an agent a equipped with a decision module i , periodically make routing policy vector decisions, and distribute and jointly plan the forwarding path for each data packet based on the routing policy vector.

[0067] Figure 2 This is a predefined structure diagram of the wireless sensor network in this embodiment. Figure 2 As shown, in this embodiment, the wireless sensor network is to be deployed in an area with a radius of 1000 meters. Figure 2 The radius of each ring is r*200 (r=1, 2, 3, 4, 5) meters. The wireless sensor network includes 1 sink node (numbered 0) and 19 sensor nodes (numbered 1-19). The communication range of each sensor is R=700 meters. The set of other nodes within the communication range of each sensor node is regarded as the neighbor node set. For example, the neighbor node set of sensor node 3 is

[0068] S102: Build a collaborative routing decision-making system:

[0069] Construct a routing collaborative decision-making system, including A decision modules, which are deployed in each sensor node’s intelligent agent a i In the present invention, the decision interval of the decision module is τ seconds, that is, the routing strategy is updated every τ seconds to generate a probability-based routing strategy vector in Indicates the sensor node n after the routing strategy is updated i Select node n j The probability of being the next forwarding node is,j=1,…,A, and t represents the time when the current routing decision is made.,In the interval between two decisions, whenever a data packet arrives at a node, the next hop forwarding node is selected,based on the current routing strategy vector sampling. Figure 3 It is the structural diagram of the decision module in the present invention. Figure 3 As shown, the decision module in the present invention includes a neural network and a correction module, wherein:

[0070] The input of the neural network is the local observation vector and position vector Pos i , processed to obtain the initial probability vector And sent to the correction module, where Indicates the sensor node n in this preliminary routing decision i Select node n j The probability of being the next hop forwarding node.

[0071] Local observation vector The determination method is: Get the sensor node n i The amount of data collected from the environment in the past B1 decision intervals Where b1=1,2,…,B1; Get sensor node n i The amount of data forwarded by the relay node to other sensor nodes in the past B2 decision intervals Where b2=1,2,…,B2; Get sensor node n i The remaining energy W at the current moment i,t ; According to the amount of data Data volume and the remaining energy W i,t Normalize each data according to the theoretical maximum value and get the normalized data volume Data volume and remaining energy

[0072]

[0073]

[0074]

[0075] in, and Respectively represent the amount of data Data volume and the remaining energy W i,t The theoretical maximum value.

[0076] Then the normalized data are spliced ​​together to obtain the local observation value vector at time t In this embodiment, B1=B2=5, so the dimension of the local observation value vector is 11.

[0077] Position vector Pos i The determination method is as follows: establish a two-dimensional rectangular coordinate system with the sink node as the origin, and the sensor node n i Get the coordinate position of the self in this coordinate system Represent the horizontal and vertical coordinates respectively. Then get the sensor node n i The distance between the sink node and i , filter and get A distances dis i The maximum value max_dis in the coordinates is normalized to obtain the position vector

[0078] Figure 4 This is the structure diagram of the neural network in this embodiment. In this embodiment, in order to improve the training efficiency of the decision module, the neural network shared parameters of the decision modules of all agents are set. Figure 4 As shown, the neural network in this embodiment includes a first fully connected layer, a second fully connected layer, a third fully connected layer, a concatenation layer, a fourth fully connected layer and a softmax layer, wherein:

[0079] The first fully connected layer is used to receive the local observation vector And process it, and input the obtained features into the second fully connected layer;

[0080] The second fully connected layer is used to process the received features and input the obtained features into the splicing layer;

[0081] The third fully connected layer is used to receive the position vector Pos i , input the obtained features into the concatenation layer;

[0082] The concatenation layer is used to concatenate the two received features and input the concatenated features into the fourth fully connected layer;

[0083] The fourth fully connected layer is used to process the received features and input the obtained features into the softmax layer;

[0084] The softmax layer is used to generate a preliminary probability vector based on the received features.

[0085] According to the above description, the neural network in this embodiment extracts the local observation value vector O through two fully connected layers. t i The state of the position vector Pos is obtained through a fully connected layer. i The embedded information of the agent identifier is extracted. This separate extraction method can make the extracted features more reasonable and improve the accuracy of the final preliminary probability vector. In this embodiment, all fully connected layers in the neural network use the ReLU (Rectified Linear Unit) activation function with a width of 256; the softmax layer uses the softmax activation function with a width of A.

[0086] The correction module is used to adjust the sensor node n i The set of neighbor nodes For the initial probability vector Make corrections and convert the corrected probability vector As a routing policy vector, the specific method is as follows:

[0087] Based on neighbor set Generate neighbor mask vector The amount Represents node n j For node n i neighbors, the two can communicate with each other; if node n j Not at node n i Within the neighborhood of In addition, setting

[0088] Then calculate the Hadamard product of the preliminary probability vector and the neighbor mask vector to obtain the routing strategy vector Finally, the routing strategy vector is normalized. The specific calculation formula for normalization of each probability component is:

[0089]

[0090] For example, for node n 3 , its neighbor set is The corresponding neighbor mask vector is Assume that the initial probability vector is Then P t3 =[0.43,0.01,0,0,0,0,0.086,0.086,0,0,0.086,0,0,0,0,0.086,0,0,0,0.086]. After normalization, P t 3 =[0.5,0,0,0,0,0,0.1,0.1,0,0,0.1,0,0,0,0,0.1,0,0,0,0.1].

[0091] S103: Training decision module:

[0092] A multi-agent reinforcement learning algorithm is used to train A agents in the routing collaborative decision-making system in a simulation environment. i The decision module is trained.

[0093] For the decision-making module, routing planning in WSN is a problem that requires an online solution and has a strong correlation between previous and subsequent decisions. It can be regarded as a sequential decision-making problem. At the same time, considering that the number of nodes in the network is usually large, the spatial complexity of a single forwarding path increases exponentially with the number of nodes, and the complexity is extremely high. When considering multiple forwarding paths for multiple data packets, the difficulty of the problem increases further. Therefore, in this embodiment, after modeling the problem as a distributed partially observable Markov decision process, the mean field actor-critic framework in the actor-critic multi-agent reinforcement learning algorithm is selected, and the training of the decision-making module is completed by simulation using a simulator. The specific method is as follows:

[0094] For different types of data, the arrival rate and volume of data packets are modeled in advance based on prior knowledge, and the real-world arrival rate and volume are simulated in the simulator. In this embodiment, the arrival of data packets at each sensor node follows a Poisson process with a parameter of 10 seconds, and the data volume follows a uniform distribution of [500 bytes, 1000 bytes].

[0095] The actor-critic framework comprises an actor network and a critic network. The actor network is used to output policies, while the critic network guides the actor network's policy learning during training. During training, the critic network is provided with additional global information, enabling more accurate value estimates of the current network state while increasing the feasibility and stability of training. In this embodiment, the decision-making modules of each agent in the routing collaborative decision-making system constructed in step S102 serve as the actor network, and the critic network is set up to guide the actor network's learning.

[0096] The routing decision process of the agent decision module is modeled as a distributed partially observable Markov decision process. Each time a decision is made, the input vector of each decision module is used as the local observation in the partially observable Markov decision process, and the current node n is used as the local observation in the partially observable Markov decision process. i A single-hop forwarding sequence for multiple packets that need to be forwarded as an action For The effective decision interval of τ seconds reaches node n i The forwarding of M data packets, record in is the routing policy vector P t i The effective decision interval of τ seconds reaches node n i The next hop forwarding node is obtained by sampling the mth data packet. Assume that in this decision interval, node n 3 Forwarded 4 data packets and sent them to n 0 、n 6 、n 7 、n 0 ,but

[0097] The reward function in the Markov decision process consists of three parts: energy loss reward, energy balance reward, and termination reward. The calculation methods are as follows:

[0098] Energy loss bonus The calculation formula is as follows:

[0099]

[0100] W i,t 、W i,t-1 Represents sensor node n i The remaining energy at time t, t-1.

[0101] The energy loss reward represents the energy consumption caused by node forwarding. Since the unit energy consumed by data transmission is quadratically related to the transmission distance, a long transmission distance will result in excessive energy consumption for the node, and the energy loss reward value will be low at this time; conversely, if the energy consumption caused by the transmission is small, the energy loss reward value will be high.

[0102] Energy Balance Rewards The calculation formula is as follows:

[0103]

[0104] in, is the forwarding frequency vector Components in the forwarding frequency vector The calculation is done as follows:

[0105] According to the action Calculate the forwarding frequency vector in For action When forwarding sampling for M different data packets, the sampling falls on The number of packets in the dimension. For example, for the action

[0106] Normalize the forwarding frequency vector to get the forwarding frequency vector Each component f t i,j The calculation formula is:

[0107]

[0108] The normalized forwarding frequency vector is

[0109] The energy balance reward represents the energy balance fluctuation caused by node forwarding. Nodes should try to select nodes with sufficient remaining energy for data forwarding. Therefore, the higher the frequency of selecting nodes with high remaining energy as the next hop node for forwarding, the higher the energy balance reward value.

[0110] Termination Rewards The calculation method is as follows:

[0111] If the network goes down, the reward will be terminated K is the total number of data packets forwarded during normal network operation; otherwise

[0112] In real applications, it is expected that the network survival time is as long as possible, that is, the number of data packets transmitted during the normal operation phase is as large as possible.

[0113] The wireless sensor network is simulated by a simulator, and the simulated data is collected to train the actor-critic network, thereby realizing the training of the decision module.

[0114] Due to the randomness of packet forwarding generation and forwarding paths, the same routing policy vector may be forwarded multiple times when multiple packets arrive. In this case, the policy gradient method cannot be directly applied to update the actor network.

[0115] In addition, when sampling each data packet, in order to avoid routing loops, the forwarding mask vector is used to mask the current routing strategy vector, that is, the actual sampling strategy is the routing sampling vector rather than the routing strategy vector, which will lead to deviations in the sampling strategy and thus deviations in the gradient direction. To solve this problem, an additional forwarding frequency vector is calculated to correct the forwarding frequency vector. The policy deviation introduced by the packet mask vector is corrected by importance sampling. For each packet sampling, the specific calculation method is:

[0116]

[0117] in, is the node n reached within the decision interval i The routing sampling vector of the mth data packet The corresponding component in the routing sampling vector The calculation method is:

[0118] According to the node n reached within the decision interval i The forwarding mask vector is generated based on the historical path field of the mth packet in Indicates that the mth data packet arrives at node n i When the node n is not passed j ; Indicates that the mth data packet arrives at node n i When node n is passed j ; For node n i Current routing policy vector and the forwarding mask vector of the mth packet Calculate the Hadamard product to obtain the routing sampling vector

[0119] I(i,j) is the indicator function. When node n is reached within the decision interval i The mth data packet sample falls on P t i The corresponding Dimension time (i.e. select n j As the next hop node for forwarding), I(i,j)=1, otherwise I(i,j)=0. Assume that node n 3 When forwarding data packets, For the first data packet forwarded in this decision interval, it has passed node n 15 ,at this time After sampling, select n 0 As the next hop node for forwarding,

[0120] Then, the gradient of the actor network corresponding to each training sample is calculated according to the preset objective function J

[0121] To accelerate training, a regularization term based on relative distance is added to the objective function of the actor network during the training phase to reduce the probability that the agent selects a neighbor node that is farther away from the sink node than the current node, thereby reducing the energy consumption of the forwarding path. Therefore, the calculation formula of the objective function of each agent decision module in this embodiment is:

[0122]

[0123] in, represents the objective function of reinforcement learning, which is used to maximize the cumulative expected reward. A(S,a) represents the advantage function calculated based on the current reward value and the state value estimate of the critic network. Represents a regularization term based on relative distance, dis i 、dis j Represents node n i Node n j The distance to the sink node.

[0124] As for the actor network as the decision module, this embodiment adopts the following Figure 4 The structure shown. Figure 5 This is the structure diagram of the critic network in this embodiment. In order to improve the training efficiency, the critic network of all agents is set to share parameters, and the position vector Pos of each node is additionally input. i As a marker to distinguish different agents, according to the requirements of the mean field actor-critic framework, the average action of all neighbors of each node is used as the global state of the agent and input into the critic network. Figure 5 As shown, the critic network in this embodiment includes a first fully connected layer, a second fully connected layer, a first concatenation layer, a third fully connected layer, a fourth fully connected layer, a second concatenation layer, a fifth fully connected layer, and a sixth fully connected layer, wherein:

[0125] The first fully connected layer is used to receive the local observation vector And process it, the features obtained Input the first splicing layer;

[0126] The second fully connected layer is used to receive the current sensor node n i The average action of all neighbor nodes in this decision interval, that is, the average of the forwarding frequency vectors of all neighbor nodes, is processed and the obtained feature Input the first splicing layer;

[0127] The first concatenation layer is used to combine the two received features and Perform splicing and input the spliced ​​features into the third fully connected layer;

[0128] The third fully connected layer is used to process the received features and obtain the features And input the second splicing layer;

[0129] The fourth fully connected layer is used to receive the position vector Pos i , the obtained feature w POS Input the second splicing layer;

[0130] The second concatenation layer is used to combine the two received features and w POS Perform splicing and input the obtained features into the fifth fully connected layer;

[0131] The fifth fully connected layer is used to process the received features and output the obtained features to the linear layer;

[0132] The sixth fully connected layer is used to process the received features to obtain the final evaluation value.

[0133] In this embodiment, the first to fifth fully connected layers of the critic network all use the ReLU activation function with a width of 256, and the sixth fully connected layer uses a linear activation function with a width of 1.

[0134] In this example, the Adam optimizer (with parameters set to 0.9 and 0.99) is used for the training of both the actor and critic networks, and the learning rate used for the actor network is 2×10 -6 , the critic network uses a learning rate of 1×10 -5 , the weight of the regularization term based on relative distance is 1×10 -4 In order to ensure the exploration intensity of the actor network and prevent it from falling into the local optimal solution too early, an entropy regularization term is added to the loss function, and the entropy regularization weight is set to 2×10 -5 .

[0135] S104: Deployment of wireless sensor networks:

[0136] After training is completed, the wireless sensor network is deployed. Figure 6 This is a flow chart of wireless sensor network deployment in the present invention. Figure 6 As shown, the specific steps of deploying the wireless sensor network in the present invention include:

[0137] S601: Generate configuration file:

[0138] Neighbor node information is written into the configuration file of each sensor node according to the predefined deployment location.

[0139] S602: Deploy sensor nodes:

[0140] Deploy each sensor node in the actual environment according to the predefined deployment location.

[0141] S603: Initialize counter:

[0142] Each sensor node n i Set the counter for the amount of data collected and forwarded data volume counters And initialize its value to 0, where the data volume counter is collected Used to count the number of sensor nodes n in a single decision interval i The amount of data sent that you collected from the environment, and the forwarded data volume counter Used to count the number of sensor nodes n in a single decision interval i The amount of data forwarded by the relay node from other sensor nodes.

[0143] S604: Calculate the initial routing policy vector:

[0144] The intelligent agent deployed at each sensor node generates an initial routing strategy vector for each sensor node through the decision module.

[0145] S105: Environmental data monitoring:

[0146] Each sensor node continuously monitors the environment and receives data sent by other sensor nodes. Whenever a data packet is generated or arrives, it selects the next-hop forwarding node based on the historical path field of the data packet and the node's current routing strategy vector sampling, and sends the data packet to the next-hop forwarding node. Figure 7 This is a flow chart of sampling and selecting the next hop forwarding node after the wireless sensor network is deployed in the present invention. For the kth data packet generated in the system, its size is l k , arrive at node n i When sampling, if Figure 7 As shown, the specific steps of sampling and selecting the next hop node in the present invention include:

[0147] S701: Generate mask vector:

[0148] The sequence number of the current data packet generated in the system is k, and the size is l k , the current node reached is n i , generate the forwarding mask vector based on the historical path field in Indicates that the kth data packet arrives at node n i When the node n is not passed j ; Indicates that the kth data packet arrives at node n i When node n is passed j . Assume that the data packet arrives at node n 3, and its history path field is

[10] , then its mask vector is

[0149] S702: Calculate the routing sampling vector:

[0150] Compute node n i Current routing policy vector The forwarding mask vector of the current packet The Hadamard product of The routing sampling vector shields the nodes that the data packet has passed in the routing strategy vector to avoid routing loops in the sampled forwarding path. but

[0151] S703: Normalize the route sampling vector:

[0152] Routing sampling vector Normalization is performed according to the following formula:

[0153]

[0154] After normalization,

[0155] S704: Select forwarding node:

[0156] Agent a i Generate a random floating point number in the range of (0,1] and determine whether it falls within the normalized routing sampling vector The interval of the probability cumulative distribution function, the node corresponding to the probability value of the interval Serves as the next hop forwarding node for the data packet.

[0157] at this time The cumulative probability distribution is [0.56, 0.56, 0.56, 0.56, 0.56, 0.67, 0.78, 0.78, 0.78, 0.89, 0.89, 0.89, 0.89, 0.89, 0.89, 0.89, 0.89, 1]. Assuming the randomly generated floating point number is 0.22, it falls on the routing sampling vector The 0th dimension corresponds to node n 0 .

[0158] S705: Update counter:

[0159] If the current historical path field is empty, it means that the data packet is from node n. i Generate, then let the amount of collected data Otherwise, it means that the data packet is generated by other nodes, then the forwarding data volume

[0160] S706: Update historical path information:

[0161] Node n i The id is written into the historical path field of the data packet, indicating that it has passed through node n i .

[0162] In this embodiment, after passing through node n 3 Afterwards, the historical path field of the data packet is [10,3].

[0163] S707: Sending data packet:

[0164] Node n i Send the data packet to the selected next-hop forwarding node In step S704 in this example, node n 3 Node n is selected 0 , so node n 3 At this time, the data packet is sent to node n 0 .

[0165] S106: Check the remaining energy of each node to determine whether any sensor node has exhausted its energy. If so, the network is considered paralyzed and the routing decision ends. Otherwise, proceed to step S107.

[0166] S107: Determine whether τ seconds have passed since the last decision. If so, proceed to step S108; otherwise, return to step S105.

[0167] S108: Update routing policy vector:

[0168] Each agent makes decisions synchronously and updates its routing strategy vector. Figure 8 This is a flow chart of updating the routing strategy vector in the present invention. Figure 8 As shown, the specific steps of updating the routing strategy vector in the present invention include the following steps:

[0169] S801: Obtain decision information:

[0170] Each agent a i Get the corresponding sensor node n respectively i The amount of collected data and forwarding data volume

[0171] S802: Reset counter:

[0172] The amount of data counters will be collected and forwarded data volume counters Set to zero.

[0173] S803: Agent Synchronous Decision-Making:

[0174] Each agent a i Get their current local observation vectors respectively Position vector Pos i And input the decision module to obtain the routing strategy vector Return to step S105.

[0175] In order to illustrate the technical effect of the present invention, a specific example is used to verify the present invention. Figure 2 The wireless sensor network shown in the figure is shown in the figure. Two distributed routing methods focusing on energy efficiency are selected as comparison methods. Comparison method 1 is the method disclosed in the document "Shah RC, Rabaey J M. Energy aware routing for low energy ad hoc sensor networks [C] / / 2002 IEEE Wireless Communications and Networking Conference Record. WCNC 2002 (Cat. No. 02TH8609). IEEE, 2002, 1: 350-355." (abbreviated as EAR); comparison method 2 is the method disclosed in the document "Ren F, Zhang J, He T, Lin C, Ren SK. EBRP: energy-balanced routing protocol for data gathering in wireless sensor networks. IEEE transactions on parallel and distributed systems. 2011 Jan 28; 22 (12): 2108-25." (abbreviated as EBRP). Then, the network survival time of the present invention and the three comparison methods is compared. For the two selected comparison algorithms, parameter search is performed in the environment in advance, and the performance of their optimal hyperparameters is selected for comparison.

[0176] Table 1 is a comparison table of network survival time of the present invention and two comparative methods.

[0177] Routing Method EAR EBRP The present invention Network survival time / second 891 988 1176

[0178] Table 1

[0179] As shown in Table 1, the present invention can prolong the network survival time. Compared with the heuristic algorithm that uses the optimal parameters after parameter search, the performance improvement reaches 32% and 19% respectively, which proves the effectiveness of the present invention.

[0180] Although the above describes the illustrative specific embodiments of the present invention to facilitate understanding of the present invention by those skilled in the art, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the appended claims, these changes are obvious, and all inventions and creations using the concepts of the present invention are protected.

Claims

1. A distributed WSN energy efficiency routing optimization method based on multi-agent reinforcement learning, characterized by: The following steps are involved: S1: For the wireless sensor network to be deployed, the sink node connected to the power supply is n 0 , the sensor nodes powered by batteries are n i , i=1,2,…,A, A represents the number of sensor nodes powered by batteries; for each sensor node n i , taking the other node sets within its communication range as its neighbor node set Each sensor node continuously collects data and packages it to the sink node when key data is collected. The data packet carries a historical path field to record the nodes that have been passed. At each sensor node n i Deploy an agent a equipped with a decision module i ,periodically make routing policy vector decisions, and distributely plan the forwarding path for each data packet based on the routing policy vector; S2: Construct a routing collaborative decision system, including A decision modules, which are deployed in each sensor node. i The decision interval of the decision module is τ seconds, that is, the routing strategy is updated every τ seconds to generate a probability-based routing strategy vector in Indicates the sensor node n after the routing strategy is updated i Select node n j The probability of being the next forwarding node is j = 1, ..., A, and t represents the time when the current routing decision is made. In the interval between two decisions, whenever a data packet arrives at the node, the next hop forwarding node is selected based on the current routing strategy vector sampling. The decision module includes a neural network and a correction module, where: The input of the neural network is the local observation vector and position vector Pos i , processed to obtain the preliminary probability vector And sent to the correction module, where Indicates the sensor node n in this preliminary routing decision i Select node n j The probability of being the next hop forwarding node; Local observation vector The determination method is: Get the sensor node n i The amount of data collected from the environment in the past B1 decision intervals Where b1=1,2,…,B1; Get sensor node n i The amount of data forwarded by the relay node to other sensor nodes in the past B2 decision intervals Where b2=1,2,…,B2; Get sensor node n i The remaining energy W at the current moment i,t ; According to the amount of data Data volume and the remaining energy W i,t Normalize each data according to the theoretical maximum value and get the normalized data volume Data volume and remaining energy in, and Respectively represent the amount of data Data volume and the remaining energy W i,t The theoretical maximum value of Then the normalized data are spliced ​​together to obtain the local observation value vector at time t Position vector Pos i The determination method is as follows: establish a two-dimensional rectangular coordinate system with the sink node as the origin, and the sensor node n i Get the coordinate position of the self in this coordinate system Represent the horizontal and vertical coordinates respectively; then get the sensor node n i The distance between the sink node and i , filter and get A distances dis i The maximum value max_dis in the coordinates is normalized to obtain the position vector The correction module is used to adjust the sensor node n i The set of neighbor nodes For the initial probability vector Make corrections and convert the corrected probability vector As a routing policy vector, the specific method is as follows: Based on neighbor set Generate neighbor mask vector The amount Represents node n j For node n i neighbors, the two can communicate with each other; if node n j Not at node n i Within the neighborhood of In addition, setting Then calculate the Hadamard product of the preliminary probability vector and the neighbor mask vector to obtain the routing strategy vector Finally, the routing strategy vector is normalized. The specific calculation formula for normalization of each probability component is: S3: Using multi-agent reinforcement learning algorithm to simulate the routing collaborative decision-making system in a simulation environment i The decision module is trained; S4: After the training is completed, the wireless sensor network is deployed. The specific method is as follows: S4.1: Write neighbor node information into the configuration file of each sensor node according to the predefined deployment location; S4.2: Deploy each sensor node in the actual environment according to the predefined deployment location; S4.3: Each sensor node n i Set the counter for the amount of data collected and forwarded data volume counters And initialize its value to 0, where the data volume counter is collected Used to count the sensor nodes n in a single decision interval i The amount of data sent that is collected from the environment, and the forwarded data volume counter Used to count the sensor nodes n in a single decision interval i The amount of data from other sensor nodes forwarded by the relay node; S4.4: The intelligent agent deployed at each sensor node generates an initial routing strategy vector for each sensor node through the decision module; S5: Each sensor node continuously monitors the environment and receives data sent by other sensor nodes. Whenever a data packet is generated or arrives, it samples the next-hop forwarding node based on the packet's historical path field and the node's current routing strategy vector, and sends the packet to the next-hop forwarding node. The specific method for sampling the next-hop forwarding node is as follows: S5.1: Let the sequence number of the current data packet generated in the system be k and the size be l k , the current node reached is n i , generate the forwarding mask vector based on the historical path field in Indicates that the kth data packet arrives at node n i When the node n is not passed j ; Indicates that the kth data packet arrives at node n i When node n is passed j ; S5.2: Calculate node n i Current routing policy vector P t i The forwarding mask vector of the current packet The Hadamard product of The route sampling vector masks the nodes that the packet has passed through in the routing strategy vector, avoiding routing loops in the sampled forwarding path; S5.3: Routing sampling vector Normalization is performed according to the following formula: S5.4: Agent a i Generate a random floating point number in the range of (0,1] and determine whether it falls within the normalized routing sampling vector The interval of the probability cumulative distribution function, the node corresponding to the probability value of the interval Acts as the next hop forwarding node for the data packet; S5.5: If the current historical path field is empty, it means that the data packet is from node n. i Generate, then let the amount of collected data Otherwise, it means that the data packet is generated by other nodes, then the forwarding data volume S5.6: Set node n i The id is written into the historical path field of the data packet, indicating that it has passed through node n i ; S5.7: Node n i Send the data packet to the selected next-hop forwarding node S6: Check the remaining energy of each node to determine whether any sensor node has exhausted its energy. If so, the network is considered paralyzed and the routing decision ends. Otherwise, proceed to step S7. S7: Determine whether τ seconds have passed since the last decision. If so, proceed to step S8; otherwise, return to step S5. S8: Each agent makes a synchronous decision and updates its routing strategy vector. The specific method is as follows: S8.1: Each agent a i Get the corresponding sensor node n respectively i Amount of collected data and forwarding data volume S8.2: Data volume counter will be collected and forwarded data volume counters Set to zero; S8.3: Each agent a i Get their current local observation vectors respectively and position vector Pos i And input the decision module to obtain the routing strategy vector Return to step S5.

2. The distributed WSN energy efficiency routing optimization method according to claim 1 is characterized in that: The neural network in step S2 includes a first fully connected layer, a second fully connected layer, a third fully connected layer, a concatenation layer, a fourth fully connected layer and a softmax layer, wherein: The first fully connected layer is used to receive the local observation vector And process it, and input the obtained features into the second fully connected layer; The second fully connected layer is used to process the received features and input the obtained features into the splicing layer; The third fully connected layer is used to receive the position vector Pos i , input the obtained features into the concatenation layer; The concatenation layer is used to concatenate the two received features and input the concatenated features into the fourth fully connected layer; The fourth fully connected layer is used to process the received features and input the obtained features into the softmax layer; The softmax layer is used to generate a preliminary probability vector based on the received features.

3. The distributed WSN energy efficiency routing optimization method according to claim 1, characterized in that: The training selection of the decision module in step S3 is completed by simulating the mean field actor-critic framework in the actor-critic multi-agent reinforcement learning algorithm using a simulator. The specific method is as follows: For different data types that sensors need to collect, the corresponding probability distribution is designed in advance based on the existing prior knowledge to simulate the amount of data collected by each sensor in the real environment in the simulation environment; The decision modules of each agent in the routing collaborative decision-making system constructed in step S2 are used as the actor network, and a critic network is set up to guide the learning of the actor network; The decision-making process of the agent decision module is modeled as a partially observable Markov decision process, where: The input vector of each decision module is treated as a local observation in a partially observable Markov decision process; Set the current node n i A single-hop forwarding sequence for multiple packets that need to be forwarded as an action For the P t i The effective decision interval of τ seconds reaches node n i The forwarding of M data packets, record in is the routing policy vector P t i The effective decision interval of τ seconds reaches node n i The next hop forwarding node obtained by sampling the mth data packet; The reward function consists of three parts: energy loss reward, energy balance reward, and termination reward. The calculation methods are as follows: Energy loss bonus The calculation formula is as follows: W i,t 、W i,t-1 Represents sensor node n i The remaining energy at time t and t-1; Energy Balance Rewards The calculation formula is as follows: Among them, f t i,j is the forwarding frequency vector F t i =[f t i,1 ,f t i,2 ,…,f t i,A ], the forwarding frequency vector F t i =[f t i,1 ,f t i,2 ,…,f t i,A ] is calculated using the following method: According to the action Calculate the forwarding frequency vector in For action When forwarding sampling for M different data packets, the sampling falls on The number of data packets of the dimension; the forwarding frequency vector is normalized to obtain the forwarding frequency vector F t i =[f t i,1 ,f t i,2 ,…,f t i,A ], where each component f t i,j The calculation formula is: Termination Rewards The calculation method is as follows: If the network goes down, the reward will be terminated K is the total number of data packets forwarded during normal network operation; otherwise Then, the wireless sensor network is simulated by the simulator, and the simulated data is collected to train the actor-critic network, thereby realizing the training of the decision module. During the training process, the actor network gradient is calculated using the following method: The modified forwarding frequency vector is calculated using the following formula: in, is the node n reached within the decision interval i The routing sampling vector of the mth data packet The corresponding component in the routing sampling vector The calculation method is: According to the node n reached within the decision interval i The forwarding mask vector is generated based on the historical path field of the mth packet in Indicates that the mth data packet arrives at node n i When the node n is not passed j ; Indicates that the mth data packet arrives at node n i When node n is passed j ; For node n i Current routing policy vector P t i and the forwarding mask vector of the mth packet Calculate the Hadamard product to obtain the routing sampling vector I(i,j) is the indicator function. When node n is reached within the decision interval i The mth data packet sample falls on P t i The corresponding When the dimension is , I(i,j)=1, otherwise I(i,j)=0; Then, the gradient ▽J of the actor network corresponding to each training sample is calculated according to the preset objective function J.

4. The distributed WSN energy efficiency routing optimization method according to claim 3 is characterized in that: The calculation formula of the objective function is: in, represents the objective function of reinforcement learning, A(S,a) represents the advantage function calculated based on the current reward value and the state value estimate of the critic network, Represents a regularization term based on relative distance, dis i 、dis j Represents node n i Node n j The distance to the sink node.

5. The distributed WSN energy efficiency routing optimization method according to claim 3 is characterized in that: The critic network includes a first fully connected layer, a second fully connected layer, a first concatenation layer, a third fully connected layer, a fourth fully connected layer, a second concatenation layer, a third fully connected layer and a linear layer, wherein: The first fully connected layer is used to receive the local observation vector And process it, the features obtained Input the first splicing layer; The second fully connected layer is used to receive the current sensor node n i The average action of all neighboring nodes in this decision interval, that is, the average of the forwarding frequency vectors of all neighboring nodes, is processed to obtain the feature Input the first splicing layer; The first concatenation layer is used to combine the two received features and Perform splicing and input the spliced ​​features into the third fully connected layer; The third fully connected layer is used to process the received features and obtain the features And input the second splicing layer; The fourth fully connected layer is used to receive the position vector Pos i , the obtained feature w POS Input the second splicing layer; The second concatenation layer is used to combine the two received features and w POS Perform splicing and input the obtained features into the fifth fully connected layer; The fifth fully connected layer is used to process the received features and output the obtained features to the linear layer; The sixth fully connected layer is used to process the received features to obtain the final evaluation value.

Images