A wireless sensor network deployment method based on deep reinforcement learning

Abstract

The present invention discloses a wireless sensor network deployment method based on deep reinforcement learning. First, a sensor network deployment simulation environment is built, and then the state space, sensor action space, state transfer function and reward in the deep reinforcement learning environment are designed; using the built environment, a controller is constructed based on a soft actor-critic algorithm, and training is performed in the built environment; the final deployment result is used as the deployment parameter of the sensor network. The method of the present invention can ensure the largest possible coverage area, network connectivity and reduce network energy consumption.

Description

Technical Field

[0001] The invention belongs to the technical field of sensors, and in particular relates to a wireless sensor network deployment method. Background Art

[0002] A wireless sensor network is a distributed communication network in which each sensor node can observe the surrounding environment. Information is transmitted between nodes and between nodes and base stations through wireless links in a multi-hop manner. This decentralized network model has been widely used in military, emergency rescue and other fields.

[0003] However, the deployment of wireless sensor networks (WSNs) is still a challenging problem today. Most of the existing wireless sensor network deployment algorithms are heuristic algorithms, but most of the heuristic algorithms are relatively intuitive methods and it is difficult to achieve satisfactory performance when deployed on 3D terrain. Therefore, deep reinforcement learning is introduced to optimize the deployment of sensor networks.

[0004] Among deep reinforcement learning algorithms, the Soft Actor-Critic (SAC) algorithm has good stability and exploration performance. Summary of the invention

[0005] In order to overcome the shortcomings of the prior art, the present invention provides a wireless sensor network deployment method based on deep reinforcement learning, firstly, a sensor network deployment simulation environment is built, then the state space, sensor action space, state transfer function and reward in the deep reinforcement learning environment are designed; using the built environment, a controller is constructed based on the soft actor-critic algorithm, and training is performed in the built environment; the final deployment result is used as the deployment parameter of the sensor network. The method of the present invention can ensure the largest possible coverage area, network connectivity and reduce network energy consumption.

[0006] The technical solution adopted by the present invention to solve the technical problem includes the following steps:

[0007] Step 1: Build a sensor network deployment simulation environment;

[0008] Restore the terrain of the target area and set the simulation area; calculate the path loss between sensors, and determine whether the sensors can communicate with each other based on the sensor transmission power and path loss;

[0009] Step 2: Design the state space S, sensor action space A, discount factor γ, state transfer function, and reward function R in the deep reinforcement learning environment;

[0010] Step 2-1: The state space S is as follows:

[0011]

[0012] The state space S consists of five parts, each of which is represented by a two-dimensional matrix, namely the network coverage of the target area C, the horizontal deployment position of each sensor L, the power PW of each sensor, the height of each sensor from the ground H, and the location of the currently operating sensor G. Each sensor and its corresponding parameters have the same position in the matrix;

[0013] Step 2-2: The actions that the sensor can choose for each action are as follows:

[0014] The sensor can choose to adjust its own position or transmission power; the position change of the sensor consists of two parts, namely the moving direction and the moving distance. The moving direction can be selected from eight directions separated by 45° on the horizontal plane and two vertical directions, and the maximum moving distance is limited to 1 / 4 of the action range of the corresponding direction;

[0015] The movement of the sensor is restricted to the target area, and movement beyond the target area boundary is invalid.

[0016] The sensor's height from the ground is limited to a set range, and actions beyond this range are invalid;

[0017] Step 2-3: Design reward function;

[0018] R=α1Δf CV -α2ΔP c -α3Δpower (2)

[0019] Where Δf CV is the change in coverage of the target area, ΔP c is the change in network connectivity, Δpower is the change in total network signal transmission power, α1, α2 and α3 are all constants used to adjust the proportion of each item;

[0020] Step 2-4: Set the discount factor γ;

[0021] Step 3: Use the environment built in step 2 to build a controller based on the soft actor-critic algorithm and train it in the built environment; the final deployment results are used as the deployment parameters of the sensor network, as follows:

[0022] Step 3-1: The controller includes a memory pool D and a neural network. The memory pool is used to store past experiences. The input of the neural network is the state s, s∈S, and the output is the sensor action a, a∈A.

[0023] The neural network consists of five networks with the same structure but different parameters, namely: an action network with parameter φ for outputting sensor actions and four evaluation networks with outputting action evaluation values. The four evaluation networks include a target evaluation network with two parameters θ1 and θ2 and two parameters and The main evaluation network of , all the network parameters are randomly initialized;

[0024] Step 3-2: During training, initialize the deployment parameters of the sensor to obtain the state s, input it into the action network, and the action network outputs the selection probability of each action. According to the probability, the action of the sensor is selected and executed to obtain the new state s_ and reward r of the simulation environment. Store <s,a,r,s_,done> in the memory pool, where done indicates whether the round of training is completed;

[0025] Step 3-3: After the memory pool is full, some memories are randomly extracted from it to train the neural network. The action network loss function is as follows. The neural network loss value is calculated based on the experience taken out of the memory pool:

[0026]

[0027] Where D is the memory pool, π t (s t ) is the action output by the strategy at time t, α is the weight coefficient of entropy, π φ (s t ) represents the action output by the current strategy, Q θ (s t ) is the action evaluation value at time t output by the main evaluation network;

[0028] The target evaluation network loss function is as follows, which calculates the neural network loss value based on the experience taken out of the memory pool:

[0029]

[0030] Among them, D represents the memory pool, Q θ (s t ,a t ) is the evaluation value of the main evaluation network for the action performed at time t, r(s t ,a t ) is the action a executed at time t t After the reward, is the new state s after executing the action at time t t+1 the value of

[0031] Step 3-4: Both the action network and the target evaluation network are updated using stochastic gradient descent;

[0032] The main evaluation network is updated as follows

[0033]

[0034] Among them, τ is a constant used to control the update speed of the main evaluation network;

[0035] Step 3-5: Repeat steps 3-2 to 3-4 until the training is completed.

[0036] Preferably, the restoration of the topography of the target area is performed using a digital elevation model (DEM).

[0037] Preferably, the neural network structure comprises three convolutional layers, each convolutional layer is followed by a pooling layer, followed by three fully connected layers, and all activation functions use ReLu.

[0038] Preferably, γ=0.99.

[0039] The beneficial effects of the present invention are as follows:

[0040] The sensor network deployment algorithm of the present invention provides a set of new methods for building a simulation environment and optimizing sensor network deployment parameters based on deep reinforcement learning, wherein a convolutional neural network is used to extract features from multiple matrices and multiple rounds of iterations are performed during training. This method can ensure the largest possible coverage area, network connectivity, and reduce network energy consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 The figure is a flow chart of the method of the present invention.

[0042] Figure 2 This is a block diagram of the wireless sensor network deployment method of the present invention. DETAILED DESCRIPTION

[0043] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0044] The present invention provides a wireless sensor deployment environment construction and a sensor network deployment location and deployment power adjustment method based on deep reinforcement learning, so that the sensor network can obtain the maximum coverage and connectivity and minimize the overall network energy consumption.

[0045] A wireless sensor network deployment method based on deep reinforcement learning includes the following steps:

[0046] Step 1: Build a sensor network deployment simulation environment;

[0047] Use the Digital Elevation Model (DEM) to restore the terrain of the target area and set the simulation area; calculate the path loss between sensors and determine whether the sensors can communicate with each other based on the sensor transmission power and path loss;

[0048] Step 2: Design the state space S, sensor action space A, discount factor γ, state transfer function, and reward function R in the deep reinforcement learning environment;

[0049] Step 2-1: The state space S is as follows:

[0050]

[0051] The state space S consists of five parts, each of which is represented by a two-dimensional matrix, namely the network coverage of the target area C, the horizontal deployment position of each sensor L, the power PW of each sensor, the height of each sensor from the ground H, and the location of the currently operating sensor G. Each sensor and its corresponding parameters have the same position in the matrix;

[0052] Step 2-2: The actions that the sensor can choose for each action are as follows:

[0053] The sensor can choose to adjust its own position or transmission power; the position change of the sensor consists of two parts, namely the moving direction and the moving distance. The moving direction can be selected from eight directions separated by 45° on the horizontal plane and two vertical directions, and the maximum moving distance is limited to 1 / 4 of the action range of the corresponding direction;

[0054] The movement of the sensor is restricted to the target area, and movement beyond the target area boundary is invalid.

[0055] The sensor's height from the ground is limited to a set range, and actions beyond this range are invalid;

[0056] Step 2-3: Design rewards;

[0057] R=α1Δf CV -α2ΔP c -α3Δpower (2)

[0058] Where Δf CV is the change in coverage of the target area, ΔP c is the change in network connectivity, Δpower is the change in total network signal transmission power, α1, α2 and α3 are all constants used to adjust the proportion of each item;

[0059] Step 2-4: Set the discount factor γ to 0.99;

[0060] Step 3: Use the environment built in step 2 to build a controller based on the soft actor-critic algorithm and train it in the built environment; the final deployment results are used as the deployment parameters of the sensor network, as follows:

[0061] Step 3-1: The controller includes a memory pool D and a neural network. The memory pool is used to store past experiences. The input of the neural network is the state s, s∈S, and the output is the sensor action a, a∈A.

[0062] The neural network consists of five networks with the same structure but different parameters, namely: an action network with parameter φ for outputting sensor actions and four evaluation networks with outputting action evaluation values. The four evaluation networks include a target evaluation network with two parameters θ1 and θ2 and two parameters and The main evaluation network of , all the network parameters are randomly initialized;

[0063] Step 3-2: During training, initialize the deployment parameters of the sensor to obtain the state s, input it into the action network, and the action network outputs the selection probability of each action. According to the probability, the action of the sensor is selected and executed to obtain the new state s_ and reward r of the simulation environment. Store <s,a,r,s_,done> in the memory pool, where done indicates whether the round of training is completed;

[0064] Step 3-3: After the memory pool is full, some memories are randomly extracted from it to train the neural network. The action network loss function is as follows. The neural network loss value is calculated based on the experience taken out of the memory pool:

[0065]

[0066] The target evaluation network loss function is as follows, which calculates the neural network loss value based on the experience taken out of the memory pool:

[0067]

[0068] Step 3-4: Both the action network and the target evaluation network are updated using stochastic gradient descent;

[0069] The main evaluation network is updated as follows

[0070]

[0071] Among them, τ is a constant used to control the update speed of the main evaluation network;

[0072] Step 3-5: Repeat steps 3-2 to 3-4 until the training is completed.

Claims

1. A wireless sensor network deployment method based on deep reinforcement learning, characterized in that: The steps include: Step 1: Build a sensor network deployment simulation environment; Restore the terrain of the target area and set the simulation area; calculate the path loss between sensors, and determine whether the sensors can communicate with each other based on the sensor transmission power and path loss; Step 2: Design the state space S, sensor action space A, discount factor γ, state transfer function, and reward function R in the deep reinforcement learning environment; Step 2-1: The state space S is as follows: The state space S consists of five parts, each of which is represented by a two-dimensional matrix, namely the network coverage of the target area C, the horizontal deployment position of each sensor L, the power PW of each sensor, the height of each sensor from the ground H, and the location of the currently operating sensor G. Each sensor and its corresponding parameters have the same position in the matrix; Step 2-2: The actions that the sensor can choose for each action are as follows: The sensor can choose to adjust its own position or transmission power; the position change of the sensor consists of two parts, namely the moving direction and the moving distance. The moving direction can be selected from eight directions separated by 45° on the horizontal plane and two vertical directions, and the maximum moving distance is limited to 1 / 4 of the action range of the corresponding direction; The movement of the sensor is limited to the target area, and movement beyond the boundary of the target area is invalid; The sensor's height from the ground is limited to a set range, and actions beyond this range are invalid; Step 2-3: Design reward function; R=α1△f CV -α2△P c -α3△power (2) In the formula, △f CV is the change in coverage of the target area, △P c is the change in network connectivity, △power is the change in total network signal transmission power, α1, α2 and α3 are all constants used to adjust the proportion of each item; Step 2-4: Set the discount factor γ; Step 3: Use the environment built in step 2 to build a controller based on the soft actor-critic algorithm and train it in the built environment; the final deployment results are used as the deployment parameters of the sensor network, as follows: Step 3-1: The controller includes a memory pool D and a neural network. The memory pool is used to store past experiences. The input of the neural network is the state s, s∈S, and the output is the sensor action a, a∈A. The neural network consists of five networks with the same structure but different parameters, namely: an action network with parameter φ for outputting sensor actions and four evaluation networks with outputting action evaluation values. The four evaluation networks include a target evaluation network with two parameters θ1 and θ2 and two parameters and The main evaluation network of , all the network parameters are randomly initialized; Step 3-2: During training, initialize the deployment parameters of the sensor to obtain the state s, input it into the action network, and the action network outputs the selection probability of each action. According to the probability, the action of the sensor is selected and executed to obtain the new state s_ and reward r of the simulation environment.<s,a,r,s_,done> Stored in the memory pool, where done indicates whether the round of training is finished; Step 3-3: After the memory pool is full, some memories are randomly extracted from it to train the neural network. The action network loss function is as follows. The neural network loss value is calculated based on the experience taken out of the memory pool: Where D is the memory pool, π t (s t ) is the action output by the strategy at time t, α is the weight coefficient of entropy, π φ (s t ) represents the action output by the current strategy, Q θ (s t ) is the action evaluation value at time t output by the main evaluation network; The target evaluation network loss function is as follows, which calculates the neural network loss value based on the experience taken out of the memory pool: Among them, D represents the memory pool, Q θ (s t ,a t ) is the evaluation value of the main evaluation network for the action performed at time t, r(s t ,a t ) is the action a executed at time t t After the reward, is the new state s after executing the action at time t t+1 the value of Step 3-4: Both the action network and the target evaluation network are updated using stochastic gradient descent; The main evaluation network is updated as follows Among them, τ is a constant used to control the update speed of the main evaluation network; Step 3-5: Repeat steps 3-2 to 3-4 until the training is completed.

2. According to claim 1, a wireless sensor network deployment method based on deep reinforcement learning is characterized in that: The topography of the target area is restored using a digital elevation model (DEM).

3. According to claim 1, a wireless sensor network deployment method based on deep reinforcement learning is characterized in that: The neural network structure includes three convolutional layers, each of which is followed by a pooling layer, followed by three fully connected layers, and all activation functions use ReLu.

4. According to claim 1, a wireless sensor network deployment method based on deep reinforcement learning is characterized in that: The γ=0.99.

Images