Adaptive selection method of wireless sensor network routing algorithm based on deep reinforcement learning

Abstract

The application discloses an adaptive selection method of a wireless sensor network routing algorithm based on deep reinforcement learning, wherein in the process of executing wireless sensor network data collection, an agent deployed on a sink node inputs the position, residual energy and buffer queue length of each sensor node into a decision network after normalization, and makes a decision on a routing algorithm; considering the energy limitation of the sensor node, the newly decided routing algorithm is informed to each sensor node in the network by the sink node which is rich in energy and computing resources; considering that the energy consumption rate and queuing state of the sensor node are different due to different amounts of forwarded data, the routing algorithm decision adopts an online mode, and the decision network of the agent will make a decision again to select a new routing algorithm after the relative change of the node state reaches a certain degree; and the application can provide timely routing algorithm adjustment for the sensor node according to the change of the energy and queuing state of the sensor node, so that dual optimization of energy efficiency and QoS is realized.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of communication, and more specifically relates to an adaptive selection method of a wireless sensor network routing algorithm based on deep reinforcement learning. BACKGROUND

[0002] As an important part of the Internet of Things (IoT), the wireless sensor network (WSN) is mainly responsible for collecting data and forwarding the sensing data to the core network for further processing. The sensor nodes in the wireless sensor network are usually deployed in unattended places and are powered by batteries to run autonomously to perform tasks with certain performance requirements, such as real-time data collection. Since users expect more data and faster collection speed, energy efficiency and Quality of Service (QoS) are both key indicators in the wireless sensor network.

[0003] In the wireless sensor network, the energy of the sensor nodes is mainly consumed in the wireless communication process. In order to improve the energy efficiency and prolong the network survival time, a series of routing algorithms considering energy indicators are proposed, such as EAR, MLDR and EBRP, etc. On the other hand, with the development of video and picture sensors, emerging applications have put forward very high QoS requirements for corresponding service transmission. In order to ensure the fast forwarding of QoS traffic, a series of routing algorithms considering QoS indicators are proposed, such as SPEED, EAR+, etc.

[0004] Since the optimization directions of energy efficiency and QoS are inconsistent, most algorithms tend to optimize a single target, while performing poorly on the other target. For example, energy-aware routing tends to select nodes with more residual energy for relaying in order to pursue higher node energy efficiency, and more evenly consume the energy of each sensor node in the network. However, this will cause congestion at high-energy nodes, and in turn increase the packet loss rate and end-to-end delay.

[0005] In the wireless sensor network, routing decisions are made based on the current system state (such as the residual energy and queuing state of the sensor nodes). At the same time, by deciding the aggregation path of the data packet, the routing decision will also affect the new system state. Therefore, the system state is dynamically changing, which puts a high adaptability requirement on the design of the routing algorithm.

[0006] The existing methods cannot take into account multiple optimization targets and cannot quickly adapt to the changing environment, so they cannot achieve the dual optimization of energy efficiency and QoS indicators. SUMMARY

[0007] The present application aims to overcome the deficiencies of the prior art, and provide an adaptive selection method of a wireless sensor network routing algorithm based on deep reinforcement learning, which selects a routing algorithm adaptively to respond to the dynamic changes of the system state, so as to realize the dual optimization of energy efficiency and QoS indicators.

[0008] In order to achieve the above-mentioned application purposes, the adaptive selection method of the wireless sensor network routing algorithm based on deep reinforcement learning comprises the following steps:

[0009] (1) In the simulation environment, the agent is trained using the deep reinforcement learning algorithm

[0010] According to the actual deployment, a wireless sensor network composed of A battery-powered sensor nodes n 1 , …, n A and a sink node n 0 connected to a power supply is established in the simulation environment, wherein the sink node n 0 is located at the origin, and each sensor node n i i=1,…,A and the sink node n 0 are deployed according to the actual environment deployment position in the simulation environment;

[0011] For the sensor node n i , its neighbor set is defined as the set of all other nodes within the communication range of the sensor node n i , wherein the other nodes are sensor nodes or sink nodes n 0 ;

[0012] All sensor nodes n i i=1,…,A perform a round of data transmission every T R seconds, that is, each sensor node n i i=1,…,A collects T R second data, and then sends the collected data to the sink node n 0 through multi-hop routing; the maximum cache space of each sensor node n i i=1,…,A is L, and the transmission bandwidth is W; when the sensor node is transmitting data, the newly arrived data packets will be stored in its cache queue, waiting for the sensor node to be idle for relaying, and the data packets exceeding the cache space will be discarded directly;

[0013] There are B selectable routing algorithms m1,…,m B , and the total routing algorithm set is defined as The sink node n 0An agent is deployed on top of the system to select a routing algorithm, and a routing algorithm decision is made every Ω rounds of transmission;

[0014] The agent is trained using an actor-critic-based deep reinforcement learning algorithm:

[0015] 1.1) Randomly generate an integer j within the range [1, B], and select the routing algorithm m. j As a routing algorithm, and based on each sensor node n i The neighbor set of A, i = 1, ... Calculate n for each sensor node i i = 1, ..., A reaches the sink node n 0 Routing;

[0016] 1.2) Based on sensor node n i Let i = 1, ..., A be the deployment location, and write neighbor information and routing information to each sensor node n. i In the configuration file of A, i = 1, ..., i = 0;

[0017] 1.3) Connect each sensor node n i The location information of A, i = 1, ..., is written to the sink node n. 0 For sensor node n i Its location information is: with the aggregation node n 0 In a two-dimensional rectangular coordinate system established with the origin, sensor node n i x and y coordinates

[0018] 1.4) Each sensor node n i Let i = 1, ..., A continuously monitor the environment and collect data at intervals T. R Seconds are sent to the aggregation node n according to the routing information. 0 Send the collected data;

[0019] 1.5) After each Ω round of transmission in the wireless sensor network, the agent will decide on a new routing algorithm. For the k-th decision, the specific decision-making process is as follows:

[0020] 1.5.1) Each sensor node n i i = 1, ..., A flows towards the sink node n 0 Send their respective remaining energy and cache queue length

[0021] 1.5.2) Convergence node n 0 For each sensor node n iremaining energy of each sensor node n buffer queue length and position coordinate The maximum value of each dimension is normalized:

[0022]

[0023]

[0024]

[0025] where E is the maximum energy of the sensor node, L is the maximum value of the buffer queue length, i.e. the maximum buffer space, H and V are the maximum values of the horizontal and vertical coordinates;

[0026] The normalized remaining energy, buffer queue length and position coordinate are concatenated into a one-dimensional vector The decision network of the agent is inputted;

[0027] 1.5.3), the decision network of the agent calculates the output probability vector where represents the probability of selecting the routing algorithm m b in the kth decision, b = 1, …, B;

[0028] 1.5.4), a floating point number in the range (0, 1] is randomly generated, the floating point number is in the s-th interval of the probability cumulative distribution function of the probability vector , and the routing algorithm m s is selected as the new routing algorithm;

[0029] 1.5.5), the sink node n 0 announces the new routing algorithm decided by the agent to each sensor node n i , i = 1, …, A, and then uses the new routing algorithm in the entire wireless sensor network to calculate the route to the sink node n i for each sensor node n 0 , i = 1, …, A;

[0030] 1.6), steps 1.4) to 1.5) are continuously executed until at least one sensor node is exhausted, at which time the wireless sensor network is paralyzed, and then the parameter of the decision network of the agent is updated: the decision network of the agent is taken as the actor network, the one-dimensional vector at the routing algorithm decision moment is taken as the input of the actor network and the critic network, and the deep reinforcement learning algorithm based on actor-critic is used for training, and the reward function R kAccording to the routing performance indicators of the wireless sensor network: the energy consumption of the sensor nodes, the average end-to-end delay and the packet loss rate, the calculation formula is:

[0031]

[0032]

[0033]

[0034] wherein, represents the total energy consumed by all the sensor nodes between the kth routing algorithm decision and the (k+1)th routing algorithm decision, represents the average end-to-end delay of all the data packets received by the sink node between the kth routing algorithm decision and the (k+1)th routing algorithm decision, represents the packet loss rate between the kth routing algorithm decision and the (k+1)th routing algorithm decision; ε, D min and D max are constants for normalization; is the reward value reflecting the energy efficiency indicator, is the reward value reflecting the QoS indicator, and ω is the weight coefficient of the two reward values and the reward value According to the specific application scenario and user preference, the weight coefficient of the two reward values is set, for the QoS application, ω is set to be smaller, and for the non-QoS application, ω is set to be larger;

[0035] 1.7), the energy of each sensor node n i , i=1,..., A is restored to 100%, and steps 1.1) to 1.6) are repeated to continuously update the network weights of the actor and critic networks until convergence;

[0036] (2), the wireless sensor network is deployed in the actual environment

[0037] According to the method of steps 1.1) to 1.3), then according to the deployment positions of each sensor node n i , i=1,..., A and the sink node n 0 , the sensor nodes and the sink node are deployed in the actual environment, wherein an actor is deployed on the sink node n 0 , and the decision network of the actor is the decision network of the actor trained in step (1);

[0038] (3), all the sensor nodes n i , i=1,..., A continuously detect the environment and collect data, and every T R seconds, the routing information is transmitted to the sink node n0 Send the collected data;

[0039] (4) Sensor node n i For each Ω-round transmission of A, i = 1, ..., the sink node n... 0 The agent on the network determines a new routing algorithm according to the method in step 1.5), for each sensor node n. i Let i = 1, ..., A calculates the distance to the sink node n. 0 The route.

[0040] The objective of this invention is achieved as follows:

[0041] The adaptive selection method for wireless sensor network routing algorithms based on deep reinforcement learning, as proposed in this invention, includes training the agent using a deep reinforcement learning algorithm in a simulation environment and utilizing the sink node n in a real-world environment. 0 The decision-making process of the agent involves two parts: First, during the data collection process of the wireless sensor network, the agent deployed at the sink node inputs the normalized information on the location, remaining energy, and buffer queue length of each sensor node into the decision-making network to determine the routing algorithm. Considering the limited energy of the sensor nodes, the newly determined routing algorithm is communicated to all sensor nodes in the network by the sink node, which has sufficient energy and computing resources. Second, considering that the energy consumption rate and queuing status of sensor nodes differ due to varying amounts of forwarded data, the routing algorithm decision-making is conducted online. Once the relative changes in node states reach a certain level, the agent will make another decision and select a new routing algorithm.

[0042] Furthermore, the adaptive selection method for wireless sensor network routing algorithms based on deep reinforcement learning, as proposed in this invention, also has the following beneficial effects:

[0043] 1. This invention designs an adaptive selection scheme for routing algorithms, which can intelligently select different routing algorithms and quickly respond to dynamic changes in system status;

[0044] 2. This invention designs an online decision-making scheme from network deployment to network paralysis, which can provide timely routing algorithm adjustments as the energy and queuing status of sensors change, thereby achieving dual optimization of energy efficiency and QoS;

[0045] 3. This invention uses deep reinforcement learning to solve the adaptive selection problem of routing algorithms. Compared with a single routing algorithm, it can achieve better performance in terms of network lifetime, average end-to-end latency and packet loss rate.

[0046] 4. This invention does not impose any particular restrictions on the optional routing algorithms and has strong scalability. BRIEF DESCRIPTION OF DRAWINGS

[0047] Figure 1 is the specific embodiment flow chart of the adaptive selection method of the wireless sensor network routing algorithm based on deep reinforcement learning of the present application;

[0048] Figure 2 is the schematic diagram of the deployment position of the wireless sensor network;

[0049] Figure 3 is the flow chart of training the agent based on the deep reinforcement learning algorithm of the actor-critic (Actor-Critic);

[0050] Figure 4 is the schematic diagram of the neural network architecture of the decision network in deep reinforcement learning. DETAILED DESCRIPTION

[0051] The specific embodiment of the present application is described below in conjunction with the drawings, so that those skilled in the art can better understand the present application. It needs to be specially reminded that in the following description, when the detailed description of the known functions and designs may obscure the main content of the present application, these descriptions will be omitted here.

[0052] Figure 1 is the specific embodiment flow chart of the adaptive selection method of the wireless sensor network routing algorithm based on deep reinforcement learning of the present application.

[0053] In this embodiment, as shown in Figure 1 , the adaptive selection method of the wireless sensor network routing algorithm based on deep reinforcement learning of the present application comprises the following steps:

[0054] Step S1: In the simulation environment, the agent is trained using a deep reinforcement learning algorithm

[0055] According to the actual deployment, a wireless sensor network composed of A battery-powered sensor nodes n 1 , …, n A and a sink node n 0 connected to the power supply is established in the simulation environment, wherein the sink node n 0 is located at the origin, and each sensor node n i i = 1, …, A and the sink node n 0 are deployed in the simulation environment according to the actual environment deployment position. For the sensor node n i , its neighbor set is defined as the set of all other nodes within the communication range of the sensor node n i , which are sensor nodes or sink nodes n 0 .

[0056] In this embodiment, as Figure 2 As shown, the wireless sensor network has one aggregation node n connected to a power source. 0 A = 19 battery-powered sensor nodes n 1 ,…,n 19 Then the set of all nodes is Assume that the initial energy (maximum energy) of each sensor node is E = 50 J, uniformly distributed within a range of H = 1000 m × V = 1000 m, where the sink node n 0 Located at the origin. Due to the rated power limitation of the sensors, the communication range of each sensor node is R = 600 meters. For sensor node n... i For i = 1, ..., 19, its neighbor set is: where dist(n i n j ) represents sensor node n i With node n j The distance between nodes (sensor nodes or aggregation nodes) within the communication range R = 600 meters is defined as sensor node n. i The neighboring nodes.

[0057] All sensor nodes n i i = 1, ..., A, per T R One round of data transmission occurs every second, meaning each sensor node performs n data transmissions per second. i ,i=1,…,A collects T R After a few seconds of data collection, the collected data is sent to the aggregation node n via a multi-hop routing mechanism. 0 Each sensor node n i Let i = 1, ..., A have a maximum buffer space of L and a transmission bandwidth of W. When a sensor node is sending data, newly arriving data packets will be stored in its buffer queue and relayed when the sensor node is idle. Data packets that exceed the buffer space will be discarded directly.

[0058] In this embodiment, T R =0.08, meaning each sensor node n i Collect T, i = 1, ..., 19 R After a few seconds of data collection, the collected data is sent to the aggregation node n via a multi-hop routing mechanism. 0 The maximum buffer space for each sensor node is L = 10kb, and the transmission bandwidth is W = 200kbps.

[0059] There are B possible routing algorithms m1, ..., m2. B Define the complete set of routing algorithms as Convergence node n 0An agent is deployed on top of the system to select a routing algorithm, making a routing algorithm decision every Ω rounds of transmission.

[0060] In this embodiment, there are B = 5 selectable routing algorithms m1, ..., m5, so the complete set of routing algorithms is: A routing algorithm decision is made every Ω = 10 rounds of transmission, and the decision result is announced to all sensor nodes n. i , i = 1, ..., 19.

[0061] The agent is trained using an actor-critic-based deep reinforcement learning algorithm, such as... Figure 3 As shown, the specific steps include:

[0062] Step S1.1: Randomly select a routing algorithm and calculate the route for each sensor node.

[0063] Randomly generate an integer j within the range [1, B], and select the routing algorithm m. j As a routing algorithm, and based on each sensor node n i The neighbor set of A, i = 1, ... Calculate n for each sensor node i i = 1, ..., A reaches the sink node n 0 The route is calculated using routing algorithm m2. In this embodiment, the generated random number j = 2, which indicates that the route is calculated using routing algorithm m2.

[0064] Step S1.2: Write neighbor information and routing information to each sensor node.

[0065] Based on sensor node n i Let i = 1, ..., A be the deployment location, and write neighbor information and routing information to each sensor node n. i In the configuration file of i = 1, ..., A.

[0066] Step S1.3: Write the location information of each sensor node into the aggregation node.

[0067] Each sensor node n i The location information of A, i = 1, ..., is written to the sink node n. 0 For sensor node n i Its location information is: with the aggregation node n 0 In a two-dimensional rectangular coordinate system established with the origin, sensor node n i x and y coordinates

[0068] Step S1.4: Each sensor node continuously monitors the environment, collects data, and sends it to the aggregation node.

[0069] Each sensor node n i Let i = 1, ..., A continuously monitor the environment and collect data at intervals T. R Seconds are sent to the aggregation node n according to the routing information. 0 Send the collected data.

[0070] Step S1.5: After each Ω-round transmission, the agent will decide on a new routing algorithm.

[0071] In each Ω-round transmission of the wireless sensor network, the agent will decide on a new routing algorithm. In this embodiment, Ω = 10. For the k-th decision, the specific decision-making process is as follows:

[0072] Step S1.5.1: Each sensor node n i i = 1, ..., A flows towards the sink node n 0 Send their respective remaining energy and cache queue length

[0073] Step S1.5.2: Convergence node n 0 For each sensor node n i The remaining energy of A, i = 1, ... Cache queue length and position coordinates Normalization is performed using the maximum value of each dimension:

[0074]

[0075]

[0076]

[0077] Where E is the maximum energy of the sensor node, L is the maximum length of the buffer queue, i.e., the maximum buffer space, and H and V are the maximum values ​​of the horizontal and vertical axes.

[0078] The normalized remaining energy, buffer queue length, and position coordinates are concatenated into a one-dimensional vector. The decision network input to the agent. (One-dimensional vector) This refers to the network status.

[0079] Step S1.5.3: The agent's decision network calculates the output probability vector. in, This indicates that the routing algorithm m is selected in the k-th decision. bthe probability of the kth decision output, b = 1, …, B.

[0080] Assume that the probability vector of the kth decision output is The corresponding cumulative distribution function vector is [0.1, 0.2, 0.3, 0.4, 1.0].

[0081] Step S1.5.4: Randomly generate a floating point number in the range of (0, 1], and the floating point number is located in the s-th interval of the probability cumulative distribution function of the probability vector , select the routing algorithm m s as the new routing algorithm.

[0082] Assume that the randomly generated floating point number is 0.5, which falls exactly in the 5th interval of the cumulative distribution function vector, and m5 is selected as the new routing algorithm.

[0083] Step S1.5.5: The sink node n 0 announces the new routing algorithm decided by the agent to each sensor node n i , i = 1, …, A, and then uses the new routing algorithm in the entire wireless sensor network to calculate the route to the sink node n i for each sensor node n 0 , i = 1, …, A.

[0084] Step S1.6: Parameter update of the decision network of the agent based on the actor-critic deep reinforcement learning algorithm

[0085] Steps S1.4 to S1.5 are continuously executed until at least one sensor node is exhausted, at which time the wireless sensor network is paralyzed, and then the parameter of the decision network of the agent is updated: the decision network of the agent is taken as the actor network, the one-dimensional vector at the routing algorithm decision moment is taken as the input of the actor network and the critic network, and the actor-critic deep reinforcement learning algorithm is used for training. The reward function R k is calculated according to the routing performance indicators of the wireless sensor network: sensor node energy consumption, average end-to-end delay, and packet loss rate, and the specific calculation formula is:

[0086]

[0087]

[0088]

[0089] wherein, denotes the total energy consumed by all sensor nodes between the kth routing algorithm decision and the (k+1)th routing algorithm decision, denotes the average end-to-end delay of all data packets received by the sink node between the kth routing algorithm decision and the (k+1)th routing algorithm decision, denotes the packet loss rate between the kth routing algorithm decision and the (k+1)th routing algorithm decision; ε, D min and D max is a constant for normalization; is a reward value reflecting the energy efficiency index, is a reward value reflecting the QoS index, and ω is a weight coefficient of the two reward values and the reward value The weight coefficient of the two reward values is set according to a specific application scenario and user preference. For a QoS application, ω is set to be relatively small, and for a non-QoS application, ω is set to be relatively large.

[0090] Step S1.7: restore the energy of each sensor node, and continuously update the weights of the actor and critic networks until convergence

[0091] restore the energy of each sensor node n i to 100%, and repeat steps S1.1 to S1.6, and continuously update the weights of the actor and critic networks until convergence.

[0092] Since the online decision of the routing algorithm affects the system state of the wireless sensor network, there is a certain correlation between multiple decisions. Moreover, the adaptive selection problem of the routing algorithm can be regarded as a continuous decision problem, which meets the deep reinforcement learning paradigm, so the neural network is trained by using the deep reinforcement learning technology. Specifically, the deep reinforcement learning algorithm based on the actor-critic framework is selected, and the proximal policy optimization (PPO) is used to complete the training.

[0093] In this embodiment, the actor network is the decision network finally deployed in the agent, as shown in FIG. 1. Figure 4 The input state vector is then passed through a fully connected layer with a width of 256 using a ReLU (Rectified Linear Unit) function activation, and then passed through a fully connected layer with a width of 512 using a ReLU function activation, and finally passed through a softmax layer to obtain an output probability vector

[0094] The critic network needs to evaluate the value of the states to guide the learning of the actor network, whose input is consistent with that of the actor network. In this implementation, the critic network uses three fully connected layers with ReLU activation and a width of 512, along with a linear layer, to obtain estimates of the state values.

[0095] In this embodiment, both the actor and critic networks were trained using the Adam (Adaptive Momentum) optimizer, with the actor network using a learning rate of 1×10⁻⁶. -5 The learning rate used by the critic network is 1×10. -4 To ensure the exploration intensity of the actor network and prevent it from getting trapped in local optima too early, an entropy regularization term is added to the loss function, with the entropy regularization weight set to 0.01.

[0096] Step S2: Deploy the wireless sensor network in the real environment

[0097] Process according to the method in steps S1.1-1.3, and then according to each sensor node n i i = 1, ..., A and the sink node n 0 The deployment location is determined in the actual environment, where the aggregation node n 0 An agent is deployed on the network, and its decision network is the decision network of the agent obtained in step S1.

[0098] Step S3: All sensor nodes n i Let i = 1, ..., A continuously monitor the environment and collect data to determine if any sensor nodes have run out of energy. If so, the wireless sensor network fails, and the optimization ends. Otherwise, every T interval... R Seconds are sent to the aggregation node n according to the routing information. 0 Send the collected data.

[0099] Step S4: Sensor node n i For i = 1, ..., A performs Ω rounds of transmission, and checks whether Ω rounds have been transmitted. If so, it merges with node n. 0 The agent determines a new routing algorithm according to the method in step S1.5, and uses the new routing algorithm for each sensor node n. i Let i = 1, ..., A calculates the distance to the sink node n. 0 The route.

[0100] To illustrate the technical effects of the present invention, a specific example is used to experimentally verify the invention. In this specific example, the following method is employed: Figure 2The wireless sensor network shown selects five routing algorithms as comparison methods, compares network survival time, average end-to-end delay and packet loss rate of the application (ω=0.1) and five comparison methods. Comparison method 1 is abbreviated as EAR (R.C. Shah and J.M. Rabaey, "Energy aware routing for low energy ad hoc sensor networks," in 2002 IEEE Wireless Communications and Networking Conference Record. WCNC 2002 (Cat. No. 02TH8609), vol. 1, pp. 350-55, IEEE, 2002.); comparison method 2 is abbreviated as MLDR (J.-H. Chang and L. Tassiulas, "Maximum lifetime routing in wireless sensor networks," IEEE / ACM Transactions on networking, vol. 12, no. 4, pp. 609-19, 2004.); comparison method 3 is abbreviated as EBRP (F. Ren, J. Zhang, T. He, C. Lin, and S.K.D. Ren, "Ebrp: energy balanced routing protocol for data gathering in wireless sensor networks," IEEE transactions on parallel and distributed systems, vol. 22, no. 12, pp. 2108-125, 2011.); comparison method 4 is abbreviated as SPEED (T. He, J.A. Stankovic, C. Lu, and T. Abdelzaher, "Speed: A stateless protocol for real-time communication in sensor networks," in 23rd International Conference on Distributed Computing Systems, 2003. Proceedings., pp. 46-5, IEEE, 2003.); comparison method 5 is abbreviated as EAR+ (M. Younis, M. Youssef, and K.Arisha, "Energy-aware routing in cluster-based sensor networks," in Proceedings. 10th IEEE international symposium on modeling, analysis and simulation of computer and telecommunications systems, pp. 129-36, IEEE, 2002.

[0101] Table 1 is a comparison table of network lifetime, average end-to-end delay and packet loss rate of the present application and five comparative methods.

[0102] Routing method EAR MLDR EBRP SPEED EAR+ The invention Network survival time / rounds 1603 1593 1012 276 1569 1598 Average end-to-end latency / msec 520.65 37.32 17.43 12.35 113.97 27.39 Packet loss rate 0.0365 0.0021 0.0000 0.0000 0.0044 0.0000

[0103] Table 1

[0104] As shown in Table 1, in the QoS application scenario, the present application can ensure a lower average end-to-end delay and packet loss rate, and at the same time, can make the network lifetime longer, which is 1.58 times that of the EBRP method and 5.79 times that of the SPEED method, proving that the present application can realize the dual optimization of energy efficiency and QoS indicators.

[0105] Although the above describes the specific embodiments of the present application in detail in order to facilitate the understanding of the present application by those skilled in the art, it should be clear that the present application is not limited to the scope of the specific embodiments, and for those skilled in the art, it is obvious that various changes are within the spirit and scope of the present application defined and determined by the appended claims, and all the inventions utilizing the concept of the present application are within the scope of protection.

Claims

1. An adaptive selection method for routing algorithms in wireless sensor networks based on deep reinforcement learning, characterized by comprising the following steps: (1) In a simulation environment, the agent is trained using a deep reinforcement learning algorithm. Based on the actual deployment, establish a sensor node n powered by A batteries in the simulation environment. 1 ,…,n A and a power convergence node n 0 The wireless sensor network consists of a collection node n. 0 Located at the origin, each sensor node n is deployed in the simulation environment according to its actual deployment location. i ,i=1,…,A and the sink node n 0 ; For sensor node n i Define its neighbor set For sensor node n i The set of all other nodes within the communication range, wherein the other nodes are sensor nodes or aggregation nodes n. 0 ; All sensor nodes n i i = 1, ..., A, each T R One round of data transmission occurs every second, meaning each sensor node performs n data transmissions per second. i ,i=1,…,A collects T R After a few seconds of data collection, the collected data is sent to the aggregation node n via a multi-hop routing mechanism. 0 Each sensor node n i The maximum buffer space of i = 1, ..., A is L, and the transmission bandwidth is W. When a sensor node is sending data, newly arrived data packets will be stored in its buffer queue and relayed when the sensor node is idle. Data packets that exceed the buffer space will be discarded directly. There are B possible routing algorithms m1,…,m B Define the complete set of routing algorithms as Convergence node n 0 An agent is deployed on top of the system to select a routing algorithm, and a routing algorithm decision is made every Ω rounds of transmission; The agent is trained using an actor-critic-based deep reinforcement learning algorithm: 1.1) Randomly generate an integer j within the range [1, B], and select the routing algorithm m. j As a routing algorithm, and based on each sensor node n i The neighbor set of A, i = 1, ... Calculate n for each sensor node i i = 1, ..., A reaches the convergence node n 0 Routing; 1.2) Based on sensor node n i The deployment location, i = 1, ..., A, writes neighbor information and routing information to each sensor node n. i In the configuration file for i = 1, ..., A; 1.3) Connect each sensor node n i The location information of A, i = 1, ..., is written to the sink node n. 0 For sensor node n i Its location information is: with the aggregation node n 0 In a two-dimensional rectangular coordinate system established with the origin, sensor node n i x and y coordinates 1.4) Each sensor node n i i = 1, ..., A continuously monitors the environment and collects data at intervals T. R Seconds are sent to the aggregation node n according to the routing information. 0 Send the collected data; 1.5) After each Ω round of transmission in the wireless sensor network, the agent will decide on a new routing algorithm. For the k-th decision, the specific decision-making process is as follows: 1.5.1) Each sensor node n i i = 1, ..., A flows towards the sink node n 0 Send their respective remaining energy and cache queue length 1.5.2) Convergence node n 0 For each sensor node n i The remaining energy of A, i = 1, ... Cache queue length and position coordinates Normalization is performed using the maximum value of each dimension: in, E is the maximum energy of the sensor node, L is the maximum length of the buffer queue, i.e., the maximum buffer space, and H and V are the maximum values ​​of the horizontal and vertical axes. The normalized remaining energy, buffer queue length, and position coordinates are concatenated into a one-dimensional vector. The decision-making network of the input agent; 1.5.3) The agent's decision network calculates the output probability vector. in, This indicates that the routing algorithm m is selected in the k-th decision. b The probability, b = 1, ..., B; 1.5.4) Randomly generate a floating-point number within the range (0,1], with the floating-point number located in the probability vector. The s-th interval of the cumulative probability distribution function, the routing algorithm m s As a new routing algorithm; 1.5.5), Convergence Node n 0 The newly determined routing algorithm by the agent is communicated to each sensor node n. i ,i=1,...,A, and then a new routing algorithm is used throughout the wireless sensor network to route each sensor node n i i = 1, ..., A calculates the distance to the sink node n 0 Routing; 1.6) Continuously execute steps 1.4) to 1.5) until at least one sensor node runs out of energy, at which point the wireless sensor network fails. Then, update the parameters of the agent's decision network: treat the agent's decision network as the actor network, and the one-dimensional vector at the decision time of the routing algorithm. As input to the actor network and critic network, a deep reinforcement learning algorithm based on actor-critic is used for training. The reward function R used during training is... k The routing performance metrics of wireless sensor networks are calculated based on sensor node energy consumption, average end-to-end latency, and packet loss rate. The specific calculation formula is as follows: in, This represents the total energy consumed by all sensor nodes between the k-th routing algorithm decision and the (k+1)-th routing algorithm decision. This represents the average end-to-end delay of all data packets received by the sink node between the k-th routing algorithm decision and the (k+1)-th routing algorithm decision. ε and D represent the packet loss rate between the k-th routing algorithm decision and the (k+1)-th routing algorithm decision; min and D max It is a constant used for normalization; It is a reward value that reflects energy efficiency. It is the reward value reflecting the QoS indicator, where ω is the reward value. and reward value The weighting coefficients of these two reward values ​​are set according to specific application scenarios and user preferences. For QoS applications, ω is set to be smaller, and for non-QoS applications, ω is set to be larger. 1.7) Connect each sensor node n i The energy of i = 1, ..., A is restored to 100%. Steps 1.1) to 1.6) are repeated to continuously update the network weights of the actor and critic networks until convergence. (2) Deploying wireless sensor networks in real-world environments Process according to steps 1.1)-1.3), and then based on each sensor node n i ,i=1,...,A and the sink node n 0 The deployment location is determined in the actual environment, where the aggregation node n 0 An agent is deployed on the network, and its decision network is the decision network of the agent obtained in step (1) during training. (3) All sensor nodes n i i = 1, ..., A continuously monitors and collects data on the environment, and every T interval R Seconds are sent to the aggregation node n according to the routing information. 0 Send the collected data; (4) Sensor node n i For each Ω-round transmission of i=1,...,A, the sink node n 0 The agent on the network determines a new routing algorithm according to the method in step 1.5), for each sensor node n. i i = 1, ..., A calculates the distance to the sink node n 0 The route.

Images