A DQN-based industrial wireless sensor network coexistence scheduling method

Abstract

The present application relates to industrial wireless sensor network, especially to a kind of industrial wireless sensor network coexistence scheduling method based on DQN, this method is for the scene of industrial wireless sensor network and multiple wireless network coexistence, industrial wireless sensor network coexistence problem is modeled as Markov decision process, while setting the optimization goal of scheduling is to minimize network data information freshness weighted sum.Utilize the characteristics of deep reinforcement learning autonomous learning and the memory of recurrent neural network, a kind of DQN dynamic real-time scheduling method based on recurrent neural network is invented, the method makes optimal decision at the beginning of each time slot using recurrent neural network to achieve the purpose of coexistence.The scheduling method proposed in the present application can be realized in the case of limited time domain resources industrial wireless sensor network and multiple wireless network coexistence, improve network utilization, while taking information freshness as optimization goal, improve the real-time performance of data transmission.

Description

TECHNICAL FIELD

[0001] The application relates to an industrial wireless sensor network, in particular to an industrial wireless sensor network coexistence scheduling method based on DQN. BACKGROUND

[0002] With the rapid development of the industrial internet, special or private networks are diversified, and the coexistence of multiple networks has become a major problem. In the face of problems such as unknown sampling period of sensor nodes in industrial wireless sensor networks and unknown deployment environment of the network, it is difficult to achieve scheduling by using traditional scheduling methods. Traditional industrial wireless sensor network scheduling methods are mainly designed for static networks, and the resource allocation problem of optimizing throughput, energy consumption and delay is modeled as a mathematical problem, and the convex optimization theory is used for solving. However, in practice, wireless networks are dynamically changing, and it is difficult to establish an ideal mathematical model. Deep reinforcement learning algorithm is used to interact with the current unknown network environment, which can well handle problems with large dimensions and unknown models, so it is necessary to use deep reinforcement learning algorithm to schedule sensor nodes in industrial wireless sensor networks to reduce conflicts.

[0003] At the same time, timely data transmission is extremely important in industrial wireless sensor networks, and outdated data may be useless or even cause irreversible errors. Traditional performance indicators, such as delay, usually evaluate the time consumption of nodes in wireless link transmission. SUMMARY

[0004] The application aims at the problems in the above background art, and proposes an industrial wireless sensor network coexistence scheduling method based on DQN, with the purpose of minimizing the weighted sum of network information freshness, and the specific steps include:

[0005] S1: According to the number of sensor nodes, channel state and information freshness of the sensor nodes in the industrial wireless sensor network, the industrial wireless sensor network coexistence scheduling is modeled as a Markov decision process;

[0006] S2: A DQN algorithm based on recurrent neural network is used to construct a deep reinforcement learning industrial wireless sensor network scheduler with action selection function and time series prediction function;

[0007] S3: The scheduler selects an action in a random manner or predicts an action according to the preset action selection rule; that is, according to the dynamic probability epsilon, an action is randomly selected from the action space in the to-be-processed time slot, or according to the probability of 1-epsilon, the channel state of the previous M continuous time slots of the to-be-processed time slot is predicted to obtain the action value, and an action is obtained through the proportional fairness utility function according to the action value, with the purpose of minimizing the weighted sum of network information freshness;

[0008] S4: send the scheduling strategy derived from the to-be-processed time slot to the sensor nodes of the industrial wireless sensor network, and execute the scheduling strategy.

[0009] Advantages of the present application:

[0010] The present application considers the information freshness in the network, and designs a coexistence scheduling algorithm based on reinforcement learning for the purpose of optimizing the information freshness weighted average of the information network, overcomes the problem of unknown network environment, introduces a penalty function in the reward function, balances the action selection of the nodes, introduces a fair benefit function and information freshness weight in the action selection rule, ensures the throughput of the network, corrects the error in the action selection, and simultaneously adopts a GRU network in the neural network of DQN, maximally accelerates the convergence speed. BRIEF DESCRIPTION OF DRAWINGS

[0011] Figure 1 A flow chart of the DQN-based industrial wireless sensor network coexistence scheduling method of the present application;

[0012] Figure 2 An application scenario diagram of the DQN-based industrial wireless sensor network coexistence scheduling method of the present application;

[0013] Figure 3 A model structure diagram based on DQN of the DQN-based industrial wireless sensor network coexistence scheduling method of the present application. DETAILED DESCRIPTION

[0014] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the protection scope of the present application.

[0015] The DQN-based industrial wireless sensor network coexistence scheduling method proposed in the present application, as shown in Figure 1 , comprises the following steps:

[0016] S1: model the industrial wireless sensor network coexistence scheduling as a Markov decision process according to the number of sensor nodes of the industrial wireless sensor network, the channel state and the information freshness of the sensor nodes;

[0017] In the embodiments of the present application, the state space, action space and reward function of the network scheduling process are determined according to the number of nodes of the industrial wireless sensor network, the channel state and the information freshness of the sensor nodes, the industrial wireless sensor network coexistence problem is modeled as a Markov decision process, and the optimization goal of the scheduling can be set as minimizing the weighted sum of data information freshness in the network.

[0018] In the present example, the action space of the Markov decision process is determined according to the number of sensor nodes of the industrial wireless sensor network, the state space of the Markov decision process is determined according to the channel state and the information freshness of the sensor nodes of the industrial wireless sensor network, and the reward function of the Markov decision process is determined according to the number of sensor nodes, the channel state and the information freshness of the sensor nodes of the industrial wireless sensor network; wherein the information freshness measures the time elapsed from the generation time to the next update from the perspective of the application layer, which is more suitable for the evaluation of the real-time data in the industrial wireless sensor network, and the initial value can be set according to experience.

[0019] The action space, the state space and the reward function of the partially observable Markov process are respectively:

[0020] (1) The action space is A = {0, 1, 2, …, N}

[0021] Wherein, the action a t ∈A at time slot t represents the node selecting to transmit the data packet in the time slot, N is the number of nodes, and 0 represents that all nodes do not transmit in the current time slot.

[0022] (2) The state space S = { {O t-m ,…,O t}, {O t-m+1 ,…,O t+1}, …}

[0023] Wherein, the state S t ∈S at time slot t represents the network state at time slot t, which is composed of a plurality of time-continuous m observation values, O t represents the observation value at time slot t, O t = {a t , S c (t), Δ(t)}, S c (t) represents the channel state in the network after the action at time slot t, S c (t) ∈ {0, 1}, 0 represents channel conflict or idle, and 1 represents successful transmission of the data packet, and Δ(t) is a set of information freshness of all nodes at time slot t.

[0024] (3) The reward function R includes the following:

[0025]

[0026] Wherein, r t+1 represents the reward function at time slot t+1, r t+1 ∈R, I(x) is an indicator function, N is the maximum freshness threshold, and Δ i(t) is the information freshness of node i at time t, ω i is the weight of node i, X i (k) e {0, 1} is whether node i is scheduled at time k, is the penalty function, V is a non-negative control parameter, U i (t) is whether node i generates a new data packet at time slot t.

[0027] In this example, the industrial wireless sensor network consists of a DQN controller and multiple sensor nodes, as shown in Figure 2 The data of the sensor nodes is periodically updated with periodic sampling. In this network, time is divided into equal-length time slots, and the access point determines the sensor node actions at the beginning of each time slot. The sensor nodes that receive the action information will complete data transmission within a time slot. The wireless channel used for data transmission is an unknown shared channel, and part of the time domain resources has been occupied, so there is a possibility of transmission failure due to channel conflict in the scheduling process.

[0028] In this example of industrial wireless sensor network deployment scenario, the information freshness is updated when the new data packet successfully arrives, and the sensor node discards the old data packet after generating a new data packet, ensuring that the data packet sent to the access point is the current latest.

[0029] The goal of scheduling is to minimize the weighted average sum of information freshness:

[0030]

[0031] where ω i is the weight of each node, Δ i (t) is the information freshness of node i at time t. The update formula of information freshness is:

[0032]

[0033] where Δ i (t+1) is the information freshness of node i at time t+1, G i (t) is the generation time of the data packet of node i at time t, G i (t-1) is the generation time of the data packet of node i at time t-1, U i (t) e {0, 1} is whether node i generates a new data packet at time slot t, if so, U i (t) = 1, otherwise U i (t) = 0, when a new data packet is generated, G i (t) is updated to 0, otherwise it increases linearly; Δ i (t) is the information freshness of node i, Xi (t)∈{0,1} indicates whether node i is scheduled at time slot t, if yes, X i (t) = 1, otherwise X i (t) = 0, C(t)∈{0,1} indicates whether node i successfully transmits at time slot t, if yes, C(t) = 1, otherwise C(t) = 0; when a data packet is scheduled and successfully transmitted, Δ i (t) is updated to G i (t) + 1, otherwise the information freshness linearly increases.

[0034] S2: using a DQN algorithm based on a recurrent neural network to construct a deep reinforcement learning industrial wireless sensor network scheduler with action selection function and time series prediction function;

[0035] The step S2 includes constructing two prediction networks and target networks with the same structure based on the DQN algorithm based on the recurrent neural network, and the prediction network and the target network each include an input layer, a GRU layer, a full connection layer and an output layer, the input of the prediction network and the target network is a two-dimensional network state with time series characteristics, and the output is a one-dimensional action value.

[0036] Before the scheduler performs an action, the network scheduler also needs to be initialized, and the initialization process includes:

[0037] (1) setting the minimum number of training samples of the network as M;

[0038] (2) setting a training interval I;

[0039] (3) setting the minimum number of iterations for updating the target network as K, and randomly selecting an initial state S0;

[0040] (4) initializing the prediction network and the target network in the network scheduler by using the same parameters;

[0041] (5) setting the capacity of the experience replay pool as L, the experience replay pool stores the data set (S t ,a t ,r t+1 ,S t+1 ) generated by the interaction between the network scheduler and the environment, and when the stored data reaches the maximum number of entries, the new data is used to overwrite the old data.

[0042] S3: the scheduler selects an action in a random manner or predicts an action according to a preset action selection rule; that is, according to a dynamic probability ε, an action is randomly selected from the action space in the to-be-processed time slot, or according to a probability of 1-ε, an action value is predicted based on the channel state of the previous M consecutive time slots of the to-be-processed time slot, and an action is obtained through a proportional fairness utility function according to the action value.

[0043] In the embodiments of the present application, the dynamic probability epsilon gradually decreases with the increase of the number of iterations, that is, the dynamic probability epsilon is determined by the number of iterations of training the deep reinforcement learning industrial wireless sensor network scheduler with time series prediction function.

[0044] In some preferred embodiments, the dynamic probability epsilon can gradually decrease in the early stage of training, and can remain unchanged in the later stage of training.

[0045] The network scheduler action selection rule is: with the probability of epsilon, a random action is selected from the action space, and with the probability of 1-epsilon, the current action is determined by the proportional fairness utility function, which is defined as:

[0046]

[0047] Wherein, F is the proportional fairness utility function, that is, the action value of the non-zero action, q(S t ,a t ) is the action value of the action a t taken by the network scheduler in the state S t at time slot t, q(S t ,0) is the action value of action 0 taken in the state S t at time slot t, and N is the number of nodes.

[0048] When F >= ln(q(S t ,0)), the action at time slot t is a Otherwise, the action at time slot t is a t =0, wherein is the information freshness of node The information freshness is the proportion of all node information freshness, and the error in the correction of action selection, q(S t ,a ) is the action value of the action a taken by the network scheduler in the state S at time slot t, is the information freshness of node at the current time slot, is the weighted sum of all node information freshness, and

[0049] N is the number of nodes. Figure 3 When the number of samples in the experience storage pool is greater than M, the network scheduler randomly selects M samples from it to train the prediction network, and the specific model is as shown in Figure 3 , taking the current state as the input of the prediction network, and outputting the value of each action. The loss function of the prediction network is:

[0050]

[0051] wherein g is a discount factor, q target (S t+1 , a'; w') is the optimal value of the target network in the next state, q predict (S t , a t ; w) is the value of the prediction network in the current state and action, r t+1 is the reward obtained by the current action. When the number of training reaches K times, the parameters of the target network are updated using the parameters of the prediction network, i.e. w' = w, and the iteration is stopped after the optimal scheduling strategy is obtained through multiple iterations.

[0052] S4: sending the scheduling strategy obtained for the to-be-processed time slot to the sensor nodes of the industrial wireless sensor network, and executing the scheduling strategy.

[0053] It can be understood that the to-be-processed time slot can be a current time slot, a historical time slot or a future time slot, and the present application does not make a specific limitation thereon.

[0054] It can be understood by those skilled in the art that all or part of the steps in the various methods of the above embodiments can be completed by programs instructing relevant hardware, and the programs can be stored in a computer readable storage medium, which can include ROM, RAM, magnetic or optical disks, etc.

[0055] Although the embodiments of the present application have been shown and described, it can be understood by those skilled in the art that various changes, modifications, replacements and variations can be made to the embodiments without departing from the principles and spirits of the present application, and the scope of the present application is defined by the appended claims and their equivalents.

Claims

1. A DQN-based industrial wireless sensor network coexistence scheduling method, characterized in that, The method comprises the following steps: S1: modeling the coexistence scheduling of the industrial wireless sensor network as a Markov decision process according to the number of sensor nodes of the industrial wireless sensor network, the channel state, and the information freshness of the sensor nodes; The update formula of the information freshness is: where Δ i (t+1) represents the information freshness of node i at time slot t+1, G i (t) represents the generation time of the data packet of node i at time slot t, G i (t-1) represents the generation time of the data packet of node i at time slot t-1, U i (t)∈{0,1} represents whether a new data packet is generated by node i at time slot t, if yes, U i (t) = 1, otherwise U i (t) = 0, when a new data packet is generated, G i (t) is updated to 0, otherwise it increases linearly; Δ i (t) is the information freshness of node i, X i (t)∈{0,1} represents whether node i is scheduled at time slot t, if yes, X i (t) = 1, otherwise X i (t) = 0, C(t)∈{0,1} represents whether node i successfully transmits at time slot t, if yes, C(t) = 1, otherwise C(t) = 0; when the data packet is scheduled and successfully transmitted, Δ i (t) is updated to G i (t) + 1, otherwise the information freshness increases linearly; The step S1 comprises determining the action space of the Markov decision process according to the number of sensor nodes of the industrial wireless sensor network, determining the state space of the Markov decision process according to the channel state and the information freshness of the sensor nodes of the industrial wireless sensor network, and determining the reward function of the Markov decision process according to the number of sensor nodes, the channel state, and the information freshness of the sensor nodes of the industrial wireless sensor network; The action space A is: A={0, 1, 2, …, N} wherein the action of time slot t is a t ∈A, represents the node selecting this time slot to transmit data packet at t time slot, N is the number of nodes, 0 represents that all nodes do not transmit at the current time slot; The state space S is: S = { { O t-m ,…, O t}, { O t-m+1 ,…, O t+1}, …} where S t ∈ S, denotes the network state at time slot t, which consists of a number of m consecutive observations in time, O t denotes the observation at time slot t, O t = {a t , S c (t), Δ(t)}, S c (t) denotes the channel state in the network after action at time slot t, S c (t) ∈ {0, 1}, 0 represents channel conflict or idle, 1 represents successful transmission of data packets, and Δ(t) is a set of all node information freshness at time slot t; The reward function R comprises the following: where r t+1 is the reward function of time slot t + 1, r t+1 ∈ R, I(x) is an indicator function, N max is the maximum freshness threshold, Δ i (t) represents the information freshness of node i at time t, ω i is the weight of node i, X i (k) ∈ {0, 1} is whether node i is scheduled at time k, is a penalty function, V is a non-negative control parameter, U i (t) represents whether node i generates a new data packet at time slot t; S2: constructing a deep reinforcement learning industrial wireless sensor network scheduler with action selection function and time series prediction function by using a DQN algorithm based on a recurrent neural network; S3: the scheduler selects an action in a random manner or predicts an action according to a preset action selection rule; that is, an action is randomly selected from the action space according to a dynamic probability ε in a to-be-processed time slot, or an action value is predicted based on the channel state of the previous M continuous time slots of the to-be-processed time slot, with the objective of minimizing the weighted sum of the information freshness in the network, and an action is obtained through a proportional fairness utility function according to the action value; The proportional fairness utility function in step S3 is: Where F is the proportional equity utility function, which represents the sum of the action values ​​of non-zero actions, q(S t ,a t The network scheduler predicts the network's state S in time slot t. t Take action a t The value of the action, q(S) t ,0) represents the state S in time slot t. t The value of taking action 0 is given below, where N is the number of nodes. The action obtained through the proportional fairness utility function in step S3 comprises: When F ≥ ln(q(S t ,0)), the action of time slot t is Otherwise, the action of time slot t is a t =0, where is the information freshness of node The information freshness is the proportion of all nodes information freshness and the error when correcting the action selection, The action value of taking action in state S t at time slot t predicted by the network scheduler, is the information freshness of node at the current time slot, is the weighted sum of all nodes information freshness, and N is the number of nodes. S4: sending the scheduling strategy obtained for the to-be-processed time slot to the sensor nodes of the industrial wireless sensor network, and executing the scheduling strategy.

2. The DQN-based industrial wireless sensor network coexistence scheduling method of claim 1, wherein, The step S2 comprises constructing two prediction networks and target networks with the same structure based on the DQN algorithm based on the recurrent neural network, wherein the prediction network and the target network each comprise an input layer, a GRU layer, a full connection layer, and an output layer, the input of the prediction network and the target network is a two-dimensional network state with time series characteristics, and the output is a one-dimensional action value.

3. The DQN-based industrial wireless sensor network coexistence scheduling method of claim 1, wherein, The dynamic probability ε in step S3 is determined by the number of iterations of training the deep reinforcement learning industrial wireless sensor network scheduler with the time series prediction function.

4. The DQN-based industrial wireless sensor network coexistence scheduling method of claim 1, wherein, The minimization of the weighted average sum of the information freshness in the network in step S3 is: where ω i is the weight of node i, Δ i (t) is the information freshness of node i at time slot t, T represents the total number of time slots, π represents the scheduling strategy; Π represents the set of scheduling strategies, and N represents the set of nodes.

Images