A joint optimization scheduling method for industrial wireless network based on information age

Abstract

The application relates to a joint optimization scheduling method of an industrial wireless network based on information age, and belongs to the technical field of industrial wireless networks. The method is used for the industrial wireless network in which hard real-time data and soft real-time data coexist, simultaneously considers the overdue rate related to the data deadline, jointly optimizes the problem of minimizing the system information age and energy consumption through the scheduling time slot mode, constructs the optimization problem into a Markov decision process, uses an improved deep Q network to approximate the state value function, utilizes a curiosity network to accelerate the convergence speed, and then adopts deep reinforcement learning to obtain a link scheduling method. Through the joint optimization method, the learning speed of the scheduling method is accelerated, the information freshness in the network is ensured, and the data overdue rate and the node energy consumption are reduced.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of industrial wireless networks, and relates to a joint optimization scheduling method for an industrial wireless network based on information age. BACKGROUND

[0002] With the rapid development of industrial wireless sensor networks (IWSN), it is widely applied to real-time business scenarios such as industrial equipment monitoring, automation control, etc. In these applications, the timeliness of data transmission plays a key role in the safety and efficiency of industrial production. In order to meet the increasingly stringent real-time requirements in industrial environments, the data scheduling in IWSN needs to effectively manage the freshness and transmission delay of data.

[0003] In order to better measure the timeliness of data transmission in the network, the information age (Age of Information, AoI) is proposed as a new indicator to quantify the time interval from data generation to successful transmission to the destination node. AoI not only focuses on transmission delay, but also considers the time of data generation, so it can comprehensively evaluate the freshness of data and become one of the important indicators to optimize the timeliness of data transmission. The data in the industrial wireless sensor network is divided into hard real-time data and soft real-time data, and the hard real-time data requires higher timeliness and needs to deliver data within the deadline. In the industrial wireless sensor network, nodes achieve low power consumption by using sleep slots. However, the existence of sleep slots may cause transmission delay of data, and even overage, which cannot be transmitted to the control center in time. Therefore, for the industrial wireless sensor network, it is an important research problem to optimize the information age while reducing energy consumption under the constraint of deadline-related overage rate of data. SUMMARY

[0004] Therefore, the purpose of the present application is to provide a joint optimization scheduling method for an industrial wireless network based on information age, which considers the deadline-related overage rate of data, formulates the multi-objective problem of minimizing the weighted sum of information age and energy consumption as a constrained Markov decision process, and obtains a link scheduling method to effectively optimize the energy consumption of the network and reduce the average information age of the system under the constraint of overage rate.

[0005] To achieve the above purpose, the present application provides the following technical scheme:

[0006] The application discloses a joint optimization scheduling method of an industrial wireless network based on information age.

[0007] The method specifically comprises the following steps:

[0008] S1: obtaining system information of the industrial wireless network, establishing an information age updating model and an energy consumption updating model based on a network structure, and constructing a scheduling network for learning an optimal scheduling strategy;

[0009] S2: constructing a multi-objective optimization problem of minimizing system average information age and energy consumption under a data deadline expiration rate constraint, and establishing a state space, an action space and a cost function of the system according to information age, energy consumption and expiration rate related parameters of a system destination node, and converting the link scheduling optimization problem into a Markov decision process with a constraint;

[0010] S3: using an improved deep Q network to approximate a state value function, and using a curiosity network to accelerate the convergence speed, specifically updating the information age of the target node at each time slot, obtaining a next state by using a current state and a decision input and storing the next state into an experience pool, waiting for a loss function of the network to be stable, and thus obtaining the link scheduling method.

[0011] Further, in step S1, the establishment of the information age updating model and the energy consumption updating model specifically comprises the following steps:

[0012] S11: an industrial wireless network composed of N energy-constrained sensor nodes and one common destination node, data sampled by the sensor nodes is transmitted to the destination node through a wireless channel; n represents an index of the sensor nodes and n e {1, 2,..., N}; the system schedules data packets through time slots, t represents a time slot index and t e {1, 2,..., T}, wherein T is a maximum time slot number; the sampling period of the sensor node n is H n time slots, the probabilities of sampling hard real-time data and soft real-time data are g n and 1-g n respectively; the deadlines of the hard real-time data and the soft real-time data of the sensor node n are X n time slots and Y n time slots respectively; the data lengths of the hard real-time data and the soft real-time data of the sensor node n are x n and y nThe sensor node's buffer can only store one data packet, and the buffer is flushed before each sampling. Only one sensor node can be scheduled to send data to the destination node in time slot t, and all data transmission can be completed within one time slot. The sensor node supports three operating modes: transmit, idle, and sleep. Sleep mode only occurs after data transmission is complete. (Use u) n (t) represents the buffer data of sensor node n at time slot t, u n (t) = 1 indicates that the buffer contains hard real-time data, and vice versa for soft real-time data;

[0013] S12: Assume the channel state of the system is time-varying, and model the channel as a Fritschman channel model; quantize the channel into K states, with a channel state space β = {0, 1, 2, ..., K}; channel states 0 to G are error-free states, and G+1 to K are error-prone states; let p g p represents the probability of successful data transmission in error-free conditions. b Let h(t) be the probability of successful data transmission under error conditions; let h(t) be the channel state at time slot t, and the transition probability between channel states be expressed as p. i,j =Pr(h(t+1)=j|h(t)=i); then the state transition matrix can be written in the following form:

[0014]

[0015] S13: Calculate the information age based on the periodic sampling characteristics of the sensor nodes in step S11, and define z. n (t) represents the information age at sensor node n, and the update process is as follows:

[0016]

[0017] Among them, f n (t)∈{0,1} represents the sampling action of sensor node n at time slot t. It is 1 when data sampling is performed, and 0 otherwise.

[0018] Define a n (t) represents the information age at the destination node, a n The update process for (t) is as follows:

[0019]

[0020] Where, d n (t)∈{0,1} indicates whether sensor node n successfully sent a data packet at time slot t; 1 indicates successful data transmission and 0 indicates unsuccessful data transmission.

[0021] S14: Define the power of sensor node n in transmit mode and idle mode as P respectively.e and P i T n (t) represents the time spent by sensor node n transmitting data in time slot t, and the energy consumption of sensor node n in time slot t is e. n (t), then T n (t) and e n The update process for (t) is as follows:

[0022]

[0023] Among them, T slot U is the time slot length, R is the transmission rate, and u is the transmission rate. n (t) represents the buffer data of sensor node n at time slot t, u n (t) = 1 indicates hard real-time data, otherwise it is soft real-time data; b n (t)∈{0,1,2} represents the operating mode of sensor node n in time slot t, b n (t) = 0 indicates sleep mode, b n (t) = 1 indicates idle mode, b n (t) = 2 indicates the sending mode.

[0024] Furthermore, in step S2, a multi-objective optimization problem is constructed to minimize the average information age and energy consumption of the system under the constraint of the data deadline expiration rate. This specifically includes the following steps:

[0025] S21: Establish a joint optimization objective for information age and energy consumption, then the long-term average information age and energy consumption weighted sum at the objective node Represented as:

[0026]

[0027] Where β represents the weight parameter;

[0028] S22: Define Π as the set of all feasible strategies, where strategy π∈Π; the optimization problem is formally expressed as:

[0029]

[0030] The constraints are:

[0031]

[0032] Where, π * Γ represents the optimal scheduling strategy, and c represents the threshold for the maximum average delinquency rate. n (t) represents the number of data points that sensor node n failed to successfully transmit within the deadline from the initial time slot to time slot t, s n(t) represents the number of data samples collected by sensor node n from the initial time slot to time slot t, U n (t)∈{0,1} represents the sensor scheduled in time slot t. If sensor node n is scheduled in time slot t, then U n (t) = 1, otherwise U n (t) = 0;

[0033] S23: Decoupling the optimization problem using the Lagrange relaxation method, transforming the delinquency rate constraint into the objective function by introducing the Lagrange multiplier λ:

[0034]

[0035] in, This indicates the optimization objective.

[0036] Furthermore, in step S2, the established system state space s(t) is:

[0037] s(t)=(a(t),z(t),b(t),u(t),h(t),e(t))

[0038] Where a(t)=(a1(t),a2(t),a3(t),…,a n (t)), z(t)=(z1(t),z2(t),z3(t),…,z n (t)), b(t)=(b1(t),b2(t),b3(t),…,b n (t)), u(t)=(u1(t),u2(t),u3(t),…,u n (t)), h(t)=(h1(t),h2(t),h3(t),…,h n (t)), e(t)=(e1(t),e2(t),e3(t),…,e n (t)).

[0039] The action space A(t) of the established system is:

[0040] A(t)=(U(t))

[0041] Among them, U(t)=(U1(t),U2(t),U3(t),…,U n (t));

[0042] The established system cost function r e (t) is:

[0043]

[0044] Among them, o n(t) indicates whether the data of sensor node n expires at time slot t.

[0045] Furthermore, step S3 specifically includes the following steps:

[0046] S31: Based on the current Q network, input the current system state s(t) into the network to obtain the decision A(t), and execute the decision;

[0047] S32: After execution, input the current state into the Curiosity Network to obtain the next state prediction value s`. c (t), and obtain the intrinsic reward r by comparing it with the actual next state s`(t) in the environment. i (t);

[0048] r i (t)=δ‖s`(t)-s` c (t)‖ 2

[0049] Where δ is the scaling factor; the loss function of the curiosity network. for:

[0050]

[0051] Total reward r total (t) represents the external reward r e (t) and intrinsic reward r i The sum of (t), then the interaction experience (s(t),A(t),r) total (t),s(t+1)) are put into the experience pool, and a batch of experience is selected from the experience pool to update the network;

[0052] S33: The target value y(t) can be obtained through the set of experiences;

[0053]

[0054] Where γ represents the discount factor, and Q' represents the Q-value of the target Q-network. This represents the Q-value of the target Q-network, Q((s(t+1),A|θ). t )|θ - t The action with the smallest value, θ t and θ - t These represent the parameters of the current Q-network and the target Q-network, respectively; Q(s,A) is the Q-value output by the current Q-network, V(s) is the value function, O(s,A) is the advantage function for each action, and N+1 represents the choice of not sending data in the scheduling decision.

[0055] S34: Obtain the loss function L(θ) from the target value y(t). t ):

[0056] L(θ t )=(y(t)-Q(s(t),A(t)|θ t )) 2

[0057] S35: Update network parameters according to gradient descent method, and wait for the network to reach the termination condition to obtain the scheduling method for industrial wireless networks.

[0058] The beneficial effects of this invention are as follows:

[0059] (1) This invention takes into account factors such as the overdue rate and link scheduling related to the data deadline in industrial wireless networks, and jointly optimizes the long-term average information age and energy consumption of the system, ensuring the timeliness of data transmission in industrial wireless networks, while effectively reducing data overdue and energy consumption.

[0060] (2) This invention proposes a link scheduling method based on information age, which uses Markov decision process to construct scheduling strategy, avoiding the problem of dimensionality curse caused by excessive state space, and uses curiosity network to accelerate the learning speed of scheduling method, and schedules in real time according to the weighted sum of channel conditions, information age and overdue rate, so that the method can be deployed in large-scale industrial wireless network system.

[0061] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0062] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein:

[0063] Figure 1 This is a schematic diagram of the industrial wireless network structure provided by the present invention;

[0064] Figure 2 Markov chain diagram for the channel model provided in this invention;

[0065] Figure 3 This is a schematic diagram of the scheduling network training.

[0066] Figure 4 This is a flowchart of the joint optimization scheduling method for industrial wireless networks based on information age according to the present invention. Detailed Implementation

[0067] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0068] Figure 1 This is a schematic diagram of the industrial wireless network structure constructed in this invention. The network consists of N energy-constrained sensor nodes and a common destination node. Data sampled by the sensor nodes is transmitted to the destination node via a wireless channel. n∈{1,2,...,N} represents the index of the sensor node; the system schedules data packets through time slots, where t∈{1,2,...,T} represents the time slot index, and T is the maximum time slot number; the sampling period of sensor node n is H. n In each time slot, the probability of sampling hard real-time data and soft real-time data is g, respectively. n and 1-g n The cutoff times for the hard real-time data and soft real-time data of sensor node n are X, respectively. n Each time slot and Y n There are 1 time slot; the data lengths of the hard real-time data and soft real-time data of sensor node n are x, respectively. n and y n The sensor node's buffer can only store one data packet, and the buffer is refreshed before each sampling. Only one sensor node can be scheduled to send data to the destination node in time slot t, and all data transmission can be completed within one time slot. The sensor node supports three working modes: send, idle, and sleep. The sleep mode will only appear after the data transmission is completed.

[0069] Figure 2 This is a Markov chain diagram of the channel model. Assume the channel state is time-varying and model the channel as a Fritschman channel model. The channel is quantized into K states, with a channel state space β = {0, 1, 2, ..., K}. Channel states 0 to G are error-free states, and G+1 to K are error-prone states. Let p... g p represents the probability of successful data transmission in error-free conditions. b Let h(t) be the probability of successful data transmission under error conditions. Let h(t) be the channel state at time slot t, and the transition probability between channel states be represented by p. i,j=Pr(h(t+1)=j|h(t)=i). Then the state transition matrix can be written in the following form:

[0070]

[0071] Figure 3 This is a schematic diagram of the scheduling network training, such as... Figure 3 As shown, the scheduling network includes a current Q-network and a target Q-network. Inputting the system state and the current decision into the environment yields the next state. Then, inputting the current state into the curiosity network obtains the predicted value s' for the next state. c (t), and obtain the internal reward r by comparing it with the actual next state s`(t) in the environment. i (t).

[0072] r i (t)=δ‖s`(t)-s` c (t)‖ 2

[0073] Where δ is the scaling factor. The loss function of a curiosity network. for:

[0074]

[0075] Total reward r total (t) represents the external reward r e (t) and intrinsic reward r i The sum of (t), then the interaction experience (s(t),A(t),r) total (t),s(t+1)) are put into the experience pool, and a batch of experience is selected from the experience pool to update the network.

[0076] The target value y(t) can be obtained through the set of experiences.

[0077]

[0078] The loss function L(θ) is obtained from the target value y(t). t ):

[0079] L(θ t )=(y(t)-Q(s(t),A(t)|θ t )) 2

[0080] The network parameters are updated using the gradient descent method, and after the network reaches the termination condition, a scheduling method for industrial wireless networks is obtained.

[0081] Figure 4This is a flowchart of the scheduling network training process for the joint optimization scheduling method for industrial wireless networks based on information age provided by the present invention, which specifically includes the following steps:

[0082] V1-V4: Obtain system parameter information of industrial wireless network, construct and schedule network structure and initialize network information, and then initialize the state space, action space and cost function in the network.

[0083] V5-V8: The target node updates the data packets of each sensor node and calculates the information age. It inputs the values ​​of the state space and action space under the current time slot into the Q network to obtain the target value, calculates the cost function under the current state and obtains the next state information, calculates the intrinsic reward through the curiosity network, and stores the obtained experience set into the experience pool.

[0084] V10-V14: A batch of interactive experiences is randomly sampled from the experience pool. The Q-value is estimated using a Q-network, and the target value is calculated based on the target Q-network. The loss function of the Q-network is defined based on the error between the target value and the Q-value, and the loss function of the curiosity network is defined based on the intrinsic reward. Gradient descent is used to update the parameters of both the Q-network and the curiosity network. Every V steps, the parameters of the Q-network are synchronized to the target Q-network to enhance training stability. As training progresses, the Q-network's approximation of the target value gradually improves, and eventually the average information age of the system tends to stabilize.

[0085] V15-V16: The trained target Q-network parameters are saved to generate a scheduling network for decision-making. The system uses this network to perform feature analysis on the current state and makes the optimal decision based on the analysis results in the current time slot.

[0086] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A joint optimization scheduling method for industrial wireless networks based on information age, characterized in that, The method specifically includes the following steps: S1: Obtain system information of industrial wireless networks, establish information age update model and energy consumption update model based on network structure, and construct a scheduling network that learns the optimal scheduling strategy; The information age is calculated based on the periodic sampling characteristics of sensor nodes, and is defined as follows: The update process for the information age at sensor node n is as follows: in, f n (t)∈{0,1} represents the sampling action of sensor node n at time slot t. It is 1 when data sampling is performed, and 0 otherwise. definition Indicates the age of the information at the destination node. The update process is as follows: in, d n ( t )∈{0,1} indicates whether sensor node n successfully sent a data packet at time slot t; 1 indicates successful data transmission, and 0 indicates unsuccessful data transmission. Define the power of sensor node n in transmit mode and idle mode as follows: P e and P i , Let be the time spent by sensor node n transmitting data in time slot t, and let be the energy consumption of sensor node n in time slot t. ,but and The update process is as follows: in, x n and y n These are the data lengths of the hard real-time data and the soft real-time data of sensor node n, respectively. The time slot length, R For transmission rate, This represents the buffer data of sensor node n at time slot t. =1 indicates hard real-time data, and the opposite indicates soft real-time data; b n ( t b ∈ {0,1,2} represents the operating mode of sensor node n in time slot t. n (t)=0 indicates sleep mode, b n (t)=1 indicates idle mode, b n (t)=2 indicates the transmission mode; S2: Construct a multi-objective optimization problem that minimizes the average information age and energy consumption of the system under the constraint of the data deadline overdue rate. Based on the relevant parameters of the information age, energy consumption and overdue rate of the system's target node, establish the system's state space, action space and cost function, and transform the link scheduling optimization problem into a constrained Markov decision process. S3: Using improved depth Q The network approximates the state value function and uses a curiosity network to accelerate the convergence speed. Specifically, the information age of the target node is updated in each time slot, the next state is obtained using the current state and decision input and stored in the experience pool, and the network loss function is stabilized, thus obtaining the link scheduling method.

2. The joint optimization scheduling method for industrial wireless networks according to claim 1, characterized in that, In step S1, establishing the information age update model and the energy consumption update model specifically includes: An industrial wireless network consists of N energy-constrained sensor nodes and a common destination node. Data sampled by the sensor nodes is transmitted to the destination node via a wireless channel; n∈{1,2,...,N} represents the index of the sensor node; the system schedules data packets through time slots, let t∈{1,2,...,T} represent the time slot index, where T is the maximum time slot number; the sampling period of sensor node n is H. n In each time slot, the probability of sampling hard real-time data and soft real-time data is g, respectively. n and 1-g n The cutoff times for the hard real-time data and soft real-time data of sensor node n are X, respectively. n Each time slot and Y n There are 1 time slot; the data lengths of the hard real-time data and soft real-time data of sensor node n are x, respectively. n and y n The sensor node's buffer can only store one data packet, and the buffer is flushed before each sampling. Only one sensor node can be scheduled to send data to the destination node in time slot t, and all data transmission can be completed within one time slot. The sensor node supports three operating modes: transmit, idle, and sleep. Sleep mode only occurs after data transmission is complete. (Use u) n (t) represents the buffer data of sensor node n at time slot t, u n (t)=1 indicates that the buffer contains hard real-time data, and vice versa for soft real-time data; Assume the channel state of the system is time-varying, and model the channel using the Fritschman channel model; quantize the channel into K states, with a channel state space β={0,1,2,...,K}; channel states 0 to G represent error-free states, and G+1 to K represent error-prone states; let p g p represents the probability of successful data transmission in error-free conditions. b Let h(t) be the probability of successful data transmission under error conditions; let h(t) be the channel state at time slot t, and the transition probability between channel states be expressed as p. i,j =Pr(h(t+1)=j|h(t=i); then the state transition matrix can be written in the following form: 。 3. The joint optimization scheduling method for industrial wireless networks according to claim 2, characterized in that, In step S2, a multi-objective optimization problem is constructed to minimize the average information age and energy consumption of the system under the constraint of the data deadline overdue rate. This specifically includes the following steps: S21: Establish a joint optimization objective for information age and energy consumption, then the long-term average information age and energy consumption weighted sum at the objective node Represented as: in, β Indicates the weighting parameter; S22: Define Π as the set of all feasible strategies, where strategy π∈Π; the optimization problem is formally expressed as: The constraints are: in, π * This represents the optimal scheduling strategy. The threshold representing the maximum average delinquency rate; This represents the number of data points that sensor node n failed to successfully transmit within the deadline from the initial time slot to time slot t. This represents the number of data samples collected by sensor node n from the initial time slot to time slot t. ∈{0,1} means If sensor node n is scheduled in time slot t, then... =1, otherwise =0; S23: Decoupling the optimization problem using the Lagrange relaxation method, by introducing Lagrange multipliers to address the delinquency rate constraint. λ Transform into the objective function: in, This indicates the optimization objective.

4. The joint optimization scheduling method for industrial wireless networks according to claim 3, characterized in that, In step S2, the state space of the system is established. for: in, , , ， ， ， ； The established system's action space for: in, ; The established system cost function for: in, This indicates whether the data of sensor node n has expired at time slot t.

5. The joint optimization scheduling method for industrial wireless networks according to claim 4, characterized in that, Step S3 specifically includes the following steps: S31: Based on the current situation Q The network will display the current system status. Input to network to obtain decision and execute decisions; S32: After execution, input the current state into the Curiosity Network to obtain the predicted value of the next state. And by comparing the actual next state in the environment. Make comparisons and obtain intrinsic rewards ; in, The scaling factor; the loss function of the curiosity network. for: Total Rewards r total ( t ) as external rewards r e ( t and intrinsic rewards r i ( t The sum of ) and then the interactive experience ( s ( t ), A ( t ), r total ( t ), s ( t Add +1) to the experience pool, select a batch of experience from the experience pool, and use it to update the network; S33: The target value can be obtained through a set of experiences. y ( t ); in, γ Indicates the discount factor. Indicate target Q network Q value, Indicates that the current Q network Q value The action with the smallest value, and They represent the current Q Network parameters and objectives Q Network parameters; For the present Q Network output Q value, It is a value function. Advantage function for each action, N +1 indicates that there is an option not to send data in the scheduling decision; S34: From the target value y ( t ) Obtain the loss function : S35: Update network parameters according to gradient descent method, and wait for the network to reach the termination condition to obtain the scheduling method for industrial wireless networks.

Images