A Time-Varying Channel Hybrid Data Stream Scheduling Method for Industrial IoT Based on Information Age
By employing a multi-link scheduling method based on information age in the Industrial Internet of Things (IIoT), and utilizing linear programming and deep reinforcement learning to optimize the scheduling strategy, the transmission problem of periodic and random data streams under time-varying channels is solved, thereby improving the determinism and timeliness of data.
Patent Information
- Application Number
- CN202311575313.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-11-23
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2043-11-23
AI Technical Summary
In the Industrial Internet of Things (IIoT), how can we meet the delay tolerance constraints of periodic data streams under time-varying channels, while reducing the information age of random data streams, achieving deterministic transmission of periodic data, and improving the freshness of random data?
A multi-link scheduling method based on information age is adopted. The scheduling scheme consists of a main policy and two sub-policies. For the Gilbert time-varying channel that follows a two-state Markov chain, the link scheduling is performed on a time-slot basis. Linear programming and deep reinforcement learning methods are used to optimize the scheduling policy to meet the time delay tolerance constraint and minimize the information age.
Under time-varying channels, deterministic transmission of periodic data and timely improvement of random data are achieved, reducing the average information age of network random data streams, making it suitable for large-scale industrial equipment deployment environments.
Smart Images

Figure CN117812641B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of industrial network technology and relates to a time-varying channel hybrid data stream scheduling method for industrial IoT based on information age. Background Technology
[0002] In the Industrial Internet of Things (IIoT), such as real-time industrial scenarios like environmental monitoring and industrial production, devices universally require timely information transmission. To more accurately quantify the timeliness of information, Age of Information (AoI) has been proposed as a new measure of data freshness. Compared to latency, which only considers transmission time, it comprehensively considers the freshness of data from its generation to the present moment from the destination perspective.
[0003] In the Industrial Internet of Things (IIoT), data streams with different arrival patterns typically support various business operations, each with different requirements for the timeliness of information transmission. Periodic sensor data is often used for precise control tasks; if the delay of this type of data stream delivered to the control center exceeds an appropriate deadline, it may lead to erroneous decisions. For random data, there are usually no strict delivery deadlines, and the focus is more on ensuring timely data delivery. Therefore, in IIoT systems with mixed data streams, a significant challenge lies in researching how to satisfy the delay tolerance constraints of periodic data streams while minimizing the information age of random data streams under time-varying channel conditions. This aims to achieve deterministic transmission of periodically arriving data and maximize the freshness of randomly arriving data. Summary of the Invention
[0004] In view of this, the purpose of this invention is to provide a time-varying channel hybrid data stream scheduling method for industrial IoT based on information age. For industrial IoT data that arrives both periodically and randomly, this method considers time-varying channels and ensures that periodically arriving data meets the delay tolerance constraint while minimizing the information age of randomly arriving data. This effectively reduces the average information age of random data streams in the network and ensures deterministic transmission of periodic data.
[0005] To achieve the above objectives, the present invention provides the following technical solution:
[0006] A hybrid data stream scheduling method for industrial IoT based on information age is proposed. For Gilbert time-varying channels that follow a two-state Markov chain, in a network where multiple nodes send data packets to the base station, the method performs link scheduling on a time-slot basis according to a multi-link scheduling scheme consisting of a main strategy and two sub-strategies. This satisfies the delay tolerance constraint of nodes with periodic arrival while minimizing the long-term average information age of nodes with random arrival.
[0007] The method specifically includes the following steps:
[0008] S1: Obtain parameter information of the industrial IoT system, calculate the queuing delay of nodes with periodic arrival, update the model according to the information age of nodes with random arrival, obtain the average information age of the corresponding nodes, and obtain the probability of the current time slot being in a certain channel state based on the Gilbert time-varying channel following a two-state Markov chain and the channel state information of the previous time slot.
[0009] S2: Based on the information age, queuing delay, channel state of the previous time slot and time delay tolerance of each node, establish the system's state space, action space and single-step cost function, construct the original optimization problem of minimizing the long-term average information age under the time delay tolerance constraint, and obtain two types of single-node optimization subproblems for nodes with different arrival modes by decoupling the original optimization problem through Lagrange multipliers.
[0010] S3: For the two types of single-node optimization subproblems, linear programming and a heuristic method based on a time-slot-based Lyapunov drift framework are used to solve them respectively. Specifically:
[0011] For the single-node optimization subproblem with periodic arrival, a linear programming method is used to solve it, including: using the stationary state distribution of the node Markov chain and the node scheduling probability, the subproblem is transformed into a linear programming problem, and the optimal opportunity scheduling strategy of the node in each time slot is obtained through the linear programming solver.
[0012] For the single-node optimization subproblem with random arrival, a heuristic method based on the time slot Lyapunov drift framework is adopted to solve it, including: by defining the Lyapunov function, constructing the Lyapunov drift expression of the system, and obtaining a heuristic scheduling sub-policy based on minimizing the Lyapunov drift of the system.
[0013] S4: For the original optimization problem, deep reinforcement learning, a truncation method, an optimal opportunity scheduling strategy, and a heuristic scheduling sub-strategy are employed to obtain a multi-link scheduling scheme consisting of one main strategy and two sub-strategies; specifically:
[0014] In the upper-level master policy-based scheduling strategy, a deep Q-network is used to approximate the system state value function, and a deep reinforcement learning method is used to obtain the scheduling opportunity allocation master policy, that is, the scheduling network of the master policy outputs the current time slot scheduling node type.
[0015] In the lower-level scheduling strategy based on two sub-policies, the corresponding sub-policy is executed to determine a specific node to be scheduled based on the decision of the upper-level scheduling, i.e. the type of the node to be scheduled. This includes: for the decision of scheduling nodes with periodic arrival, a truncated opportunistic scheduling sub-policy is obtained based on the truncation method and the optimal opportunistic scheduling strategy; for the decision of scheduling nodes with random arrival, a heuristic scheduling sub-policy is adopted.
[0016] Further, in step S1, parameter information of the industrial IoT system is obtained, including the channel state transition probability, the average information age of nodes arriving randomly, and the latency tolerance of nodes arriving periodically. This specifically includes the following steps:
[0017] S11: Let N be the origin of N P A node with periodic arrival and N R A set consisting of randomly reachable nodes And a hybrid data stream industrial IoT system consisting of a base station; each time slot can only schedule one data delivery, for nodes with random arrival. The arrival of data follows a Bernoulli distribution with an arrival probability of r. i For nodes with periodic arrival times The data arrival period is P j The deterministic deadline is D. j Let t∈{1,2,...,T} be the time slot index, u k This represents the scheduling decision of the base station; a value of 1 indicates the scheduling node. The channel is a Gilbert time-varying channel based on a two-state Markov chain, where the good state is G, the channel state H(t) = 1, and the probability of successful transmission is 1; the bad state is B, the channel state H(t) = 0, and the probability of successful transmission is 0. The state transition matrix is as follows:
[0018]
[0019] S12: Calculate the information age based on the arrival characteristics of the randomly arriving node i data packets from step S11. The local age is... Where f i (t) = 1 indicates that a new data packet has arrived. The destination age expression is as follows:
[0020]
[0021] Among them, u i (t) represents the base station scheduling decision at time slot t, when u i If (t) = 1, then node i is scheduled; otherwise, node i is not scheduled. When a node is scheduled and the channel state is good, the transmission is successful and the destination age is updated to the local age; otherwise, the destination age is incremented by one per time slot.
[0022] S13: Calculate the node's latency tolerance based on the arrival characteristics of periodically arriving node j data packets from step S11. The node queuing latency is... Where f j (t) = 1 indicates that a new data packet has arrived; let ω j (t) indicates whether a data packet is discarded due to exceeding the deadline. A value of 1 indicates that a data packet has been discarded. The expression is as follows:
[0023]
[0024] in, Represents the time window of a node; u j (τ) represents the base station scheduling decision at time slot τ, when u j If (τ) = 1, then schedule node j; otherwise, do not schedule node j.
[0025] Let M = T / P j The time delay tolerance constraint expression is as follows:
[0026]
[0027] Among them, B j The maximum tolerable packet drop rate.
[0028] Furthermore, in step S2, the system's state space, action space, and single-step cost function are established, and the constrained original optimization problem and the two types of decoupled sub-problems are constructed. Specifically, this includes the following steps:
[0029] S21: The system's state space is established as follows:
[0030] s(t)=(Δ(t),Ω(t),h(t))
[0031] Where Δ(t) is the set of information ages of randomly arriving node data packets at time slot t, and Ω(t) is the ω of periodically arriving nodes at time slot t. j The set of (t), h(t) = H(t-1) is the channel state of the previous time slot known at time slot t;
[0032] S22: The system's action space is And there are This represents the scheduling decision of the base station in time slot t;
[0033] S23: The system's single-step cost function includes delay tolerance cost. Information age cost and transmission costs
[0034] S24: Based on the single-step cost function established in step S23, construct the original optimization problem of minimizing the information age under the time delay tolerance constraint, as shown in the following expression:
[0035]
[0036]
[0037]
[0038] Where, π * J(π) represents the optimal strategy for the original optimization problem. * ) represents the minimum long-term average information age of the system achieved under the optimal strategy, and Π represents the set of feasible strategies;
[0039] S25: Based on the original optimization problem established in step S24, the relaxed transmission cost is obtained by using the Lagrange multiplier λ to relax the transmission cost to the time-averaged transmission cost, as shown in the following expression:
[0040]
[0041]
[0042] in, This represents the optimal strategy for the optimization problem after relaxation; further decoupling yields two types of single-node optimization subproblems, namely, single-node optimization subproblems with randomly reachable node i:
[0043]
[0044] And a single-node optimization subproblem with periodically arriving node j:
[0045]
[0046]
[0047] in, and The optimal strategy for the corresponding subproblem Let represent the optimal solution to the single-node optimization subproblem of node i under the Lagrange multiplier λ. Let represent the optimal solution to the single-node optimization subproblem of node j under the Lagrange multiplier λ.
[0048] Furthermore, in step S3, for the single-node optimization subproblem of node j with periodic arrival, a random variable φ is defined. j (t) = 1 indicates that node j has successfully transmitted the data packet to the base station in time slot t. If the data packet was not successfully delivered to the base station φ, then φ is not valid. j (t) = 0, and the expression is as follows:
[0049]
[0050] Then the state of the Markov chain of node j is The optimal opportunity scheduling strategy is obtained by using linear programming. in This indicates that node j is in a certain state. The probability of being scheduled by a base station includes the following steps:
[0051] S311: Order Let be the stationary state distribution of the Markov chain at node j, where and This indicates that node j is in a certain state. The probability; introducing variables This indicates that node j is in a certain state. And the probability of being scheduled by the base station, among which This indicates that node j is in a certain state. The probability of being scheduled by the base station; this step omits the time slot index, then we have
[0052] S312: Based on step S311, the single-node optimization subproblem with periodically arriving node j is transformed into a linear programming problem, expressed as follows:
[0053]
[0054]
[0055]
[0056]
[0057]
[0058]
[0059]
[0060]
[0061]
[0062] Solve the above linear programming problem using a linear programming solver to obtain the optimal chance scheduling strategy for this subproblem.
[0063] S313: Based on step S312, establish the queuing delay of node j at different times. Linear programming problem Then, the optimal opportunity scheduling strategy under different queuing delays is obtained by solving the problem.
[0064] Furthermore, in step S3, for the single-node optimization subproblem with randomly arriving node i, a heuristic method based on the time-slot-based Lyapunov drift framework is adopted, specifically including the following steps:
[0065] S321: Define the Lyapunov function The expression for the Lyapunov drift of the system is obtained as follows:
[0066]
[0067] S322: Based on steps S321 and S12, introduce variable g. i (t) represents the difference between the local age and the destination age of node i:
[0068]
[0069] The Lyapunov drift expression is obtained as follows:
[0070]
[0071] in,
[0072]
[0073]
[0074] S323: Make Q i The strategy for maximizing the value of (t) is a heuristic scheduling sub-strategy for the current time slot with randomly arriving nodes.
[0075] Furthermore, step S4 specifically includes the following steps:
[0076] S41: The reconstructed state space of the system is as follows:
[0077]
[0078] Where G(t) represents the decision scheduling node Expected returns v(t) represents the node The schedulable set, ||·||1 represents the 1-norm, i.e., the sum of the absolute values of the vector elements, ||·|| ∞ This represents the ∞-norm, which is the maximum absolute value among all vector elements.
[0079] S42: The system's state space is... in and The value space is {(0,0),(1,0),(0,1)}, which represent idle, nodes with random arrivals, and nodes with periodic arrivals, respectively.
[0080] S43: Based on the relaxed optimization objective function from step S25 and step S41, construct the reward function, as shown in the following expression:
[0081]
[0082] Where η represents the penalty weight;
[0083] S44: Based on step S43 and the deep Q-neural network reinforcement learning algorithm, obtain the main strategy for scheduling opportunity allocation; the deep Q-neural network reinforcement learning algorithm includes a target network and an online network, and approximates the Q-value function of the state-action pair through the neural network; the algorithm ensures training stability through the target network and the experience replay mechanism, and adopts the ε-greedy strategy to explore actions, selects the action with the maximum value function estimated by the online network with probability ε, otherwise randomly selects an action to allocate the scheduling opportunity for the current time slot;
[0084] S45: Based on step S44, obtain the scheduling opportunity allocation result for the current time slot. If the scheduling involves nodes with periodic arrival, then execute the truncated opportunity scheduling sub-strategy, i.e., obtain the allocation result for each node based on step S313. Scheduling probability in the current time slot The scheduling is based on the principle of prioritizing the node with the highest scheduling probability. If the probabilities are the same, the scheduling is based on the principle of scheduling the node with the highest drop rate.
[0085] S46: Based on step S44, the scheduling opportunity allocation for the current time slot is obtained. If the scheduling involves nodes with random arrival, then the heuristic scheduling sub-policy is executed according to step S323, and the scheduling has the maximum Q. i Nodes with (t) values.
[0086] The beneficial effects of this invention are as follows:
[0087] (1) The multi-link scheduling scheme based on information age designed in this invention is applied to industrial Internet of Things systems where periodic arrival and random arrival data modes coexist under time-varying channels. This ensures deterministic transmission of periodically arriving data while maximizing the timeliness of randomly arriving data delivery. By introducing time-varying channel conditions, the scheme is more suitable for time-varying channel conditions in actual networks.
[0088] (2) The multi-link scheduling scheme based on information age designed in this invention effectively reduces the complexity of the system state space and action space by decoupling the constrained information age optimization problem and its action space during the introduction of time-varying channels and multi-link scheduling, thereby reducing the complexity of the scheduling algorithm and enabling the scheduling scheme to cope with large-scale industrial equipment deployment environments.
[0089] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description
[0090] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein:
[0091] Figure 1 A schematic diagram of an industrial Internet of Things system with both periodic and random data arrival modes coexisting under a time-varying channel, as provided by the present invention;
[0092] Figure 2 This is a schematic diagram illustrating the training process based on deep reinforcement learning in this invention.
[0093] Figure 3 This is a flowchart of the industrial IoT hybrid data stream scheduling method based on information age that satisfies the time-varying channel constraint under the present invention. Detailed Implementation
[0094] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.
[0095] Please see Figures 1-3This invention addresses industrial IoT systems operating in time-varying channels where periodically arriving and randomly arriving data streams coexist. While satisfying the delay tolerance constraint for nodes with periodic arrivals, it minimizes the long-term average information age of nodes with random arrivals. Specifically, for a scenario modeled as a Gilbert channel based on a two-state Markov chain, a constrained information age optimization problem is expressed using a constrained Markov decision process. Lagrange multipliers are used to decouple this optimization problem, resulting in two types of single-node optimization subproblems. Linear programming, Lyapunov drift frameworks, and deep reinforcement learning are employed to solve both the single-node optimization subproblems and the original optimization problem, yielding a multi-link scheduling scheme consisting of one main policy and two sub-policies.
[0096] Figure 1 This is a schematic diagram of an industrial IoT system where periodic and random arrival modes coexist in a time-varying channel, as shown below. Figure 1 As shown, the base station is in N P A node with periodic arrival and N R A set consisting of nodes that arrive at randomly. In the node scheduling, at most one node is selected per time slot for transmission via the wireless time-varying channel. Assume randomly arriving data follows a Bernoulli distribution, and periodically arriving data has an arrival period of P. j The deterministic deadline is D. j Let t∈{1,2,...,T} be the time slot index, u k This represents the scheduling decision of the base station; a value of 1 indicates the scheduling node. Set variable ω j (t) represents the data packet that is dropped if the deadline is exceeded at time slot t; a value of 1 indicates that the packet has been dropped. To ensure deterministic transmission of periodically arriving data, a delay tolerance constraint is set, i.e., the upper bound of the tolerable drop rate of periodically arriving data packets is B. j The expression is as follows:
[0097]
[0098] Among them, f j (τ) indicates whether a new data packet arrives at time slot τ. The information age is calculated based on the arrival characteristics of data packets arriving at node i randomly, and the local age is δ. i s(t+1)=(δ i s(t)+1)(1-f i (t)), where f i (t) = 1 indicates that a new data packet has arrived. The destination age expression is as follows:
[0099]
[0100] If a node is scheduled and the channel status is good, the transmission is successful and the destination age is updated to the local age; otherwise, the destination age increments by one per time slot.
[0101] The channel is a Gilbert channel based on a two-state Markov chain, where the good state is G, the channel state H(t) = 1, and the probability of successful transmission is 1; the bad state is B, the channel state H(t) = 0, and the probability of successful transmission is 0. The state transition matrix is as follows:
[0102]
[0103] Based on the above analysis, a constrained Markov decision process and the original optimization problem are established, with the following specific expression:
[0104]
[0105] Further utilizing the Lagrange multiplier λ to relax the transmission cost to the time-averaged transmission cost, we obtain the relaxed optimization problem, which is then decoupled into two subproblems. The single-node optimization subproblem with randomly arriving node i is as follows:
[0106]
[0107] And a single-node optimization subproblem with periodically arriving node j:
[0108]
[0109] in and This is the optimal strategy for the corresponding subproblem.
[0110] For a single-node optimization subproblem with periodically arriving node j, define a random variable φ. j (t) = 1 indicates that node j has successfully transmitted the data packet to the base station in time slot t. If the data packet was not successfully delivered to the base station φ, then φ is not valid. j (t) = 0, and the expression is as follows:
[0111]
[0112] Then the state of the Markov chain of node j is The optimal opportunity scheduling strategy is obtained by using linear programming. in This indicates that node j is in a certain state. The probability of being scheduled by the base station.
[0113] make Let be the stationary state distribution of the Markov chain at node j, where and This indicates that node j is in a certain state. The probability; introducing variables This indicates that node j is in a certain state. And the probability of being scheduled by the base station; transforming the single-node optimization subproblem of node j with periodic arrivals into a linear programming problem, the expression is as follows:
[0114]
[0115] Solve the above linear programming problem using a linear programming solver to obtain the optimal chance scheduling strategy for this subproblem. Establish node j with different queuing delays Linear programming problem Then, the optimal opportunity scheduling strategy under different queuing delays is obtained by solving the problem.
[0116] For the single-node optimization subproblem with randomly arriving node i, a heuristic method based on a time-slot-based Lyapunov drift framework is adopted to define the Lyapunov function. The expression for the Lyapunov drift of the system is obtained as follows:
[0117]
[0118] Introducing variable g i (t) represents the difference between the local age and the destination age of node i:
[0119]
[0120] The Lyapunov drift expression is obtained as follows:
[0121]
[0122] in,
[0123]
[0124]
[0125] Make Q i The strategy for maximizing the value of (t) is the heuristic scheduling strategy for the current time slot for nodes with random arrivals.
[0126] Figure 2 This is a schematic diagram of the deep reinforcement learning training method of the present invention, as shown below. Figure 2As shown, deep reinforcement learning is used to learn the main strategy for scheduling opportunity allocation. A system state space, action space, and reward function are established. A deep Q-learning network is introduced to approximate the Q-value function of the state-action pair, constructing an online network and a target network. During the interaction between the online network and the industrial IoT system environment, the system state and current action are used as inputs. An ε-greedy strategy is employed to explore actions, selecting the action with the largest Q-value estimated by the online network with probability ε; otherwise, an action is randomly selected for scheduling opportunity allocation in the current time slot. Based on the action output by the online network, the corresponding sub-strategy is executed for link scheduling. Environmental feedback yields the reward function C-value and the next system state for each state-decision pair, resulting in an experience set. The data is stored in the memory. During each training iteration, the online network randomly selects a small batch of samples from the memory for learning and calculates the loss function for the current state. The parameters of the online network are updated using gradient descent. Every K iterations, the parameters of the online network are copied to the target network. The trained target network, obtained by setting a maximum number of iterations, serves as the main policy scheduling network.
[0127] Figure 3 The flowchart below illustrates the industrial IoT hybrid data stream scheduling method based on information age that satisfies delay tolerance constraints under time-varying channels, as described in this invention. Figure 3 As shown, the specific steps include:
[0128] V1: The time slot scheduling process begins.
[0129] V2~V4: Obtain parameters of the industrial IoT system, construct a scheduling network and memory for learning the optimal master strategy, and determine the update step size, maximum number of iterations, penalty weight, system state space, action space, and reward function.
[0130] V5: Updates the information age of each node with random arrivals and the delay tolerance of nodes with periodic arrivals at the start of the update time slot, as well as the channel state information of the previous time slot.
[0131] V6~V9: Execute the ε-greedy action exploration strategy. If the generated ε is greater than the set threshold ε greedy If the Q-value is positive, then a random action is selected from the action space for output; otherwise, the action with the largest Q-value is selected as the output.
[0132] V10~V15: Based on the actions output by the main policy scheduling network, determine the corresponding sub-policy to execute. If the action is to schedule nodes with randomly arriving nodes, then schedule the node with the maximum Lyapunov drift according to the heuristic scheduling sub-policy; if the action is to schedule nodes with periodically arriving nodes, then schedule according to the truncated opportunistic scheduling sub-policy. Execute the scheduling decision in the current state, calculate the reward function, and store the current system state, action, reward function, and next state in the memory bank.
[0133] V16~V20: The target network randomly selects a batch of data from the memory to calculate the target Q value, calculates the gradient loss between the current Q value and the target Q value, and updates the online network parameters through gradient descent. The target network parameters are updated every K steps. With continuous iterative training, the obtained policy tends to stabilize the average information age while satisfying the time delay tolerance constraint.
[0134] V21~V23: Save the trained target network parameters and generate a main policy scheduling network for scheduling opportunity allocation. The system performs scheduling through a multi-link scheduling scheme consisting of one main policy and two sub-policies.
[0135] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.
Claims
1. A time-varying channel hybrid data stream scheduling method for industrial IoT based on information age, characterized in that, For Gilbert time-varying channels that follow a two-state Markov chain, in a network where multiple nodes send data packets to the base station, link scheduling is performed on a time-slot basis according to a multi-link scheduling scheme consisting of a main strategy and two sub-strategies. This satisfies the delay tolerance constraint of nodes with periodic arrivals while minimizing the long-term average information age of nodes with random arrivals. The method specifically includes the following steps: S1: Obtain parameter information of the industrial IoT system, calculate the queuing delay of nodes with periodic arrival, update the model according to the information age of nodes with random arrival, obtain the average information age of the corresponding nodes, and obtain the probability of the current time slot being in a certain channel state based on the Gilbert time-varying channel following a two-state Markov chain and the channel state information of the previous time slot. S2: Based on the information age, queuing delay, channel state of the previous time slot and random variables related to delay tolerance of each node, establish the system's state space, action space and single-step cost function, construct the original optimization problem of minimizing the long-term average information age under the delay tolerance constraint, and obtain two types of single-node optimization subproblems for nodes with different arrival modes by decoupling the original optimization problem through Lagrange multipliers. S3: For the two types of single-node optimization subproblems, linear programming and a heuristic method based on a time-slot-based Lyapunov drift framework are used to solve them respectively. Specifically: For the single-node optimization subproblem with periodic arrival, a linear programming method is used to solve it, including: using the stationary state distribution of the node Markov chain and the node scheduling probability, the subproblem is transformed into a linear programming problem, and the optimal opportunity scheduling strategy of the node in each time slot is obtained through the linear programming solver. For the single-node optimization subproblem with random arrival, a heuristic method based on the time slot Lyapunov drift framework is adopted to solve it, including: by defining the Lyapunov function, constructing the Lyapunov drift expression of the system, and obtaining a heuristic scheduling sub-policy based on minimizing the Lyapunov drift of the system. S4: For the original optimization problem, deep reinforcement learning, a truncation method, an optimal opportunity scheduling strategy, and a heuristic scheduling sub-strategy are employed to obtain a multi-link scheduling scheme consisting of one main strategy and two sub-strategies; specifically: In the upper-level master policy-based scheduling strategy, a deep Q-network is used to approximate the system state value function, and a deep reinforcement learning method is used to obtain the scheduling opportunity allocation master policy, that is, the scheduling network of the master policy outputs the current time slot scheduling node type. In the lower-level scheduling strategy based on two sub-policies, the corresponding sub-policy is executed to determine a specific node to be scheduled based on the decision of the upper-level scheduling, i.e. the type of the node to be scheduled. This includes: for the decision of scheduling nodes with periodic arrival, a truncated opportunistic scheduling sub-policy is obtained based on the truncation method and the optimal opportunistic scheduling strategy; for the decision of scheduling nodes with random arrival, a heuristic scheduling sub-policy is adopted.
2. The time-varying channel industrial IoT hybrid data stream scheduling method based on information age according to claim 1, characterized in that, Step S1 specifically includes the following steps: S11: Let it be... A node with periodic arrival and A set consisting of randomly reachable nodes A hybrid data stream industrial IoT system consisting of a base station; each time slot can only schedule one data delivery, for nodes with random arrivals. The arrival of data follows a Bernoulli distribution with an arrival probability of . For nodes with periodic arrival times The data arrival cycle is The deterministic deadline is ;make For time slot index, This represents the scheduling decision of the base station; a value of 1 indicates the scheduling node. The channel is a Gilbert time-varying channel based on a two-state Markov chain, where the good state is... Channel state The probability of successful transmission is 1, and the bad state is... Channel state The probability of successful transmission is 0, and the state transition matrix is as follows: S12: Based on the nodes that arrive randomly in step S11 Data packet arrival characteristics calculate information age, local age is ,in This indicates that a new data packet has arrived. The destination age expression is as follows: in, Indicates time slot The scheduling decision of the base station, when Then the scheduling node Otherwise, do not schedule nodes. If a node is scheduled and the channel status is good, the transmission is successful and the destination age is updated to the local age; otherwise, the destination age is incremented by one per time slot. S13: Based on the nodes that arrive periodically in step S11 The latency tolerance of the data packet arrival characteristic calculation node is given by the node queuing latency. ,in This indicates that a new data packet has arrived; let This indicates whether a data packet has exceeded its deadline and should be discarded. A value of 1 indicates that a data packet has been discarded. The expression is as follows: in, , represents the time window of the node; Indicates time slot The scheduling decision of the base station, when Then the scheduling node Otherwise, do not schedule nodes. ; make The time delay tolerance constraint expression is as follows: in, The maximum tolerable packet drop rate.
3. The time-varying channel industrial IoT hybrid data stream scheduling method based on information age according to claim 2, characterized in that, Step S2 specifically includes the following steps: S21: The system's state space is established as follows: in, For time slots A set of information about the age of data packets arriving at randomly selected nodes. For time slots There are nodes that are periodically reached. The set, For time slots The known channel state of the previous time slot; S22: The system's action space is And there are This indicates that the base station is in the time slot. Scheduling decisions; S23: The system's single-step cost function includes delay tolerance cost. Information age cost and transmission costs ; S24: Based on the single-step cost function established in step S23, construct the original optimization problem of minimizing the information age under the time delay tolerance constraint, as shown in the following expression: in, This represents the optimal strategy for the original optimization problem. This represents the minimum long-term average information age of the system achieved under the optimal strategy. Represents the set of feasible strategies; S25: Based on the original optimization problem established in step S24, use the Lagrange multipliers. The relaxed transmission cost is the time-averaged transmission cost, resulting in the following relaxation optimization problem: in, This represents the optimal strategy for the relaxed optimization problem; further decoupling yields two types of single-node optimization subproblems, namely those with randomly reachable nodes. Single-node optimization subproblem: And nodes that arrive periodically Single-node optimization subproblem: in, and The optimal strategy for the corresponding subproblem Represents Lagrange multipliers Next node The optimal solution to the single-node optimization subproblem. Represents Lagrange multipliers Next node The optimal solution to the single-node optimization subproblem.
4. The time-varying channel industrial IoT hybrid data stream scheduling method based on information age according to claim 3, characterized in that, In step S3, for nodes with periodic arrival... The single-node optimization subproblem, defining random variables. Represents a node In the time slot The data packet has been successfully transmitted to the base station. If the data packet was not successfully delivered to the base station... The expression is as follows: Then node The state of the Markov chain is The optimal opportunity scheduling strategy is obtained by using linear programming. ,in Represents a node In state The probability of being scheduled by a base station includes the following steps: S311: Order For nodes The stationary state distribution of the Markov chain, where ,and Represents a node In state The probability; introducing variables , representing a node In state And the probability of being scheduled by the base station, among which Represents a node In state The probability of being scheduled by the base station; this step omits the time slot index, then we have ; S312: According to step S311, nodes with periodic arrival times will be... The single-node optimization subproblem is transformed into a linear programming problem, expressed as follows: Solve the above linear programming problem using a linear programming solver to obtain the optimal chance scheduling strategy for this subproblem. ; S313: Establish nodes according to step S312. Different queuing delays Linear programming problem Then, the optimal opportunity scheduling strategy under different queuing delays is obtained by solving the problem. .
5. The time-varying channel industrial IoT hybrid data stream scheduling method based on information age according to claim 4, characterized in that, In step S3, for nodes with random arrival... The single-node optimization subproblem is addressed using a heuristic method based on a time-slot-based Lyapunov drift framework, specifically including the following steps: S321: Define the Lyapunov function The expression for the Lyapunov drift of the system is obtained as follows: S322: Based on steps S321 and S12, introduce variables. Represents a node The difference between local age and destination age: The Lyapunov drift expression is obtained as follows: in, S323: Make The strategy for maximizing the value is a heuristic scheduling sub-strategy for the current time slot for nodes with random arrivals.
6. The time-varying channel hybrid data stream scheduling method for industrial IoT based on information age according to claim 5, characterized in that, Step S4 specifically includes the following steps: S41: The reconstructed state space of the system is as follows: in, Indicates decision scheduling node Expected returns , Represents a node The set of schedulable properties The 1-norm represents the sum of the absolute values of the vector elements. express - Norm, which is the maximum absolute value among all vector elements; S42: The system's state space is... ,in ,and The value space is , respectively represent idle, nodes with random arrivals scheduled, and nodes with periodic arrivals scheduled; S43: Based on the relaxed optimization objective function from step S25 and step S41, construct the reward function, as shown in the following expression: in, Indicates the penalty weight; S44: Based on step S43 and the deep Q-neural network reinforcement learning algorithm, obtain the primary policy for scheduling opportunity allocation; the deep Q-neural network reinforcement learning algorithm includes a target network and an online network, which approximates state-action pairs through neural networks. Value function; the algorithm ensures training stability through the target network and experience replay mechanism, employing... The strategy explores actions based on probability. Select the action with the maximum value function estimated by the online network; otherwise, randomly select an action to allocate scheduling opportunities for the current time slot. S45: Based on step S44, obtain the scheduling opportunity allocation result for the current time slot. If the scheduling involves nodes with periodic arrival, then execute the truncated opportunity scheduling sub-strategy, i.e., obtain the allocation result for each node based on step S313. Scheduling probability in the current time slot The scheduling is based on the principle of prioritizing the node with the highest scheduling probability. If the probabilities are the same, the scheduling is based on the principle of prioritizing the node with the highest drop rate. S46: Based on step S44, the scheduling opportunity allocation for the current time slot is obtained. If the scheduling involves nodes with random arrival, then the heuristic scheduling sub-policy is executed according to step S323, and the scheduling has the maximum... The node with the value.
Citation Information
Patent Citations
Mixed updating industrial wireless sensor network scheduling method based on information age
CN113891276A
Industrial Internet of Things scheduling method based on information age under cut-off time delay constraint
CN115174419A
