Mixed channel industrial internet of things scheduling method based on information age and throughput

Abstract

The invention relates to a mixed channel industrial internet of things scheduling method based on information age and throughput, and belongs to the technical field of industrial wireless networks. According to the method, for a multi-node industrial Internet of Things system in which a fast unreliable channel and a slow reliable channel coexist, the overdue probability that the information age of the system exceeds a threshold value is minimized under the condition that the limited bandwidth constraint and the minimum throughput demand of all nodes are met. An original optimization problem is decoupled into a node link scheduling sub-problem and a scheduling node channel selection sub-problem, a channel parameter driven region division mechanism, random network distillation and deep reinforcement learning are adopted to cooperatively solve the two types of sub-problems, and a multi-node scheduling scheme formed by two sub-strategies is formed. According to the method, joint optimization of information age and throughput of multiple sensors under a mixed channel is considered, strict freshness of industrial wireless network data delivery is guaranteed, and differentiated throughput requirements of all nodes are ensured.

Description

Technical Field

[0001] This invention belongs to the field of industrial wireless network technology and relates to a hybrid channel industrial IoT scheduling method based on information age and throughput. Background Technology

[0002] With the deepening of Industry 4.0 and the large-scale deployment of Industrial Internet of Things (IIoT) technologies, typical industrial scenarios such as environmental monitoring and smart manufacturing are placing more stringent demands on the timeliness of networks. To accurately quantify the timeliness of data delivery, Age of Information (AoI) has been proposed as a new measure of data freshness, defined as the time elapsed since the most recently received data at the destination was generated.

[0003] In practical applications of the Industrial Internet of Things (IIoT), average information age is insufficient to effectively capture low-probability, high-impact extreme age events. These sporadic high information age phenomena can lead to system failures such as decreased equipment control precision and delayed production decisions, thus impacting system safety. Furthermore, the significantly different transmission characteristics of heterogeneous channels in industrial environments further constrain the optimization of system timeliness. Fast, unreliable channels can achieve high-speed, low-latency transmission but are susceptible to environmental interference, resulting in high packet loss rates. While slow, reliable channels offer stable connections and high transmission success rates, their low-speed characteristics make them unsuitable for meeting low-latency requirements. Therefore, in multi-source, single-destination IIoT systems, researching how to minimize the average information age expiration probability of all nodes while satisfying the constraints of limited bandwidth and minimum node throughput under a hybrid channel environment where fast, unreliable, and slow, reliable channels coexist has become a significant challenge. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide a hybrid channel industrial IoT scheduling method based on information age and throughput. For hybrid channel multi-source industrial wireless networks, under the conditions of satisfying bandwidth constraints and minimum node throughput constraints, the method minimizes the probability of information age exceeding the threshold, thereby ensuring both the strict freshness of data delivery in industrial wireless networks and the real-time reliable transmission of data.

[0005] To achieve the above objectives, the present invention provides the following technical solution: A hybrid channel industrial IoT scheduling method based on information age and throughput, comprising the following steps: S1. Obtain the network parameters of the industrial IoT system, calculate the transmission success probability of the fast unreliable channel in the mixed channel, and calculate the transmission status of the slow reliable channel at the current time slot. S2. Calculate the information age and age delinquency probability of all nodes, and update the model according to the throughput of the nodes to obtain the throughput debt of the corresponding nodes. S3. Establish an optimization problem that minimizes the probability of the system's information age exceeding the threshold under the constraints of limited bandwidth and minimum throughput of the base station, and decouple it into a link selection subproblem for all nodes and a channel selection subproblem for the scheduling node; S4. For the link scheduling subproblem and the channel allocation subproblem, deep reinforcement learning method and channel parameter-driven region partitioning heuristic method are used to solve them respectively; S5. For the original optimization problem, a scheduling scheme consisting of two sub-policies is obtained by using deep reinforcement learning, random network distillation and heuristic channel selection sub-policies.

[0006] Furthermore, in step S1, the hybrid channel industrial IoT system is composed of... M It consists of one node, two heterogeneous channels, and one base station. The base station transmits data packets of interest to the corresponding nodes through either a fast unreliable channel or a slow reliable channel. This represents the node's index; the system delivers data based on time slots, one data slot per data slot, and... This represents the index of a time slot; only one data packet can be scheduled for delivery per time slot, and the arrival of data packets at the base station follows an on-demand generation model. Indicates the base station in the time slot The scheduling decision, in which This indicates that no node has been scheduled. Represents a node Scheduled for data updates, if the current time slot is used to schedule sensor nodes m ,but ,otherwise ; The base station and nodes are equipped with antennas that switch between two different frequency bands to support hybrid channel transmission; it is assumed that data packets can be transmitted through either a fast unreliable channel or a slow reliable channel; the fast unreliable channel is a Gilbert time-varying channel based on a two-state Markov chain, where the good state is... Channel state The probability of successful transmission is 1, and the bad state is... Channel state The probability of successful transmission is 0, and the channel state is... and The self-transition probabilities are respectively and Slow reliable channels can be used continuously Maintain stable transmission and connection within each time slot; make For time slots The initial state of the slow reliable channel, i.e., the remaining transmission time slots for the data packets transmitted through the slow reliable channel in the current time slot, where... This indicates that the slow, reliable channel is idle and available for new data packet transmission; assuming Indicates time slot Initially, the base station makes a channel selection decision, where This indicates that the data packet will be transmitted via a fast unreliable channel or a slow reliable channel. This indicates that no new data packets were allocated to any channel; Considering that the mixed channel state directly affects node scheduling and channel selection decisions, the relationship between node scheduling and channel selection is determined by the current state of the slow reliable channel, which is expressed as:

[0007] In the above formula, when the slow reliable channel is idle, i.e. This indicates that both channels are ready to send data packets, and the base station needs to simultaneously decide on the scheduling equipment. and transmission channel When the slow, reliable channel is occupied, i.e. When this happens, sending new packets is not allowed. and .

[0008] Furthermore, in step S2, it is set that in the time slot Start and Node The associated information is age. It depends on and The combined effects of these factors; when choosing a fast, unreliable channel for data transmission, i.e. When, if node Scheduled and fast unreliable channels in time slots In state ,Right now and At that time, node Successfully received data packets from the base station, node The information age decreases to the transmission duration of a fast, unreliable channel; otherwise, the node... The age increases by 1 with each time slot, node The expression for age is as follows:

[0009] When a slow, reliable channel is selected for data transmission, that is... At that time, in the current time slot Node scheduling decisions made in This only takes effect after transmission on a slow, reliable channel is completed, and the information age of all nodes will increase by 1 in the next time slot, i.e. Meanwhile, the remaining transmission time of the slow reliable channel is updated to ; When waiting for a slow, reliable channel to complete data transmission, i.e. When, if node In the time slot Time is scheduled and time slot It is the transmission completion time, i.e. and When, then the node Data packet successfully received, node The information age decreases to the transmission duration of a slow, reliable channel; otherwise, the node... As the information age increases by 1, the remaining transmission time slots of the slow, reliable channel decrease by 1; thus, when At that time, node The expression for age is as follows:

[0010] The remaining transmission time of the slow reliable channel The evolution process is as follows:

[0011] Therefore, given a node age threshold ,node The age of being overdue is expressed as:

[0012] in Table of the probability of a random event occurring. This represents the total number of time slots. It is an indicator function that returns 1 when the condition is met; Through random variables , and ,node Long-term throughput is defined as:

[0013] in For nodes Is it in a time slot? Data packets were successfully received; assuming Represents a node Minimum throughput requirement, In time slot Start node The associated throughput liability, its update and iteration process is as follows: .

[0014] Furthermore, in step S3, the process of establishing an optimization problem based on the acquired information age, throughput debt, and random variables related to the hybrid channel for each node is as follows: over a long period of time Under the condition of any feasible strategy Below, the expected average information age deprecation probability of all nodes. Represented as:

[0015] in, This indicates the initial system state information, specifically the age. Expressing conditional expectation; The original optimization problem of minimizing the probability of the system's average information age becoming expired is expressed as:

[0016] In the formula, Let represent the set of all feasible scheduling strategies, where ; This represents the optimal scheduling strategy.

[0017] Furthermore, in step S3, the hard constraints related to throughput are relaxed to average time constraints, and the relaxed constraints are incorporated into the optimization objective by introducing throughput debt, thereby obtaining the node scheduling sub-strategy. The expression is as follows:

[0018] Similarly, given the node scheduling actions, a channel selection sub-strategy can be obtained. The expression is:

[0019] Therefore, the original optimization problem is decoupled into two sub-policy problems that are executed synchronously: channel selection and node scheduling.

[0020] Furthermore, in step S4, for the link scheduling subproblem of all nodes, a deep reinforcement learning method and random network distillation are used to solve it. Based on the system information age, throughput debt, the previous time slot state of the fast unreliable channel, and the transmission state of the slow reliable channel, the system's state space, action space, and single-step reward function are established to obtain the optimal node scheduling sub-policy for each time slot. The process includes: S411, Define the system state as follows ,in The information age of the node, For throughput debt, For the known channel state of the previous time slot for the fast unreliable channel, and The remaining transmission time slots of the slow, reliable channel; the system's action space is... , This represents the base station's scheduling decision in a time slot; the age overdue number and throughput debt in the next time slot after the action are used as the system's single-step extrinsic reward function. ,in ; S412. Employ a proximal policy optimization algorithm based on actor criticism structure and set up a new policy network. An old policy network and a value network Random network distillation is used to generate auxiliary intrinsic rewards, which are generated by a random network with fixed parameters. Predictive networks with learnable parameters Composition; intelligent agents through With the environment Interaction, collecting experience data To update the parameters in the sub-policy ; S413. Calculate the intrinsic reward using the feature error between the prediction network and the random network. The goal is to minimize relative to a fixed random network The mean squared error of the feature, and its one-step loss function are:

[0021] in, yes The square of the norm; assumption until the time slot The error moving average is updated as follows:

[0022] in, It is a smoothing factor, thus through The non-negative intrinsic reward generated by smooth normalization is represented as ; S414. The policy network and value network of the near-policy optimization algorithm based on the total reward update of extrinsic and intrinsic rewards are calculated as follows:

[0023] in, The weight of intrinsic rewards is used to balance the contributions of extrinsic and intrinsic rewards; At each time step, the policy network State As input, the output is a probability distribution of actions, and then the agent samples an action from the probability distribution; the probability ratio of the new policy to the old policy is defined as:

[0024] Estimated value of the dominance function , It is a state-action value function, expressed as the state-action value function. Take action below Actual rewards received , The value function estimated for a value network represents the state. Initial expected cumulative rewards; S415. Update the policy network via gradient ascent. parameters ,Right now , The learning rate of the new policy network is represented by the pruning function. The probability ratio between the old and new strategies is constrained within an interval. The loss function of the new policy network is then expressed as:

[0025] Update the value network using gradient descent. parameters ,Right now , Let the learning rate of the new policy network be denoted as , then the loss function of the value network is defined as:

[0026] S416, New Strategy Network The final loss function is It is represented as:

[0027] in, For time step Inside entropy, and They represent and Importance weights.

[0028] Furthermore, in step S4, the channel selection sub-problem of the scheduling node is solved using a channel parameter-driven region partitioning heuristic method. Complementary channel regions of the system are constructed using the channel parameters of fast unreliable channels and slow reliable channels. Based on the transmission characteristics of each channel region, a channel region-based channel selection sub-strategy is obtained. The process is as follows: S421. Considering the impact of different channel parameters on the relative performance of fast unreliable channels and slow reliable channels, the mixed channel parameters will be... The possible values ​​are divided into four complementary regions. Their expressions are as follows:

[0029] in, Represented as a fast unreliable channel slave state Switch to state Average latency and transmission delay of slow reliable channels Comparison between them Represented as fast unreliable channel hold-up state Time and transmission delay of slow reliable channels Comparison between them, and This represents the average latency of a fast unreliable channel and the transmission delay of a slow reliable channel under the steady-state distribution of the Gilbert model. Comparison between them; S422, When the hybrid channel is in the channel region At that time, the fast unreliable channel transitions from state Restore to state The required latency and the average latency of a fast, unreliable channel in steady state are both greater than the transmission delay of a slow, reliable channel. For The channel selection strategy is as follows:

[0030] S423, When the hybrid channel is in the channel region At that time, assuming This refers to the number of devices whose information age remains below a threshold after transmission is complete when using a slow, reliable channel. This represents the timeliness debt incurred when using a slow, reliable channel. If both channels are idle, then... Then targeting The channel selection strategy is as follows:

[0031] Otherwise when At that time, ; S424. When the hybrid channel is in the channel region At that time, the previous time slot state of the fast unreliable channel transitions to state. The waiting time cost always exceeds the transmission delay of a slow, reliable channel. Then targeting The channel selection strategy is as follows:

[0032] S425. When the hybrid channel is in the channel region When the final state of the fast unreliable channel is ,Right now When the channel is in use, a fast, unreliable channel is always chosen; conversely, if... Then a decision needs to be made between the diminishing returns of fast unreliable channels and the deterministic costs of slow reliable channels. and If both channels are idle, a slow, reliable channel should be selected; otherwise, a fast, unreliable channel should be used. Then targeting The channel selection strategy is as follows:

[0033] Otherwise when At that time, ; S426, Based on the hybrid channel parameters Obtain the corresponding channel area, and then in the idle time slot of the mixed channel. In the process, the base station observes the current status of all devices and finally makes a channel selection decision by considering the information age of all nodes and the status of fast unreliable channels and slow reliable channels. .

[0034] Furthermore, in step S5, the current system state is observed, node scheduling actions are obtained through deep reinforcement learning and random network distillation, and the corresponding channel selection decision is obtained based on the channel selection sub-policy. The two decisions are executed simultaneously to update the system state, and finally a scheduling scheme for multi-node hybrid channels is formed.

[0035] The beneficial effects of this invention are as follows: (1) The hybrid channel scheduling scheme based on information age and throughput designed in this invention is applied to the downlink transmission industrial Internet of Things system of multi-source single base station under hybrid channel. This ensures that while limiting bandwidth resources and minimum node throughput are guaranteed, the probability of information age exceeding the threshold is minimized. This ensures both the strict freshness of data delivery in industrial wireless network and the real-time reliable transmission of data. By introducing hybrid channel conditions, the scheme is more suitable for channel conditions in actual networks.

[0036] (2) The hybrid channel scheduling scheme based on information age and throughput designed in this invention, in the process of introducing multi-link scheduling and two heterogeneous channel selection, for the hybrid channel composed of fast unreliable channel with two-state Markov chain and slow reliable channel with long-term stable transmission, in the network where the base station sends data packets to multiple nodes, according to the scheduling scheme composed of two sub-policies, link scheduling and channel selection are performed on a time-slot basis to meet the constraints of limited bandwidth and minimum node throughput, while minimizing the long-term average overdue probability of all nodes; by decoupling the constrained information age overdue probability and throughput joint optimization problem and its action space, the complexity of system state space and action space is effectively reduced, thereby reducing the complexity of scheduling algorithm, so that the scheduling scheme can cope with large-scale industrial equipment deployment environment.

[0037] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0038] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein: Figure 1 This is a schematic diagram of the multi-channel, multi-sensor industrial wireless network structure provided in an embodiment of the present invention; Figure 2 This is a flowchart of a hybrid channel industrial IoT scheduling method based on information age and throughput according to an embodiment of the present invention. Detailed Implementation

[0039] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0040] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0041] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "front," and "rear" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0042] Please see Figures 1-2 This invention presents a hybrid channel scheduling method for industrial IoT systems based on information age and throughput. Specifically, it addresses multi-source, single-base station industrial IoT systems operating under hybrid channel conditions. While satisfying bandwidth limitations and minimum node throughput constraints, it minimizes the probability of information age exceeding a threshold. By decoupling the scheduling strategy into link selection among all nodes and channel selection for the scheduling node, it employs deep reinforcement learning, random network distillation, and heuristic channel selection strategies to reduce the state and action space of the problem, resulting in a scheduling scheme composed of two sub-strategies.

[0043] Example This embodiment describes in detail the specific implementation process of a hybrid channel industrial IoT scheduling method based on information age and throughput, which specifically includes the following steps: Step 1: Obtain the system parameters of the industrial IoT system, calculate the information age and age expiration probability of all nodes, update the model according to the throughput of the nodes, obtain the throughput debt of the corresponding nodes, obtain the transmission success probability of the fast unreliable channel at the current time slot based on the Gilbert model that follows a two-state Markov chain and the channel state information of the previous time slot, and calculate the transmission state of the slow reliable channel at the current time slot. Step 2: Based on the information age, throughput debt, and random variables related to the hybrid channel of each node, establish an optimization problem to minimize the overdue probability of the system's information age exceeding the threshold under the constraints of limited bandwidth and minimum throughput of the base station. Considering the strong coupling relationship between the constraints, decouple it into a link selection subproblem for all nodes and a channel selection subproblem for the scheduling node. Step 3: For the link scheduling subproblem and the channel allocation subproblem, deep reinforcement learning and a channel parameter-driven heuristic method for region partitioning are used respectively to solve them, as follows: For the link scheduling subproblem of all nodes, a deep reinforcement learning method and random network distillation are used to solve it, including: establishing the system's state space, action space and single-step reward function based on the system information age, throughput debt, the previous time slot state of the fast unreliable channel and the transmission state of the slow reliable channel, and obtaining the optimal node scheduling sub-policy of the node in each time slot; For the channel selection subproblem of scheduling nodes, a channel parameter-driven region partitioning heuristic method is adopted to solve it, including: constructing complementary channel regions of the system through the channel parameters of fast unreliable channels and slow reliable channels, and obtaining a channel region-based channel selection sub-strategy based on the transmission characteristics of each channel region. Step 4: For the original optimization problem, a scheduling scheme consisting of two sub-policies is obtained by using deep reinforcement learning, random network distillation and heuristic channel selection sub-policy. Specifically, the optimal node scheduling sub-policy is obtained by using deep reinforcement learning, that is, the scheduling network of the sub-policy outputs the number of the current time slot scheduling node, and the corresponding channel selection sub-policy is executed according to the current state of the scheduling node to determine the specific channel to be selected.

[0044] In step 1 of this embodiment, as Figure 1 The diagram shows the structure of a hybrid channel multi-source industrial IoT system. The hybrid channel industrial IoT system consists of... M It consists of one node, two heterogeneous channels, and one base station. The base station transmits data packets of interest to the corresponding nodes through either a fast unreliable channel or a slow reliable channel. This represents the node index; the system delivers data based on time slots, and due to bandwidth limitations, one data segment is delivered per time slot. This represents the index of a time slot; only one data packet can be scheduled for delivery per time slot, and the arrival of data packets at the base station follows an on-demand generation model. Indicates the base station in the time slot The scheduling decision, in which This indicates that no node has been scheduled. Represents a node Scheduled for data updates, if the current time slot is used to schedule sensor nodes m ,but ,otherwise .

[0045] To avoid electromagnetic interference and reduce transceiver size, the antennas equipped with base stations and nodes can switch between two different frequency bands to support hybrid channel transmission; assuming that data packets can be transmitted through either a fast unreliable channel or a slow reliable channel; the fast unreliable channel (referred to as Channel 1) is a Gilbert time-varying channel based on a two-state Markov chain, where the good state is... Channel state The probability of successful transmission is 1, and the bad state is... Channel state The probability of successful transmission is 0, and the channel state is... and The self-transition probabilities are respectively and Slow reliable channels (channel 2 for short) can continuously Maintaining stable transmission and connection within each time slot, For time slots The initial state of channel 2, i.e., the remaining number of transmission time slots for the data packets transmitted through channel 2 in the current time slot, where This indicates that channel 2 is idle and can be used for new data packet transmission; assuming Indicates time slot Initially, the base station makes a channel selection decision, where This indicates that the data packet will be transmitted through channel 1 (or channel 2). This indicates that no new data packets have been assigned to any channel.

[0046] Considering that mixed channel states directly affect node scheduling and channel selection decisions, when channel 2 is idle (i.e. When this happens, it means that both channels are ready to send data packets, and the base station needs to simultaneously decide which device to schedule and which channel to select. and When channel 2 is occupied (i.e. If the condition is met, then sending new packets is not allowed. and Therefore, the relationship between node scheduling and channel selection is determined by the current state of channel 2, which can be expressed as:

[0047] Set in time slot Start and Node The associated information is age. It depends on and The combined effects; when channel 1 is selected for data transmission (i.e. When, if node Scheduled and channel 1 in time slot In state (Right now and When ), node Successfully received data packets from the base station, node The information age is reduced to the transmission duration of channel 1; otherwise, the node... The age increases by 1 with each time slot, node The expression for age is as follows:

[0048] When channel 2 is selected for data transmission (i.e. When ), in the current time slot Node scheduling decisions made in This only takes effect after transmission on channel 2 is completed, and the information age of all nodes will increase by 1 in the next time slot, i.e. Meanwhile, the remaining transmission time for channel 2 is updated to When waiting for channel 2 to complete data transmission (i.e. When, if node In the time slot Time is scheduled and time slot It is the transmission completion time (i.e.) and When ), then the node Data packet successfully received, node The information age is reduced to the transmission duration of channel 2; otherwise, the node... As the information age increases by 1, the remaining transmission time slots of channel 2 decrease by 1; thus, when At that time, node The expression for age is as follows:

[0049] Based on the above analysis, the remaining transmission time of channel 2 The evolution process is as follows:

[0050] Therefore, given a node age threshold ,node The age of overdue age is expressed as ,in It is an indicator function that returns 1 when the condition is met.

[0051] Through random variables , and ,node Long-term throughput is defined as ,in For nodes Is it in a time slot? Data packets were successfully received; assuming Represents a node Minimum throughput requirement, In time slot Start node The associated throughput liability, its update and iteration process is as follows: .

[0052] Step 2 in this embodiment specifically includes the following steps: Step 21: Over a long period of time Under the condition of any feasible strategy Below, the expected average information age deprecation probability of all nodes. Represented as:

[0053] in, This indicates the initial system state information, specifically the age. Expressing conditional expectation; Step 22: To avoid conflicts between links and the constraint of minimum node throughput during scheduling, define... Let represent the set of all feasible scheduling strategies, where Therefore, the original optimization problem of minimizing the probability of the system's average information age becoming expired can be expressed as:

[0054] in, This represents the optimal scheduling strategy.

[0055] Step 23: Channel selection action for multi-source industrial IoT systems with mixed channels. The node scheduling action depends on the state freshness of all nodes. The effectiveness of the optimization is also limited by the transmission properties of the selected channel. These two factors are coupled during the decision-making process and jointly affect the overall optimization performance. Therefore, the original optimization problem is decoupled into two synchronously executed sub-strategies: channel selection and node scheduling. First, the hard constraints related to throughput are relaxed to average time constraints. Then, by introducing throughput debt, the relaxed constraints are incorporated into the optimization objective, thus obtaining the node scheduling sub-strategy. The expression is as follows:

[0056] Similarly, given the node scheduling actions, a channel selection sub-strategy can be obtained. The expression is:

[0057] In step 3 of this embodiment, the link scheduling subproblem for all nodes is solved using deep reinforcement learning and random network distillation, specifically including the following steps: Step 311: To capture system information more completely, the system state is defined as follows: ,in The information age of the node, For throughput debt, Given the known channel state of the previous time slot for channel 1, and The remaining transmission time slots for channel 2; the system's action space is... , This represents the base station's scheduling decision in a time slot; the age overdue number and throughput debt in the next time slot after the action are used as the system's single-step extrinsic reward function. ; Step 312: The proximal policy optimization algorithm employs an actor-critique structure, including a new policy network. An old policy network and a value network Random network distillation is used to generate auxiliary intrinsic rewards, which are generated by a random network with fixed parameters. Predictive networks with learnable parameters Composition; for updating parameters in sub-policies Intelligent agents through With the environment Interactive, collect a batch of experience data, denoted as ; Step 313: The intrinsic reward can be calculated using the feature error between the prediction network and the random network. The parameters of the prediction network remain fixed (initialized only once). The goal is to minimize the mean squared error relative to the fixed random network features, and its one-step loss function is: ,in yes The square of the norm; assumption until the time slot The error moving average, its update process is as follows: ,in It is a smoothing factor, thus through The non-negative intrinsic reward generated by smooth normalization is represented as ; Step 314: Updating the policy network and value network of the proximal policy optimization algorithm requires the total reward based on extrinsic and intrinsic rewards, which is calculated as follows: ,in The weights for intrinsic rewards are used to balance the contributions of extrinsic and intrinsic rewards; at each time step, the policy network... State As input, the output is a probability distribution of actions; the agent then samples an action from this probability distribution. The probability ratio of the new policy to the old policy is defined as follows: Assuming The estimated value of the advantage function can be expressed as: ; It is a state-action value function, expressed as the state-action value function. Take action below Actual rewards received , The value function estimated for a value network represents the state. Initial expected cumulative rewards; Step 315: For the policy network , Updated via gradient ascent, i.e. , The learning rate of the new policy network is represented by the pruning function. The probability ratio between the old and new strategies is constrained within an interval. This ensures the stability of strategy updates. This is expressed as the loss function of the new policy network; for the value network... , Updated via gradient descent, i.e. , Let be the learning rate of the new policy network, and the loss function of the value network is defined as follows: ; Step 316: To ensure the algorithm's full exploration, a new policy network is used. The final loss function is , can be written as ,in For time step Inside entropy, and They represent and Importance weights.

[0058] In step 3, the channel selection subproblem of the scheduling node is solved using a channel parameter-driven heuristic method for region partitioning, specifically including the following steps: Step 321: Considering the relative performance impact of different channel parameters on channel 1 and channel 2, the mixed channel parameters will be... The possible values ​​are divided into four complementary regions. Their expressions are as follows:

[0059] in Represented as channel 1 slave state Switch to state Average waiting time and transmission delay of channel 2 Comparison between them This is represented as the channel 1 hold state. Time and transmission delay of channel 2 Comparison between them, and This represents the average latency of channel 1 and the transmission delay of channel 2 under the steady-state distribution of the Gilbert model. A comparison between them.

[0060] Step 322: When the hybrid channel is in the channel region At that time, channel 1 changes from state Restore to state The required latency and the average latency of channel 1 in steady state are both greater than the transmission delay of channel 2. This indicates that the high-speed transmission advantage of channel 1 is sufficient to compensate for its lack of stability. In this case, for The channel selection strategy is as follows:

[0061] Step 323: When the hybrid channel is in the channel region At that time, assuming This refers to the number of devices whose information age remains below a threshold after transmission is complete when using channel 2. This represents the timeliness debt incurred using channel 2, indirectly reflecting the potential risk of exceeding the age threshold during transmission; if both channels are idle, i.e. Then targeting The channel selection strategy is as follows:

[0062] Otherwise when At that time, ; Step 324: When the hybrid channel is in the channel region At that time, regardless of the state of the previous time slot of channel 1, it transitions to state The waiting time cost always exceeds the transmission delay of channel 2. Therefore, targeting The channel selection strategy is as follows:

[0063] Step 325: When the hybrid channel is in the channel region At that time, the transmission delay with channel 2 In comparison, channel 1 changes from state Restore to state The initial waiting time is relatively short, but the steady-state average waiting time is relatively long; when the final state of channel 1 is (Right now When considering the possibility that channel 1 will be activated in the short term, channel 1 is always chosen; conversely, if Since channel 1 is in state Subsequently, the performance tends to deteriorate, and its expected performance will drop below the steady-state average. A decision then needs to be made between the diminishing returns of channel 1 and the deterministic cost of channel 2. and If the channel is idle, then channel 2 should be selected; otherwise, channel 1 should continue to be used; if both channels are idle, then... Then targeting The channel selection strategy is as follows:

[0064] Otherwise when At that time, ; Step 326: Based on the hybrid channel parameters Obtain the corresponding channel area, and then in the idle time slot of the mixed channel. In the process, the base station observes the current status of all devices and finally makes a channel selection decision by considering the information age of all nodes and the status of channel 1 and channel 2. .

[0065] In step 4 of this embodiment, a deep reinforcement learning method, random network distillation, and heuristic channel selection sub-policy are used to combine the optimal node scheduling sub-policy with the channel region-based channel selection sub-policy to obtain a scheduling scheme consisting of two sub-policies. Specifically, the current system state is observed, node scheduling actions are obtained through deep reinforcement learning and random network distillation, and the corresponding channel selection decision is obtained based on the channel selection sub-policy. The two decisions are executed simultaneously to update the system state, ultimately forming a scheduling scheme for multi-node hybrid channels.

[0066] Figure 2 The following is a flowchart illustrating the execution of the hybrid channel industrial IoT scheduling method based on information age and throughput according to the present invention, which specifically includes the following steps: V1: The time slot scheduling process begins.

[0067] V2~V4: Obtain the parameters of the industrial wireless network system model, construct the policy network and value network that learn the optimal node scheduling sub-policy, and determine the network's memory, update step size, maximum number of iterations, system state space, action space, and reward function.

[0068] V5~V10: Execute the scheduling action exploration strategy. If the maximum number of iterations has not been reached, then randomly select an action output from the action space; otherwise, select the action with the maximum number of iterations. The action of the value is output.

[0069] V10~V13: Based on the actions output by the main strategy scheduling network, determine the corresponding sub-strategy to execute. Allocate the number of channels obtained by all scheduling nodes according to the heuristic channel allocation sub-strategy. If the allocation result meets the channel number constraint, the current channel allocation scheme is adopted; otherwise, based on exhaustive enumeration, obtain all possible channel allocation schemes, compare the results using comprehensive weights, and select the one closest to the result as the channel allocation scheme to obtain the channel allocation sub-strategy.

[0070] V14: Executes scheduling decisions and channel allocation results in the current state, calculates the cost function, and stores the current system state, actions, cost function, and next state into the memory.

[0071] V15~V19: The target network randomly draws a batch of data from the memory bank to calculate the target. Value, calculate the current Values ​​and Targets The gradient loss is calculated and the online network parameters are updated using gradient descent, every [percentage missing]. The target network parameters are updated step by step. With continuous iterative training, the obtained strategy, while meeting the constraints of the monitoring center's processing capacity and the number of wireless channels, tends to stabilize the average weighted sum of information age and the importance level of received data.

[0072] V20: Saves the trained target network parameters and generates a main policy scheduling network for sensor nodes. The system schedules nodes using a scheduling scheme consisting of a main policy and sub-policies.

[0073] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A hybrid channel industrial Internet of things scheduling method based on information age and throughput, characterized in that: The method comprises the following steps: S1, acquiring network parameters of an industrial Internet of Things system, and calculating a transmission success probability of a fast unreliable channel in a mixed channel and a transmission state of a slow reliable channel at a current time slot; S2, calculating information age and age expiration probability of all nodes, and updating a throughput debt of a corresponding node according to a throughput of the node; S3, establishing an optimization problem of minimizing an expiration probability of information age exceeding a threshold under a base station bandwidth constraint and a minimum throughput constraint, and decoupling the optimization problem into a link selection sub-problem of all nodes and a channel selection sub-problem of a scheduling node; S4, solving the link scheduling sub-problem and the channel allocation sub-problem by using a deep reinforcement learning method and a heuristic method of region division driven by channel parameters, respectively; S5, obtaining a scheduling scheme composed of two sub-strategies by using the deep reinforcement learning method, random network distillation and a heuristic channel selection sub-strategy for the original optimization problem.

2. The hybrid channel industrial IoT scheduling method based on information age and throughput according to claim 1, characterized in that: In step S1, the hybrid channel industrial IoT system consists of M It consists of one node, two heterogeneous channels, and one base station. The base station transmits data packets of interest to the corresponding nodes through either a fast unreliable channel or a slow reliable channel. This represents the node's index; the system delivers data based on time slots, one data slot per data slot, and... This represents the index of a time slot; only one data packet can be scheduled for delivery per time slot, and the arrival of data packets at the base station follows an on-demand generation model. Indicates the base station in the time slot The scheduling decision, in which This indicates that no node has been scheduled. Represents a node Scheduled for data updates, if the current time slot is used to schedule sensor nodes m ,but ,otherwise ; The base station and node equipped antennas switch between two different frequency bands to support transmission of hybrid channels; it is assumed that data packets can be transmitted over either a fast unreliable channel or a slow reliable channel; the fast unreliable channel is a Gilbert time-varying channel based on a two-state Markov chain, where the good state is , the channel state , the transmission success probability is 1, the bad state is , the channel state , the transmission success probability is 0, the self-transition probabilities of the channel states and are and respectively; the slow reliable channel can maintain stable transmission and connection for consecutive slots; make For time slots The initial state of the slow reliable channel, i.e., the remaining transmission time slots for the data packets transmitted through the slow reliable channel in the current time slot, where... This indicates that the slow, reliable channel is idle and available for new data packet transmission; assuming Indicates time slot Initially, the base station makes a channel selection decision, where This indicates that the data packet will be transmitted via a fast unreliable channel or a slow reliable channel. This indicates that no new data packets were allocated to any channel; Considering that the mixed channel state directly affects the decision of node scheduling and channel selection, and the relationship between node scheduling and channel selection is determined by the current state of the slow reliable channel, the relationship is represented as: In the above formula, when the slow reliable channel is idle, i.e. , it indicates that both channels are ready to send data packets, and the base station needs to simultaneously decide the scheduling device and the transmission channel ; when the slow reliable channel is occupied, i.e. , then a new packet is not allowed to be sent, i.e. and .

3. The hybrid channel industrial IoT scheduling method based on information age and throughput according to claim 2, characterized in that: In step S2, the information age of node at the beginning of the time slot is which depends on the joint influence of and ; when a fast unreliable channel is selected for data transmission, i.e. , if node is scheduled and the fast unreliable channel is in state at time slot , i.e. and , node successfully receives the data packet from the base station, and the information age of node is reduced to the transmission duration of the fast unreliable channel, otherwise the age of node is increased by 1 per time slot, and the age expression of node is as follows: When a slow, reliable channel is selected for data transmission, that is... At that time, in the current time slot Node scheduling decisions made in This only takes effect after transmission on a slow, reliable channel is completed, and the information age of all nodes will increase by 1 in the next time slot, i.e. Meanwhile, the remaining transmission time of the slow reliable channel is updated to ; When waiting for the slow reliable channel to finish data transmission, i.e. if the node is scheduled in time slot and time slot is the transmission finish time, i.e. and , then the node successfully receives the data packet, and the information age of the node is reduced to the transmission duration of the slow reliable channel, otherwise the information age of the node is increased by 1 and the remaining transmission time slot of the slow reliable channel is reduced by 1; thus, when , the age expression of the node is as follows: the remaining transmission time of the slow reliable channel The evolution of the slow reliable channel is as follows: Thus, given a node an age threshold , the age of a node is expressed as: wherein denotes the probability that a random event occurs, denotes the total number of time slots, is an indicator function that returns 1 when the condition is met; Through the random variable , and , the long-term throughput of a node is defined as: wherein is a node whether a time slot successfully received a data packet; assume denotes a node minimum throughput requirement, is a time slot a node associated throughput debt, whose update iteration process is .

4. The hybrid channel industrial IoT scheduling method based on information age and throughput according to claim 3, characterized in that: In step S3, the process of establishing the optimization problem according to the acquired information age of each node, the throughput debt and the random variable related to the mixed channel is as follows: Under the condition of long-term time , the expected average information age expiration probability of all nodes under any feasible strategy is expressed as: ​ wherein, represents the system initial system state information age, represents the conditional expectation; The original optimization problem of minimizing the average information age expiration probability of the system is expressed as: wherein denotes the set of all feasible scheduling strategies, wherein ; denotes the optimal scheduling strategy.

5. The hybrid channel industrial IoT scheduling method based on information age and throughput according to claim 4, characterized in that: In step S3, the hard constraint related to throughput is relaxed to an average time constraint, and the relaxed constraint is incorporated into the optimization objective by introducing a throughput debt, resulting in a node scheduling sub-policy : In the case of scheduling an action at a given node, a channel selection sub-strategy is obtained : Therefore, the original optimization problem is decoupled into two sub-strategy problems of channel selection and node scheduling which are executed synchronously.

6. The hybrid channel industrial IoT scheduling method based on information age and throughput according to claim 5, characterized in that: In step S4, the deep reinforcement learning method and random network distillation are used to solve the link scheduling sub-problem of all nodes, and the state space, action space and single-step reward function of the system are established according to the system information age, the throughput debt, the state of the last time slot of the fast unreliable channel and the transmission state of the slow reliable channel, so as to obtain the optimal node scheduling sub-strategy of the node at each time slot, and the process includes: S411, define the system state as where is the information age of the node, is the throughput debt, is the last time slot channel state known by the fast unreliable channel, and is the number of remaining transmission time slots of the slow reliable channel; the action space of the system is , denotes the scheduling decision of the base station at the time slot; the next time slot age expiration number and the throughput debt after the action is performed are used as the single-step external reward function of the system where ; S412. Employ a proximal policy optimization algorithm based on actor criticism structure and set up a new policy network. An old policy network and a value network Random network distillation is used to generate auxiliary intrinsic rewards, which are generated by a random network with fixed parameters. Predictive networks with learnable parameters Composition; intelligent agents through With the environment Interaction, collecting experience data To update the parameters in the sub-policy ; S413, compute intrinsic reward by predicting network and random network The goal is to minimize the mean squared error of the features with respect to the fixed random network The one-step loss function is: where is the square of the norm; assuming is the moving average of the error up to time slot with the update process wherein, is a smoothing factor, whereby the non-negative intrinsic reward resulting from smoothed normalization is expressed as ; S414, updating the policy network and the value network of the proximal policy optimization algorithm based on the total reward of the extrinsic and intrinsic rewards, which is calculated as: wherein, is a weight for the intrinsic reward, balancing the contribution of the extrinsic reward and the intrinsic reward; At each time step, the policy network The state As input, outputs a probability distribution over actions, and then the agent samples an action from the probability distribution; the probability ratio of the new policy to the old policy is defined as: estimated value of the advantage function , is the state-action value function, denoted as the actual reward obtained by taking action in state , , is the value function estimated by the value network, denoted as the expected cumulative reward starting from state ; S415, update the policy network by gradient ascent parameters of the old policy network i.e. , denotes the learning rate of the new policy network; introduce a clipping function , which constrains the probability ratio between the new and old policies in the interval , the loss function of the new policy network is represented as: updating the parameters of the value network by gradient descent i.e. , denoted as the learning rate of the new policy network, the loss function of the value network is defined as:​ S416, new policy network The final loss function is which is expressed as: wherein, is a time step inside the entropy of and represent the importance weights of and respectively.

7. The hybrid channel industrial IoT scheduling method based on information age and throughput according to claim 5, characterized in that: In step S4, the heuristic method of region division driven by channel parameters is used to solve the channel selection sub-problem of the scheduling node, the complementary channel region of the system is constructed through the channel parameters of the fast unreliable channel and the slow reliable channel, and the channel selection sub-strategy based on the channel region is obtained according to the transmission characteristics presented by each channel region; The process is as follows: S421、Considering the relative performance impact of different channel parameters on fast unreliable channels and slow reliable channels, the possible values of the mixed channel parameter are divided into four complementary regions whose expressions are respectively as follows: in, Represented as a fast unreliable channel slave state Switch to state Average latency and transmission delay of slow reliable channels Comparison between them Represented as fast unreliable channel hold-up state Time and transmission delay of slow reliable channels Comparison between them, and This represents the average latency of a fast unreliable channel and the transmission delay of a slow reliable channel under the steady-state distribution of the Gilbert model. Comparison between them; S422、When the mixed channel is in the channel region , the fast unreliable channel recovers from state to state The required waiting time and the average waiting time when the fast unreliable channel is in the steady state are both greater than the transmission delay of the slow reliable channel , the channel selection strategy for is as follows: S423、When the mixed channel is in the channel region , assume is the number of devices whose information age remains below the threshold after the transmission is completed when the slow reliable channel is used, is expressed as the timeliness debt generated by using the slow reliable channel, if both channels are free, i.e. , then the channel selection strategy for is as follows: Otherwise when then ; S424、When the mixed channel is in the channel region of the fast unreliable channel is converted to the state of the slow reliable channel, the waiting time cost is always more than the transmission delay of the slow reliable channel, then the channel selection strategy for is: S425, when the mixed channel is in the channel region , if the last state of the fast unreliable channel is , i.e. , then the fast unreliable channel is always selected; otherwise, if , a decision has to be made between the decreasing benefit of the fast unreliable channel and the deterministic cost of the slow reliable channel. When and , the slow reliable channel should be selected; otherwise, the fast unreliable channel is used; if both channels are free, i.e. , the channel selection strategy for is as follows: Otherwise when then ; S426, according to the mixed channel parameters The corresponding channel region is obtained, and then the time slot in which the mixed channel is idle In this way, the base station observes the current state of all devices, and finally makes a channel selection decision taking into account the information age of all nodes and the state of fast unreliable channels and slow reliable channels .

8. The hybrid channel industrial IoT scheduling method based on information age and throughput according to claim 5, characterized in that: In step S5, the current system state is observed, the node scheduling action is obtained by the deep reinforcement learning method and random network distillation, the corresponding channel selection decision is obtained based on the channel selection sub-strategy, and the two decisions are executed to update the system state, so as to finally form a scheduling scheme for the mixed channel multi-node.

Images