Real-time centralized wireless network scheduling method and device based on deep reinforcement learning

Abstract

The present invention proposes a real-time centralized wireless network scheduling method and system based on deep reinforcement learning. The method comprises: obtaining a wireless network consisting of an access point and multiple interconnected user nodes; at each time slot, the access point obtains the status of each data stream based on the send queue information corresponding to all data streams; the status of all data streams is aggregated to form the environmental state of the current time slot; the access point obtains the traffic model and link quality of all data streams as environmental feature information; inputs the environmental state and environmental feature information into a decision model; the access point executes a scheduling decision corresponding to the output of the decision model; after executing the scheduling decision, the access point receives feedback from the network environment; stores the interaction information, environmental state, and environmental feature information as experience in a sub-region; and extracts experience from an experience replay pool to train and update the current decision model. The present invention does not increase training time rapidly with the number of data streams, and can quickly converge to the optimal real-time throughput.

Description

Technical Field

[0001] The present invention belongs to the technical field of routing network information scheduling in information technology, and in particular relates to a real-time centralized wireless network scheduling method and device based on deep reinforcement learning. Background Art

[0002] In recent years, real-time wireless networks have been increasingly used in a variety of latency-constrained applications. Typical applications include cyber-physical systems (CPSs) used in the Industrial Internet of Things and networked transportation systems, or multimedia communication systems for live video transmission. These systems typically require that data packets be received by the destination node within a deadline, otherwise the packets will be removed from the transmission queue. A key metric for measuring the performance of real-time wireless networks is real-time throughput, which is defined as the ratio of the number of packets successfully received by the destination node to the total number of packets generated in the network.

[0003] In recent years, the problem of how to schedule real-time networks subject to latency constraints to achieve higher throughput has attracted widespread attention. Existing scheduling strategies only consider frame-synchronous traffic models, in which the packet deadlines and arrival periods of data flows in the network are equal. This frame-synchronous traffic model often fails to model many important real-world scenarios. For example, in a typical mobile video conferencing scenario, the latency deadline is often much shorter than the frame generation period. In this scenario, the previously mentioned scheduling methods cannot meet the latency constraints of such real-time data flows.

[0004] Research on scheduling algorithms for general traffic models often fails to achieve good results in real-world scenarios due to high computational complexity or inability to achieve optimal real-time throughput. Furthermore, traditional methods, such as greedy algorithms and dynamic programming, cannot automatically adapt to dynamic network environments. Consequently, existing research efforts have been unable to provide efficient, adaptive, and real-time network scheduling strategies. Summary of the Invention

[0005] Specifically, this application proposes a real-time centralized wireless network scheduling method based on deep reinforcement learning, which includes:

[0006] Step 1: Obtain a wireless network consisting of an access point and multiple interconnected user nodes. The access point schedules all data flows in the wireless network. Before interacting with the network environment, the access point in the wireless network creates a new sub-area in the experience replay pool.

[0007] Step 2: When the access point interacts with the network environment, in each time slot, the access point obtains the state of each data stream based on the sending queue information corresponding to all data streams, and the state of all data streams is combined to form the environment state s of the current t-th time slot.t At the same time, the access point obtains the traffic model and link quality of all data flows as environmental feature information, inputs the environmental state and the environmental feature information into the decision model, and the access point executes the scheduling decision a corresponding to the output result of the decision model. t ;

[0008] Step 3: The access point executes scheduling decision a t After receiving feedback from the network environment t and the end mark end t , the status of all data streams is updated to s t+1 ; The interactive information (s t ,a t ,r t ,s t+1 ,end t ) and the environmental state and the environmental feature information as experience, and store them in the sub-area;

[0009] Step 4: Extract experience from the experience replay pool to train and update the current decision model.

[0010] In the real-time centralized wireless network scheduling method based on deep reinforcement learning, the training process of the decision model includes:

[0011] Create two neural networks with the same structure, namely the current network and the target network. When the access point needs to make a scheduling decision, it inputs the environmental state and environmental characteristics of the current time slot t into the target network to obtain the value of each scheduling action. The access point randomly selects an action to perform scheduling with a probability of ε, and selects the action with the highest value to perform scheduling with a probability of 1-ε, where ε is a decimal between 0 and 1.

[0012] When the access point performs action a t After that, get feedback r t and the end mark end t , the environment state is updated to s t+1 ; The interactive information (s t ,a t ,r t ,s t+1 ,end t ) is stored in the experience replay pool, where end t Is the termination indicator, when t≥L*Prd and t%Prd=0, end t =true, otherwise end t =false, L is a constant positive integer that satisfies:

[0013] L*Prd≥max k∈[1,K] (offset k+D k )

[0014] Among them, Prd is the least common multiple of all data flow periods, D k Indicates the deadline of the data packet in data stream k, offset k Indicates the offset between the arrival time of the first data packet of data stream k and the network start time. t = true, the access point ends its interaction with the current training environment and starts interacting with another training environment;

[0015] After randomly extracting some interactive information from the experience replay pool, the parameters of the current network are updated by minimizing the loss function. The parameters of the target network θ target Every scheduled time, the parameters θ of the current network will be current After reaching the preset training time, the current target network is saved as the decision model.

[0016] In the real-time centralized wireless network scheduling method based on deep reinforcement learning, the calculation method of ε is as follows:

[0017]

[0018] Among them, ε0=0.99, ε min =0.001, train_time refers to the number of environments that the access point has interacted with. The initial value is 0. When the access point finishes interacting with an environment, the value increases by 1.

[0019] In the real-time centralized wireless network scheduling method based on deep reinforcement learning, the state of the data flow is composed of the urgency of the data flow to be scheduled and the data packet arrival factor of the data flow.

[0020] The present invention also proposes a real-time centralized wireless network scheduling system based on deep reinforcement learning, which includes:

[0021] A preparation module is used to obtain a wireless network consisting of an access point and multiple interconnected user nodes. The access point schedules all data flows in the wireless network. Before interacting with the network environment, the access point in the wireless network creates a new sub-area in the experience replay pool;

[0022] The scheduling decision module is used to obtain the status of each data flow according to the sending queue information corresponding to all data flows in each time slot when the access point interacts with the network environment, and the status of all data flows is combined to form the environment state s of the current t-th time slot. tAt the same time, the access point obtains the traffic model and link quality of all data flows as environmental feature information, inputs the environmental state and the environmental feature information into the decision model, and the access point executes the scheduling decision a corresponding to the output result of the decision model. t ;

[0023] Storage module, used for the access point to perform scheduling decision a t After receiving feedback from the network environment t and the end mark end t , the status of all data streams is updated to s t+1 ; The interactive information (s t ,a t ,r t ,s t+1 ,end t ) and the environmental state and the environmental feature information as experience, and store them in the sub-area;

[0024] The update module is used to extract experience from the experience replay pool to train and update the current decision model.

[0025] In the real-time centralized wireless network scheduling system based on deep reinforcement learning, the training process of the decision model includes:

[0026] Create two neural networks with the same structure, namely the current network and the target network. When the access point needs to make a scheduling decision, it inputs the environmental state and environmental characteristics of the current time slot t into the target network to obtain the value of each scheduling action. The access point randomly selects an action to perform scheduling with a probability of ε, and selects the action with the highest value to perform scheduling with a probability of 1-ε, where ε is a decimal between 0 and 1.

[0027] When the access point performs action a t After that, get feedback r t and the end mark end t , the environment state is updated to s t+1 ; The interactive information (s t ,a t ,r t ,s t+1 ,end t ) is stored in the experience replay pool, where end t Is the termination indicator, when t≥L*Prd and t%Prd=0, end t =true, otherwise end t =false, L is a constant positive integer that satisfies:

[0028] L*Prd≥max k∈[1,K] (offset k +Dk )

[0029] Among them, Prd is the least common multiple of all data flow periods, D k Indicates the deadline of the data packet in data stream k, offset k Indicates the offset between the arrival time of the first data packet of data stream k and the network start time. t = true, the access point ends its interaction with the current training environment and starts interacting with another training environment;

[0030] After randomly extracting some interactive information from the experience replay pool, the parameters of the current network are updated by minimizing the loss function. The parameters of the target network θ target Every scheduled time, the parameters θ of the current network will be current After reaching the preset training time, the current target network is saved as the decision model.

[0031] In the real-time centralized wireless network scheduling system based on deep reinforcement learning, the calculation system of ε is as follows:

[0032]

[0033] Among them, ε0=0.99, ε min =0.001, train_time refers to the number of environments that the access point has interacted with. The initial value is 0. When the access point finishes interacting with an environment, the value increases by 1.

[0034] In the real-time centralized wireless network scheduling system based on deep reinforcement learning, the state of the data flow is composed of the urgency of the data flow to be scheduled and the data packet arrival factor of the data flow.

[0035] The present invention proposes a computer-readable storage medium for storing one or more computer programs, which, when executed, are used to implement any of the real-time centralized wireless network scheduling methods based on deep reinforcement learning.

[0036] The present invention proposes a client for use in any of the above-mentioned real-time centralized wireless network scheduling systems based on deep reinforcement learning.

[0037] Compared with the prior art, the present invention has the following advantages:

[0038] 1. This paper proposes a centralized and distributed scheduling method based on deep reinforcement learning. Existing methods, such as the RAC method, are highly complex, and the time required to solve the optimization problem and obtain the optimal scheduling strategy increases exponentially with the number of data streams. The method proposed in this paper does not increase the training time as the number of data streams increases, and can quickly converge to the optimal real-time throughput.

[0039] 2. To adapt to the dynamic changes of real-time wireless networks, this paper modifies the Q function used in deep reinforcement learning, making it dependent not only on state and action but also on environmental characteristics. Compared with existing technologies, this paper is more adaptable to new environments and has higher generalization capabilities in dynamic real-time wireless network environments. The strength of this generalization determines whether the proposed method can be extended to networks with different traffic models and link qualities.

[0040] 3. The distributed scheduling method of the present invention adopts a centralized training and distributed execution framework. The distributed real-time wireless network scheduling method based on deep reinforcement learning is executed distributedly on each sending node. During training, the intelligent agent allows the use of global information to simplify the training process.

[0041] 4. This invention transforms the state of each flow using an exponential function, allowing the state to numerically and intuitively reflect the urgency of scheduling data flow k. Compared to the binary string representation methods proposed in existing methods, the state representation method proposed in this invention enables deep reinforcement learning methods to converge quickly and achieve optimal real-time throughput. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is the implementation framework of the centralized real-time wireless network scheduling method based on deep reinforcement learning in the present invention.

[0043] Figure 2 This is the implementation framework of the distributed real-time wireless network scheduling method based on deep reinforcement learning in the present invention.

[0044] Figure 3 These are network topology diagrams of two classic scenarios to which the present invention is applicable.

[0045] Figure 4 is a functional configuration block diagram of a device for real-time wireless network scheduling for deep reinforcement learning according to an embodiment of the present invention;

[0046] Figures 5 to 12 It is a simulation experiment result diagram of an example of the present invention. DETAILED DESCRIPTION

[0047] The present invention relates to a real-time wireless network scheduling method and device based on deep reinforcement learning, aiming to solve the technical problem of being unable to provide an adaptive dynamic scheduling strategy for data flows under dynamic network conditions and general traffic models. The method includes a centralized method and a distributed method.

[0048] Centralized approaches include: a single-hop network with one access point (AP) and multiple user nodes. The AP acquires global information during each time slot and determines which user node to serve. A general traffic model representation method is designed, and a framework for a centralized real-time wireless network scheduling method based on deep reinforcement learning is established. The interaction between each module is also designed, and the structure of the centralized real-time wireless network scheduling method based on deep reinforcement learning is designed.

[0049] The distributed method includes: a single-hop network has multiple access points (APs) and multiple user nodes, each access point is independently responsible for serving multiple different user nodes, each access point can obtain local information at the beginning of each time slot, and combine the information observed by the wireless channel to decide which user node to serve. When multiple access points want to serve user nodes in the same time slot, a transmission conflict will occur in the network, resulting in transmission failure. A distributed real-time wireless network scheduling method framework based on deep reinforcement learning is established, and a distributed real-time wireless network scheduling method structure based on deep reinforcement learning is designed. The present invention has good model generalization ability, and can quickly converge to the optimal real-time throughput, and can achieve better performance in dynamic network environments and general traffic model scenarios.

[0050] To address the shortcomings of existing technologies, according to a first aspect of the present invention, a real-time wireless network scheduling method based on deep reinforcement learning is proposed. This method provides an efficient, adaptive dynamic scheduling strategy under dynamic network conditions and general traffic models. The technical solutions employed in this invention are divided into two parts: a centralized scheduling method and a distributed scheduling method.

[0051] (1) Centralized scheduling method, including the following steps:

[0052] The centralized scheduling method is applicable to the following scenario: there is one access point (AP) and multiple user nodes in a single-hop network.

[0053] There are multiple data flows in the network, each of which follows a common traffic model. Each data flow has a send queue. Data packets arrive in the send queue sequentially according to the data flow's traffic model, waiting to be sent by the sending node. When a data packet is successfully received by the destination node or has not been successfully received by the destination node after the deadline, the data packet is removed from the send queue. If the data flow is on the AP, the AP acts as the send queue. If the data flow is on the user node, the user node acts as the sending node.

[0054] Data packets in a data stream can be transmitted from a user node to an access point or from an access point to a user node. When the data stream is on a user node, the user node acts as a sending node and the access point acts as a receiving node (i.e., a destination node), and an uplink is formed between the user node and the access point. When the data stream is on an access point, the user node acts as a receiving node and the access point acts as a sending node (i.e., a destination node). A downlink is formed between the user node and the access point. The uplink and downlink are collectively referred to as a link. When data packets are transmitted on a link, they may be affected by the link quality and packet loss may occur. After the sending node sends a data packet in each time slot, the receiving node will send an ACK message to the sending node to inform it whether the data packet has been successfully received. In the present invention, the transmission of ACK is not considered to be affected by the link quality.

[0055] In each time slot, the access point can obtain global information about the environment, namely, information about each data packet in the transmit queue of each data flow and information about the traffic model of each data flow. Based on this global information, the access point decides which data flow to schedule and then sends the data packets in the transmit queue corresponding to that data flow. Since a link is formed between the access point and the user node, scheduling a data flow is equivalent to scheduling a link. Assuming that there are K data flows in the network, each data flow belongs to a common traffic model, that is, data packets arrive in the transmit queue periodically, and data packets in each data flow have a uniform deadline. The common traffic model of the data flow can be represented by the following vector:

[0056] (offset k , prd k , D k , B k , pro k ), k∈K

[0057] Its offset k prd represents the offset between the arrival time of the first data packet of data stream k and the network start time; k represents the arrival period of the data packet in the kth data stream; D k represents the deadline of the data packet in data stream k; B k ∈[1,0] represents the arrival probability of data packets in data stream k. Data packets generated by the data stream of the application layer will be affected by the arrival probability when entering the sending queue and thus cannot enter the sending queue; k represents the link quality of link k. When data packets are transmitted on the link, they will be affected by wireless channel interference and noise, resulting in packet loss. Then the destination node will not be able to successfully receive the data packet.

[0058] Establish a real-time wireless network scheduling method framework based on deep reinforcement learning, including two major modules: the deep reinforcement learning agent and the environment, and design the content of the interaction between each module;

[0059] Furthermore, the design of the two major modules of deep reinforcement learning, the agent and the environment, is as follows:

[0060] (1) Access points in real-time wireless networks are considered as intelligent agents for deep reinforcement learning;

[0061] (2) The abstract environment is a real-time wireless network, including all data flow information in the network and the status information of the corresponding data packets in the sending queue;

[0062] (3) For the centralized scheduling method, there is only one access point in the network, that is, one intelligent agent. The access point obtains global information at each moment, that is, the state of the environment, and performs actions. The environment receives the action and feeds back a reward value to the intelligent agent, thereby realizing the interaction between the access point and the environment. The interaction process between the access point and the environment is modeled as a Markov decision process.

[0063] (4) For distributed scheduling methods, there are multiple access points (APs), i.e., multiple agents, in the network. Each AP obtains local information, i.e., part of the environment state, and performs an action at each moment. The environment receives the action and then feeds back a reward value, thus enabling interaction between the AP and the environment. The interaction between the AP and the environment is modeled as a partially observable Markov decision process.

[0064] Furthermore, in the centralized scheduling method, the state of the Markov decision process is defined as follows: the state of the data flow at time slot t is represented by a two-dimensional vector (q t (k),l t (k)) represents the state s of the environment at time slot t t It is the set of states of all data in time slot t, that is: s t =((q t (1),l t (1)),...,(q t (k),l t (k)),...,(q t (K),l t (K))) indicates.

[0065] Among them, l t (k) represents the arrival factor of the data packet of data stream k at time slot t. If data stream k may have new data packets arriving at the sending queue in the next time slot t+1, then l t (k)=1, otherwise l t (k)=0.

[0066]

[0067] l t (k) can be calculated from the traffic model of the data flow. That is:

[0068]

[0069] q t (k) represents the urgency of data flow k to be scheduled at time slot t:

[0070] Where G is the number of unexpired packets in the send queue of data stream k at time slot t. g is the arrival order of the packets in G data. The arrival time and expiration time of the gth packet in data stream k can be expressed as and The expiration time of a data packet can be calculated based on the traffic model of data stream k using the following formula:

[0071]

[0072] The remaining time until the data packet expires It can be expressed as:

[0073]

[0074] Furthermore, in the centralized scheduling method, the actions in the Markov decision process are defined as follows: the action a of the access point in time slot t is t It selects a certain sending queue and sends the data packet with the shortest expiration time in the queue. t is the number of the scheduled data stream. The action space of the access point can be expressed as A = {0, 1, 2, ..., K} and a t ∈A.

[0075] Furthermore, in the centralized scheduling method, the reward function of the Markov decision process is defined as follows. There are two reward functions to choose from. The first reward function is designed based on the result of data packet transmission. When the data packet sent by the sending node is successfully received by the destination node, r t =1, otherwise r t = 0. The second reward function is designed based on the link quality of the scheduled data flow. If there are data packets in the sending queue of the scheduled data flow, then r t =pro k , otherwise r t = 0. In practical applications, one of the reward functions can be selected based on whether the quality of the link is known.

[0076] Furthermore, the centralized real-time wireless network scheduling method based on deep reinforcement learning is structured as follows: a scheduling strategy based on a Double Deep Q-Network (DDQN) is executed at the access point. Simultaneously, the DDQN is centrally trained by a pre-configured optimizer. The DDQN's inputs are traffic model information of the data stream and environmental status information, and its output is the value of taking different actions (i.e., Q-values), which can be expressed as follows:

[0077] q(env feas,s,a;θ)

[0078] Among them, env feas represents the traffic model information of all data flows, also known as the environmental characteristics of the current environment. Each time slot access point obtains the current state s t The traffic model information env feas of all data streams is used as the input of the DDQN network to obtain the Q value of taking different actions.

[0079] The centralized real-time wireless network scheduling method is structured into two main parts: the training process and the execution process.

[0080] During the training process, the access point interacts with multiple training environments, and different training environments are described by different environmental characteristics. In each time slot, the access point obtains the environmental state s t , then perform action a t , the access point will randomly select an action in the action space to execute or execute the action corresponding to the maximum Q value output by the DDQN network. t After that, the environment will give feedback r t and end t The environment status will be updated to s t+1 . The interaction information between the access point and the environment (s t ,a t ,r t ,s t+1 ,end t ) will be stored in the experience replay pool. t is the termination indicator. The specific location of the experience replay pool can be specified as needed. In the optimal embodiment, it is located at the access point, i.e., the agent. When t≥L*Prd and t%Prd=0, end t =true, otherwise end t =false. L is a constant positive integer that satisfies:

[0081] L*Prd≥max k∈[1,K] (offset k +D k )

[0082] Among them, Prd=Least Common Multiple(prd1,prd2,…,prd K ), that is: Prd is the least common multiple of the periods of all data streams, when end t = true, the access point ends its interaction with the current training environment and begins interacting with another new training environment.

[0083] The access point's experience replay pool is divided into multiple sub-regions, each storing information about the access point's interactions with a specific environment. During each time slot, a certain amount of interaction information is randomly extracted from all experience replay pools. Each interaction information belongs to a specific environment, described by environmental features. These interaction information and environmental features are used together to train and update the neural network's parameters. The amount of information extracted should be neither too large nor too small. If it is too large, memory space will be insufficient, and the neural network may converge to a local optimum. If it is too small, training time will be excessive, and the network parameters will fluctuate significantly between training sessions. In this experiment, the number of interaction information extracted, i.e., batch_size, was set to 64.

[0084] Establish two neural networks with the same structure, namely the current network and the target network. The current network is used for training, and the target network is used to select actions. At a fixed time interval, the parameter θ of the target network is target Will be from the current network parameters θ current in the Copy.

[0085] During the training process, a certain amount of interaction information (batch_size=64) is randomly extracted from the experience replay pool, and the extracted interaction information is used (s j ,a j ,r j ,s′ j ,end j ) represents, where j∈[1,batch_size] represents the sequence number of each interactive information, s′ j Indicates s j Take action a j The next state after that is: the s mentioned above t+1 . Update the parameters of the current network by minimizing a preset loss function:

[0086]

[0087] Among them, y j is the Q value of the target network, when end t = true, y j =Rj ; when end t = false, y j =R j +γQ(s j+1 ,a′;θ tar get ).

[0088] During the test, the access point receives the state of the environment s in each time slot. t The traffic model information of all data flows is used as the input of the target network. The target network outputs the Q value of different actions. The access point selects the action with the largest Q value to execute. After the environment receives the action, it updates to the next state s t+1 .

[0089] The distributed scheduling method includes the following steps:

[0090] (2) The distributed scheduling method is applicable to the following scenarios: there are multiple access points (APs) and multiple user nodes in a single-hop network.

[0091] The characteristics of data flows are the same as those described in the centralized scheduling method. The difference is that in the centralized scheduling method, there is only one access point in the network, that is, one agent. This access point can obtain the status information of all data flows. In the distributed scheduling method, there are multiple access points in the network, that is, multiple agents. Each access point can only obtain partial information about the environment state, that is, the status information of a part of the data flow. The data flow information that each access point can obtain is independent and disjoint.

[0092] In each time slot, an access point selects the transmit queue corresponding to the data stream and sends one of the packets. Because there are multiple access points, each making independent scheduling decisions, collisions can occur if multiple access points simultaneously select the managed data stream to send packets in the same time slot, leading to packet transmission failure. The present invention assumes that in a distributed scheduling method, a complete packet transmission process includes sending RTS, CTS, DATA (i.e., data packet), and ACK packets. The transmission of control packets (RTS, CTS, and ACK) is unaffected by link quality. However, the transmission of DATA (i.e., data packet) is affected by link quality, resulting in packet loss.

[0093] Furthermore, in the distributed scheduling method, the state definition in the partially observable Markov decision process is the same as the state definition in the centralized scheduling method. The difference between the distributed scheduling method and the centralized scheduling method is that in the distributed scheduling method, each access point cannot obtain the state information of all data flow sending queues in the network, but can only obtain the state information of the data flow it manages and the traffic model information of the data flow.

[0094] Furthermore, in the distributed scheduling method, the observation information in the partially observable Markov decision process is defined as follows: In each time slot, the access point can monitor the transmission status of the data packet through wireless shared information, which can be divided into three specific cases:

[0095] (1) No data packets are being transmitted. This may be due to a collision or the data flow managed by the access point is not sending data packets.

[0096] (2) A sending node successfully sends a data packet, but the data packet is lost due to the link quality, resulting in the destination node not receiving it;

[0097] (3) A sending node successfully sends a data packet, which is successfully received by the destination node.

[0098] In the above three cases, the node can distinguish them by the received RTS, CTS, DATA and ACK. The information observed by access point i (assuming that the total number of access points in the distributed scheduling method is I) in time slot t is defined as For case (1), For cases (2) and (3), if the sending node of the data packet is node i, then Otherwise, for case (2), For case (3), Historical Information It is defined as the set of observation information of the sending node i from time slot t-Prd to time slot t-1. That is: historical information It can be expressed as

[0099] Furthermore, in the distributed scheduling method, the action definition of the partially observable Markov decision process is similar to that of the centralized scheduling method. The action of each access point is It schedules one data flow or does not schedule any data flow. For example, access point i wants to manage N i data streams, then the action space A of access point i i The size is N i +1. In each time slot, each access point makes a scheduling decision independently, and the actions of all access points are combined into a joint action. The size of the joint action space is

[0100] Furthermore, a distributed real-time wireless network scheduling method based on deep reinforcement learning uses a centralized training and distributed execution framework. During the centralized training phase, the agent is allowed to access global information. During the execution phase, the agent makes independent scheduling decisions based only on local information. The details are as follows:

[0101] In the distributed scheduling algorithm, an apprentice network is created on each access point, and its own experience replay pool is created. At the same time, an expert network is created in the entire network. During the centralized training phase, the expert network can collect the status information of the sending queues of all data streams and give the optimal scheduling decision action for each access point. Each access point will execute the optimal scheduling decision action given by the expert network during the training phase and use the interactive information composed of local information and the corresponding optimal scheduling decision action to Put them into their respective experience replay pools.

[0102] For each access point, in each time slot, a certain amount of interaction information is randomly extracted from the respective experience replay pool to update the apprentice network parameters on the access point. The same as the setting in the centralized method, the number of extracted interactive information, that is, batch_size is set to 64. For the extracted interactive information, (h j ,s j ,opt_a j ), where j∈[1,batch_size] represents the sequence number of each interactive information, and the parameters of the apprentice network are updated by minimizing a preset loss function:

[0103]

[0104] Among them, p jc is the output of the apprentice network, which means that when the input is h j , s j The probability of taking action c when y jc When action c is the optimal action, y jc =1, otherwise y jc =0.

[0105] According to a second aspect of the present invention, a computer-readable storage medium is provided, in which one or more computer programs are stored. When the computer programs are executed, the computer programs are used to implement the real-time wireless network scheduling method of the present invention.

[0106] According to a third aspect of the present invention, a network coding computing system is provided, comprising a storage device and one or more processors; wherein the storage device is used to store one or more computer programs, and the computer programs, when executed by the processors, are used to implement the real-time wireless network scheduling method of the present invention.

[0107] In order to make the above features and effects of the present invention more clearly understood, embodiments are given below and described in detail with reference to the accompanying drawings.

[0108] The centralized real-time wireless network scheduling method based on deep reinforcement learning of the present invention is implemented as follows: Figure 1 As shown, it mainly includes the following steps:

[0109] Before the sending node interacts with a new environment, it creates a new sub-area in the experience replay pool to store the interaction information between the sending node and the environment.

[0110] When the access point interacts with the environment, in each time slot, the access point converts the queue information of the sending queue corresponding to all data flows. The state of each data flow is composed of the urgency of the data flow to be scheduled and the data packet arrival factor of the data flow. The collection of the states of all data flows constitutes the environment state s of the current time slot. t At the same time, the access point can obtain environmental feature information, namely: traffic model information and link quality information (offset, prd, D, B, pro) of all data flows 1～K , the agent performs scheduling decisions based on the environment state and environment feature information t The agent performs scheduling decisions a t After that, we will receive feedback from the environment, including the reward value r t and the end mark t The status of all data streams will be updated to s t+1 . Interaction information (s t ,a t ,r t ,s t+1 ,end t ) will be stored in the corresponding area of ​​the experience replay pool.

[0111] During each time slot, the access point randomly extracts a certain number of interaction messages from the experience replay pool. Each interaction message belongs to a specific environment, which is described by environmental features. In this experiment, the number of extracted messages is set to 64. The interaction messages and environmental features are used together to train and update the network parameters. The neural network parameters are updated using the dual Q-learning (DDQN) method.

[0112] Specifically, the access point creates two neural networks with identical structures: the current network and the target network. The current network is used to update network parameters, while the target network is used to select actions. Whether in the training or testing phase, when the access point needs to make a scheduling decision, it obtains the current time slot's environmental state and the current environment's environmental characteristics as input to the target network. After receiving the input, the target network outputs the value of taking different actions. The access point then adopts different action selection strategies depending on whether the current phase is training or testing. If the current phase is training, the access point randomly selects an action from the action space with probability ε and selects the scheduling action corresponding to the maximum value in the target network's output with probability 1-ε. ε is a decimal between 0 and 1 that balances exploitation and exploration. Exploitation refers to the agent selecting the action with the highest value when selecting an action, while exploration refers to the agent randomly selecting actions in the action space. Exploitation maximizes the expected return at the current moment in the short term, while exploration maximizes the total return in the long term. In the early stages of agent training, a large ε value is set to allow the agent to fully obtain information about the value of different actions. As the training process progresses, ε is gradually reduced to a small number very close to 0 to ensure that the actions taken by the agent can converge to the action with the greatest value. Therefore, the calculation formula for ε is as follows:

[0113]

[0114] Among them, ε0=0.99, ε min = 0.001, train_time refers to the number of environments that the access point has interacted with. The initial value is 0. When the access point completes each interaction with an environment, the value increases by 1. If the current phase is the test phase, the access point selects the scheduling action corresponding to the maximum value of the output of the target network. The parameter θ of the target network target Every certain period of time, the current network parameter θ current Covered.

[0115] The distributed real-time wireless network scheduling method based on deep reinforcement learning of the present invention is implemented as follows: Figure 2 As shown, it mainly includes the following steps:

[0116] The distributed implementation framework adopts centralized training and distributed execution, as follows.

[0117] In the centralized training phase, there is an expert network that can obtain global information, including: the status information of all data flows, the traffic model information of all data flows, and the link quality information. The expert network makes the optimal scheduling decision based on the global information. Since the expert network can obtain global information, it can be pre-trained in advance using the centralized scheduling method. In each time slot, the expert network outputs the optimal scheduling decision. Transmitted to each access point, each access point performs the received optimal scheduling decision action, and then, the local information that the access point can obtain in the time slot: including historical information and status information of the scheduled data streams and optimal scheduling decision action Combined into interactive information, stored in the experience replay pool, each access point has an apprentice network, the apprentice network consists of two types of neural networks, namely the long time series network LSTM and the fully connected neural network FCN. The LSTM network is used to process historical information The FCN network is used to process the status information of the scheduled data flow In each time slot, the access point randomly extracts a small batch of experiences from the experience replay pool to update the parameters of the apprentice network.

[0118] During the distributed execution phase, the expert network will be removed. When making scheduling decisions, the access point will no longer be guided by the optimal scheduling decision output by the expert network. The apprentice network of each access point outputs the value of each scheduling decision based on local information, and the access point will select the scheduling decision with the greatest value for execution.

[0119] The following is a simulation experiment and explanation of the real-time wireless network scheduling method based on deep reinforcement learning described in the present invention through specific examples.

[0120] Figure 3 Two typical network topologies are shown: a single-hop single access point serving multiple user nodes and a single-hop multiple access points serving multiple user nodes.

[0121] In this experiment, the centralized scheduling method in this invention's example was compared with existing LDF, RAC, and RAC-Approx methods, and the distributed scheduling method was compared with the existing DB_DP method. The real-time throughput achieved by each method was compared with the theoretical optimal value. The adaptability of the different algorithms in dynamic environments was also compared, as was the importance of certain improvements within the algorithms. These experiments were used to evaluate the real-time wireless network scheduling method based on deep reinforcement learning described in this invention.

[0122] Figure 4A functional block diagram of a device for real-time wireless network scheduling based on deep reinforcement learning, according to an embodiment of the present invention, is presented. The device includes: a network state conversion unit configured to convert the current network state into a specific format and transmit it to a neural network; an intelligent scheduling unit configured to select the optimal scheduling strategy based on the network state; and a packet forwarding unit configured to forward packets in the data queue corresponding to the scheduled data flow.

[0123] Figure 5 The real-time throughput of the centralized scheduling method and the existing method under different data flow numbers is shown. The number of data flows K∈[2,10], the traffic model of each data flow (offset k ,prd k ,D k ,B k ,pro k ) is set as follows: When k is an odd number, offset k =k-1, B k =0.8, pro k =0.9, D k =5. When k is an even number, offset k =k-1, B k =0.8, pro k =0.7, D k =4. Two different experiments were set up according to different arrival periods. In one experiment, the arrival period of the data flow will be greater than the deadline of the data flow, and in the other experiment, the arrival period of the data flow will be less than the deadline of the data flow. For the first experiment, if k is an odd number, then prd k =6, if k is an even number, then prd k = 7. For the second set of experiments, if k is an odd number, then prd k =2, if k is an even number, then prd k =3. Figure 5 The results show that the real-time throughput of the centralized scheduling method and the RAC method can both achieve the optimal real-time throughput. However, as the number of data streams increases, the number of states increases exponentially. RAC cannot provide a scheduling strategy when the number of data streams is 6 and 8, while the centralized scheduling method can still provide a scheduling strategy. At the same time, the real-time throughput of the centralized scheduling method under different numbers of data streams is also higher than that of the heuristic methods RAC-Approx method and LDF method.

[0124] Figure 6 The real-time throughput of the centralized scheduling method and the existing methods under different deadlines is shown. The number of data flows K = 4, the deadline range is D∈[2,6], and the traffic model offset of each data flow is k ,prdk ,D k ,B k ,pro k ) is set as follows: When k is an odd number, offset k =k-1, B k =0.8, pro k =0.9, D k =D+1. When k is an even number, offset k =k-1, B k =0.8, pro k =0.7, D k =D. Two different experiments were set up according to different arrival periods. In one experiment, the arrival period of the data flow will be greater than the deadline of the data flow, and in the other experiment, the arrival period of the data flow will be less than the deadline of the data flow. For the first experiment, if k is an odd number, then prd k =6, if k is an even number, then prd k = 7. For the second set of experiments, if k is an odd number, then prd k =2, if k is an even number, then prd k =3. Figure 5 The results show that the real-time throughput of the centralized scheduling method and the RAC method can both achieve the optimal real-time throughput. At the same time, the real-time throughput of the centralized scheduling method under different numbers of data flows is also higher than that of the heuristic methods RAC-Approx method and LDF method.

[0125] Figure 7 The real-time throughput of the distributed scheduling method and the existing methods under different numbers of data flows is shown. The experimental setting is the same as Figure 5 Same as in, however, B of each data stream k =1. Figure 7 The results show that the real-time throughput of the distributed scheduling method can achieve the optimal real-time throughput. At the same time, the real-time throughput of the distributed scheduling method under different numbers of data streams is also higher than that of the DB-DP method.

[0126] Figure 8 The real-time throughput of the distributed scheduling method and existing methods under different deadlines is shown. The experimental setup is similar to Figure 6 Same as in, however, B of each data stream k =1. Figure 8 The results show that the real-time throughput of the distributed scheduling method can achieve the optimal real-time throughput. At the same time, the real-time throughput of the distributed scheduling method under different deadlines is also higher than that of the DB-DP method.

[0127] Figure 9This paper demonstrates the role of state transitions in centralized scheduling and the impact of different reward function definitions. It compares two state definitions: a binary string state definition (Type 1) and the state transition definition of the present invention (Type 2). It also compares two reward functions: a reward function based on packet delivery results (Type 1) and a reward function based on the link quality of the scheduled link (Type 2). Figure 9 It shows that when the same reward function is used, the method cannot converge to the optimal real-time throughput when the state definition of type 1 is used, but it can converge to the optimal real-time throughput when the state definition of type 2 is used. This shows that the state representation proposed in this invention is better than the existing state representation method. At the same time, according to Figure 9 ,When using the same state definition, using different reward ,functions will not affect the real-time throughput after the method converges, but it will affect the ,convergence time of the method. Using type 1 reward ,function design will prolong the convergence time of the method. In ,practical cases, if the sending node can know the link quality, type 2 reward ,function can be used, otherwise type 1 reward ,function can be used.

[0128] Figure 10 The importance of expert networks in distributed scheduling methods is demonstrated, and the changes in real-time throughput of distributed scheduling methods under two conditions during training are compared. Figure 10 It can be seen that the use of the expert network allows the distributed scheduling method's real-time throughput to gradually increase to the optimal real-time throughput as the number of training times increases. However, without the expert network, each sending node's apprentice network is trained independently, which causes the distributed scheduling method's real-time throughput to fluctuate unsteadily with increasing training times, eventually reaching zero.

[0129] Figure 11 and Figure 12 The real-time throughput changes of centralized scheduling method and distributed scheduling method in dynamic wireless network environment are demonstrated. Figure 11 and Figure 12 There are two different network environments in two different stages. Figure 11 It shows that the centralized scheduling method can adjust the scheduling strategy in time when the network environment changes, so that the real-time throughput can be restored to the optimal real-time throughput in the new environment. Although other methods LDF and RAC-Approx can schedule the scheduling strategy in a shorter time, they cannot achieve the optimal real-time throughput. The RAC method needs more time to calculate the optimal scheduling strategy in the new environment after the environment changes.

[0130] Therefore, for a period of time, the real-time throughput of the RAC method is 0. Figure 12It shows that although the distributed scheduling method cannot quickly recover to the optimal real-time throughput after the environment changes, it can continue training on the original network parameters so that the real-time throughput gradually reaches the optimal real-time throughput. Although the DB-DP method can quickly adjust the scheduling strategy, it cannot achieve the optimal real-time throughput.

[0131] The following is a system embodiment corresponding to the above method embodiment. This embodiment can be implemented in conjunction with the above embodiment. The relevant technical details mentioned in the above embodiment are still valid in this embodiment and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above embodiment.

[0132] The present invention also proposes a real-time centralized wireless network scheduling system based on deep reinforcement learning, which includes:

[0133] A preparation module is used to obtain a wireless network consisting of an access point and multiple interconnected user nodes. The access point schedules all data flows in the wireless network. Before interacting with the network environment, the access point in the wireless network creates a new sub-area in the experience replay pool;

[0134] The scheduling decision module is used to obtain the status of each data flow according to the sending queue information corresponding to all data flows in each time slot when the access point interacts with the network environment, and the status of all data flows is combined to form the environment state s of the current t-th time slot. t At the same time, the access point obtains the traffic model and link quality of all data flows as environmental feature information, inputs the environmental state and the environmental feature information into the decision model, and the access point executes the scheduling decision a corresponding to the output result of the decision model. t ;

[0135] Storage module, used for the access point to perform scheduling decision a t After receiving feedback from the network environment t and the end mark end t , the status of all data streams is updated to s t+1 ; The interactive information (s t ,a t ,r t ,s t+1 ,end t ) and the environmental state and the environmental feature information as experience, and store them in the sub-area;

[0136] The update module is used to extract experience from the experience replay pool to train and update the current decision model.

[0137] In the real-time centralized wireless network scheduling system based on deep reinforcement learning, the training process of the decision model includes:

[0138] Create two neural networks with the same structure, namely the current network and the target network. When the access point needs to make a scheduling decision, it inputs the environmental state and environmental characteristics of the current time slot t into the target network to obtain the value of each scheduling action. The access point randomly selects an action to perform scheduling with a probability of ε, and selects the action with the highest value to perform scheduling with a probability of 1-ε, where ε is a decimal between 0 and 1.

[0139] When the access point performs action a t After that, get feedback r t and the end mark end t , the environment state is updated to s t+1 ; The interactive information (s t ,a t ,r t ,s t+1 ,end t ) is stored in the experience replay pool, where end t Is the termination indicator, when t≥L*Prd and t%Prd=0, end t =true, otherwise end t =false, L is a constant positive integer that satisfies:

[0140] L*Prd≥max k∈[1,K] (offset k +D k )

[0141] Among them, Prd is the least common multiple of all data flow periods, D k Indicates the deadline of the data packet in data stream k, offset k Indicates the offset between the arrival time of the first data packet of data stream k and the network start time. t = true, the access point ends its interaction with the current training environment and starts interacting with another training environment;

[0142] After randomly extracting some interactive information from the experience replay pool, the parameters of the current network are updated by minimizing the loss function. The parameters of the target network θ target Every scheduled time, the parameters θ of the current network will be current After reaching the preset training time, the current target network is saved as the decision model.

[0143] In the real-time centralized wireless network scheduling system based on deep reinforcement learning, the calculation system of ε is as follows:

[0144]

[0145] Among them, ε0=0.99, ε min =0.001, train_time refers to the number of environments that the access point has interacted with. The initial value is 0. When the access point finishes interacting with an environment, the value increases by 1.

[0146] In the real-time centralized wireless network scheduling system based on deep reinforcement learning, the state of the data flow is composed of the urgency of the data flow to be scheduled and the data packet arrival factor of the data flow.

[0147] The present invention proposes a computer-readable storage medium for storing one or more computer programs, which, when executed, are used to implement any of the real-time centralized wireless network scheduling methods based on deep reinforcement learning.

[0148] The present invention proposes a client for use in any of the above-mentioned real-time centralized wireless network scheduling systems based on deep reinforcement learning.

[0149] It should be noted that not all steps described in the above embodiments are necessary, and those skilled in the art may make appropriate selections, replacements, modifications, etc. according to actual needs.

[0150] Finally, it should be noted that the above embodiments are intended only to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the embodiments above, it should be understood by those skilled in the art that modifications or equivalent substitutions to the technical solutions of the present invention do not depart from the spirit and scope of the technical solutions of the present invention and are intended to be encompassed by the claims of the present invention.

Claims

1. A real-time centralized wireless network scheduling method based on deep reinforcement learning, characterized in that: include: Step 1: Obtain a wireless network consisting of an access point and multiple interconnected user nodes. The access point schedules all data flows in the wireless network. Before interacting with the network environment, the access point in the wireless network creates a new sub-area in the experience replay pool. Step 2: When the access point interacts with the network environment, in each time slot, the access point obtains the state of each data stream based on the sending queue information corresponding to all data streams, and the state of all data streams is combined to form the environment state s of the current t-th time slot. t At the same time, the access point obtains the traffic model and link quality of all data flows as environmental feature information, inputs the environmental state and the environmental feature information into the decision model, and the access point executes the scheduling decision a corresponding to the output result of the decision model. t ; Step 3: The access point executes scheduling decision a t After receiving feedback from the network environment t and the end mark end t , the status of all data streams is updated to s t+1 ; The interactive information (s t ,a t ,r t ,s t+1 ,end t ) and the environmental state and the environmental feature information as experience, and store them in the sub-area; Step 4: Extract experience from the experience replay pool to train and update the current decision model; The state of the data flow is composed of the urgency of the data flow to be scheduled and the data packet arrival factor of the data flow; The training process of the decision model includes: Create two neural networks with the same structure, namely the current network and the target network. When the access point needs to make a scheduling decision, it inputs the environmental state and environmental characteristics of the current time slot t into the target network to obtain the value of each scheduling action. The access point randomly selects an action to perform scheduling with a probability of ε, and selects the action with the highest value to perform scheduling with a probability of 1-ε, where ε is a decimal between 0 and 1. When the access point performs action a t After that, get feedback r t and the end mark end t , the environment state is updated to s t+1 ; The interactive information (s t ,a t ,r t ,s t+1 ,end t ) is stored in the experience replay pool, where end t Is the termination indicator, when t≥L*Prd and t%Prd=0, end t =true, otherwise end t =false, L is a constant positive integer that satisfies: L*Prd≥max k∈[1,K] (offset k +D k ) Among them, Prd is the least common multiple of all data flow periods, D k Indicates the deadline of the data packet in data stream k, offset k Indicates the offset between the arrival time of the first data packet of data stream k and the network start time. t = true, the access point ends its interaction with the current training environment and starts interacting with another training environment; After randomly extracting some interactive information from the experience replay pool, the parameters of the current network are updated by minimizing the loss function. The parameters of the target network θ target Every scheduled time, the parameters θ of the current network will be current After reaching the preset training time, the current target network is saved as the decision model; ε is calculated as follows: Among them, ε0=0.99, ε min =0.001, train_time refers to the number of environments that the access point has interacted with. The initial value is 0. When the access point finishes interacting with an environment, the value increases by 1.

2. A real-time centralized wireless network scheduling system based on deep reinforcement learning, characterized in that: include: A preparation module is used to obtain a wireless network consisting of an access point and multiple interconnected user nodes. The access point schedules all data flows in the wireless network. Before interacting with the network environment, the access point in the wireless network creates a new sub-area in the experience replay pool; The scheduling decision module is used to obtain the status of each data flow according to the sending queue information corresponding to all data flows in each time slot when the access point interacts with the network environment, and the status of all data flows is combined to form the environment state s of the current t-th time slot. t At the same time, the access point obtains the traffic model and link quality of all data flows as environmental feature information, inputs the environmental state and the environmental feature information into the decision model, and the access point executes the scheduling decision a corresponding to the output result of the decision model. t ; Storage module, used for the access point to perform scheduling decision a t After receiving feedback from the network environment t and the end mark end t , the status of all data streams is updated to s t+1 ; The interactive information (s t ,a t ,r t ,s t+1 ,end t ) and the environmental state and the environmental feature information as experience, and store them in the sub-area; An update module is used to extract experience from the experience replay pool to train and update the current decision model; The training process of the decision model includes: Create two neural networks with the same structure, namely the current network and the target network. When the access point needs to make a scheduling decision, it inputs the environmental state and environmental characteristics of the current time slot t into the target network to obtain the value of each scheduling action. The access point randomly selects an action to perform scheduling with a probability of ε, and selects the action with the highest value to perform scheduling with a probability of 1-ε, where ε is a decimal between 0 and 1. When the access point performs action a t After that, get feedback r t and the end mark end t , the environment state is updated to s t+1 ; The interactive information (s t ,a t ,r t ,s t+1 ,end t ) is stored in the experience replay pool, where end t Is the termination indicator, when t≥L*Prd and t%Prd=0, end t =true, otherwise end t =false, L is a constant positive integer that satisfies: L*Prd≥max k∈[1,K] (offset k +D k ) Among them, Prd is the least common multiple of all data flow periods, D k Indicates the deadline of the data packet in data stream k, offset k Indicates the offset between the arrival time of the first data packet of data stream k and the network start time. t = true, the access point ends its interaction with the current training environment and starts interacting with another training environment; After randomly extracting some interactive information from the experience replay pool, the parameters of the current network are updated by minimizing the loss function. The parameters of the target network θ target Every scheduled time, the parameters θ of the current network will be current After reaching the preset training time, the current target network is saved as the decision model; The calculation system of ε is as follows: Among them, ε0=0.99, ε min =0.001, train_time refers to the number of environments that the access point has interacted with. The initial value is 0. When the access point finishes interacting with an environment, the value increases by 1; The state of the data flow is composed of the urgency of the data flow to be scheduled and the data packet arrival factor of the data flow.

3. A computer-readable storage medium for storing one or more computer programs, which, when executed, are used to implement the real-time centralized wireless network scheduling method based on deep reinforcement learning as described in claim 1.

4. A client for the real-time centralized wireless network scheduling system based on deep reinforcement learning as claimed in claim 2.

Images