Real-time distributed wireless network scheduling method and system based on deep reinforcement learning

Abstract

The present invention proposes a real-time distributed wireless network scheduling method and system based on deep reinforcement learning, including: obtaining a wireless network composed of multiple access points and user nodes; in each time slot of the test phase, the access point obtains the state of each data stream according to the sending queue information corresponding to the part of the data stream, and simultaneously obtains the historical observation information of the wireless channel before, inputs the state of the data stream and the historical observation information into the decision model, and the access point executes the scheduling decision corresponding to the output result of the decision model; in the training phase, the global decision model existing in the network obtains the state information of all data streams, and gives the optimal decision action of each access point, the access point executes the optimal decision action given by the expert network, and simultaneously stores the state of the part of the data stream, the historical observation information of the channel, and the optimal scheduling decision action as interactive information in their respective experience replay pools, extracts experience from their respective experience replay pools, and trains and updates the current decision model.

Description

Technical Field

[0001] The present invention belongs to the technical field of routing network information scheduling in information technology, and particularly relates to a real-time distributed wireless network scheduling method and system based on deep reinforcement learning. Background Art

[0002] In recent years, real-time wireless networks have been increasingly used in various applications that require delay constraints. Typical applications of this type include cyber-physical systems (CPSs) used in industrial IoT and networked transportation systems, or multimedia communication systems for live video transmission. These systems usually require that data packets be received by the destination node within a deadline, otherwise the data packets will be removed from the sending queue. An important indicator for measuring real-time wireless networks is real-time throughput, which is defined as the ratio of the number of data packets successfully received by the destination node to the sum of the number of data packets generated in the network.

[0003] In recent years, the problem of how to schedule real-time networks subject to latency constraints to achieve higher real-time throughput has attracted widespread attention. The scheduling strategies that have been proposed so far only consider the frame-synchronous traffic model, in which the packet deadline and arrival period of the data flow in the network are equal. The frame-synchronous traffic model often fails to model many important practical scenarios. For example, for a typical mobile video conferencing scenario, the latency deadline is usually much shorter than the frame generation period. In this scenario, the scheduling methods mentioned above cannot meet the latency deadline of this real-time data flow.

[0004] Research on scheduling algorithms for general traffic models often fails to achieve good results in real-world scenarios due to high computational complexity or inability to achieve optimal real-time throughput. In addition, traditional methods, such as greedy algorithms and dynamic programming, cannot automatically adapt to dynamic network environments. Therefore, existing research work cannot provide efficient and adaptive real-time network scheduling strategies. Summary of the invention

[0005] Specifically, this application proposes a real-time distributed wireless network scheduling method based on deep reinforcement learning, which includes:

[0006] Step 1: obtain a wireless network consisting of multiple access points and multiple user nodes interconnected, each access point schedules part of the data flow in the network, and each access point has an experience playback pool locally;

[0007] Step 2: When each access point i interacts with the network environment, in each time slot t during the training phase, the global expert network in the network collects the status information, traffic model information, and link quality information of all data flows, and gives the optimal scheduling decision action for each access point. Each access point i executes the optimal scheduling decision action given by the global expert network. Each access point i obtains the state of the data stream scheduled by access point i in time slot t according to the sending queue information corresponding to some data streams At the same time, obtain the historical observation information of the wireless channel before the cutoff time slot t Interactive Information Stored in the local experience replay pool of access point i, at each time slot t in the test phase, each access point i will replay the state of part of the data flow and historical observation information Input to the decision model, the access point executes the scheduling decision corresponding to the output of the decision model

[0008] Step 3: Each access point i extracts experience from the local experience replay pool to train and update the decision model of access point i.

[0009] The real-time distributed wireless network scheduling method based on deep reinforcement learning, wherein the training process of the decision model includes:

[0010] An expert network for acquiring global information is constructed. The global information includes the status information, traffic model information and link quality information of all data flows in the wireless network. In each time slot, the expert network makes the optimal scheduling decision for each access point based on the global information. Each access point executes the received optimal scheduling decision action. At the same time, the access point stores the historical observation information, local environmental status and optimal scheduling decision action obtained in the current time slot into the local experience replay pool of the access point. Each access point is equipped with an apprentice network. In each time slot, the access point randomly extracts batches of experience from the local experience replay pool to update the parameters of the apprentice network until the scheduling decision output by the apprentice network reaches the preset standard or reaches the preset number of updates, and the current apprentice network is saved as the decision model.

[0011] The real-time distributed wireless network scheduling method based on deep reinforcement learning, wherein the apprentice network includes a long time series network and a fully connected neural network, the long time series network is used to process historical observation information, and the fully connected neural network is used to process the state of the scheduled data stream;

[0012] The historical information is the data packet transmission status monitored by the access point in each historical time slot. The data packet transmission status includes the following three types:

[0013] No data packets are transmitted, the sending node successfully sends a data packet but the data packet is not received by the destination node, and the sending node successfully sends a data packet and is successfully received by the destination node.

[0014] The real-time distributed wireless network scheduling method based on deep reinforcement learning, wherein the loss function is based on the following Update the apprentice network parameters of the i-th access point:

[0015]

[0016] Among them, p jc is the output of the apprentice network, indicating that when the input is h j ,s j The probability of taking scheduling action c when j ,s j The distribution is the jth historical information and environmental state in the batch experience; y jc When action c is the optimal action, y jc =1, otherwise y jc =0.

[0017] The present invention also proposes a real-time distributed wireless network scheduling system based on deep reinforcement learning, which includes:

[0018] An initial module is used to obtain a wireless network consisting of multiple access points and multiple user nodes interconnected, each access point schedules part of the data flow in the network, and each access point has an experience playback pool locally;

[0019] Scheduling module, used when each access point i interacts with the network environment, in each time slot t of the training phase, the global expert network in the network collects the status information, traffic model information and link quality information of all data flows, and gives the optimal scheduling decision action for each access point, and each access point i executes the optimal scheduling decision action given by the global expert network Each access point i obtains the state of the data stream scheduled by access point i in time slot t according to the sending queue information corresponding to some data streams At the same time, obtain the historical observation information of the wireless channel before the cutoff time slot t Interactive Information Stored in the local experience replay pool of access point i, at each time slot t in the test phase, each access point i will replay the state of part of the data flow and historical observation information Input to the decision model, the access point executes the scheduling decision corresponding to the output of the decision model

[0020] The training module is used to enable each access point i to extract experience from a local experience replay pool to train and update the decision model of the access point i.

[0021] The real-time distributed wireless network scheduling system based on deep reinforcement learning, wherein the training of the decision model includes:

[0022] An expert network for acquiring global information is constructed. The global information includes the status information, traffic model information and link quality information of all data flows in the wireless network. In each time slot, the expert network makes the optimal scheduling decision for each access point based on the global information. Each access point executes the received optimal scheduling decision action. At the same time, the access point stores the historical observation information, local environmental status and optimal scheduling decision action obtained in the current time slot into the local experience replay pool of the access point. Each access point is equipped with an apprentice network. In each time slot, the access point randomly extracts batches of experience from the local experience replay pool to update the parameters of the apprentice network until the scheduling decision output by the apprentice network reaches the preset standard or reaches the preset number of updates, and the current apprentice network is saved as the decision model.

[0023] The real-time distributed wireless network scheduling method based on deep reinforcement learning, wherein the apprentice network includes a long time series network and a fully connected neural network, the long time series network is used to process historical observation information, and the fully connected neural network is used to process the state of the scheduled data stream;

[0024] The historical information is the data packet transmission status monitored by the access point in each historical time slot. The data packet transmission status includes the following three types:

[0025] No data packets are transmitted, the sending node successfully sends a data packet but the data packet is not received by the destination node, and the sending node successfully sends a data packet and is successfully received by the destination node.

[0026] The real-time distributed wireless network scheduling method based on deep reinforcement learning, wherein the loss function is based on the following Update the apprentice network parameters of the i-th access point:

[0027]

[0028] Among them, p jc is the output of the apprentice network, indicating that when the input is h j ,s j The probability of taking scheduling action c when j ,s j are the historical information and environmental state of the jth experience in the batch experience respectively; y jc When action c is the optimal action, y cj =1, otherwise ycj =0.

[0029] The present invention also proposes a computer-readable storage medium for storing one or more computer programs, which, when executed, are used to implement any of the real-time distributed wireless network scheduling methods based on deep reinforcement learning.

[0030] The present invention also proposes a client for use in any of the real-time distributed wireless network scheduling systems based on deep reinforcement learning.

[0031] Compared with the prior art, the present invention has the following advantages:

[0032] 1. The present invention proposes a centralized scheduling method and a distributed scheduling method based on deep reinforcement learning. Existing methods such as the RAC method have high complexity, and the time cost of solving the optimal problem to obtain the optimal scheduling strategy increases exponentially with the increase in the number of data streams. The training time of the method proposed in the present invention will not increase rapidly with the increase in the number of data streams, and can quickly converge to the optimal real-time throughput.

[0033] 2. In order to adapt to the dynamic changes of wireless real-time networks, the present invention modifies the Q function in deep reinforcement learning so that it is not only related to the state and action, but also to the environmental characteristics. Compared with the prior art, the present invention can better adapt to new environments and has higher generalization capabilities in dynamic real-time wireless network environments. The strength of the generalization capability determines whether the method proposed in the present invention can be extended to networks with different traffic models and link qualities.

[0034] 3. The distributed scheduling method of the present invention adopts a centralized training and distributed execution framework. The distributed real-time wireless network scheduling method based on deep reinforcement learning is executed in a distributed manner on each sending node, and during training, the intelligent agent allows the use of global information to simplify the training process.

[0035] 4. The present invention transforms the state of each flow through an exponential function, so that the state can intuitively reflect the urgency of scheduling data flow k in numerical terms. Compared with the binary string representation method proposed by the existing method, the state representation method stated in the present invention can make the deep reinforcement learning method converge faster and achieve the optimal real-time throughput. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is the implementation framework of the centralized real-time wireless network scheduling method based on deep reinforcement learning of the present invention.

[0037] Figure 2 It is the implementation framework of the distributed real-time wireless network scheduling method based on deep reinforcement learning of the present invention.

[0038] Figure 3 The following are network topology diagrams of two classic scenarios to which the present invention is applicable.

[0039] Figure 4 is a functional configuration block diagram of a device for real-time wireless network scheduling for deep reinforcement learning according to an embodiment of the present invention;

[0040] Figures 5 to 12 It is a simulation experiment result diagram of an example of the present invention. DETAILED DESCRIPTION

[0041] The present invention relates to a real-time wireless network scheduling method and device based on deep reinforcement learning, aiming to solve the technical problem that data flow cannot provide an adaptive dynamic scheduling strategy under dynamic network conditions and general traffic models. The method includes a centralized method and a distributed method.

[0042] The centralized method includes: a single-hop network has an access point (AP) and multiple user nodes. The access point can obtain global information in each time slot and decide which user node to serve. Design a general traffic model representation method, establish a centralized real-time wireless network scheduling method framework based on deep reinforcement learning, and design the content of each module interaction; design a centralized real-time wireless network scheduling method structure based on deep reinforcement learning.

[0043] The distributed method includes: there are multiple access points (APs) and multiple user nodes in a single-hop network, each access point is independently responsible for serving multiple different user nodes, each access point can obtain local information at the beginning of each time slot, and combine the information observed by the wireless channel to decide which user node to serve. When there are multiple access points to serve user nodes in the same time slot, transmission conflicts will occur in the network, resulting in transmission failure. A distributed real-time wireless network scheduling method framework based on deep reinforcement learning is established, and a distributed real-time wireless network scheduling method structure based on deep reinforcement learning is designed. The present invention has good model generalization capabilities, and can quickly converge to the optimal real-time throughput, and can achieve better performance in dynamic network environments and general traffic model scenarios.

[0044] In view of the shortcomings of the prior art, according to the first aspect of the present invention, a real-time wireless network scheduling method based on deep reinforcement learning is proposed to provide an efficient adaptive dynamic scheduling strategy under dynamic network conditions and general traffic models. The technical solution adopted by the present invention is divided into two parts: a centralized scheduling method and a distributed scheduling method.

[0045] (1) A centralized scheduling method includes the following steps:

[0046] The centralized scheduling method is applicable to the following scenario: there is one access point (AP) and multiple user nodes in a single-hop network.

[0047] There are multiple data flows in the network, each of which belongs to a common traffic model. Each data flow has a sending queue. Data packets arrive at the sending queue in sequence according to the traffic model of the data flow, waiting to be sent by the sending node. When the data packet is successfully received by the destination node or the data packet exceeds the deadline but has not been successfully received by the destination node, the data packet will be removed from the sending queue. If the data flow is on the AP, the AP acts as the sending queue, and if the data flow is on the user node, the user node acts as the sending node.

[0048] The data packets in the data stream can be transmitted from the user node to the access point or from the access point to the user node. When the data stream is located on the user node, the user node acts as a sending node, the access point acts as a receiving node (i.e., the destination node), and an uplink is formed between the user node and the access point. When the data stream is located on the access point, the user node acts as a receiving node, and the access point acts as a sending node (i.e., the destination node). A downlink is formed between the user node and the access point. The uplink and the downlink are collectively referred to as a link. When a data packet is transmitted on a link, it will be affected by the link quality and packet loss will occur. After the sending node sends a data packet in each time slot, the receiving node will send ACK information to the sending node to inform whether the data packet is successfully received. In the present invention, the transmission of ACK is not considered to be affected by the link quality.

[0049] In each time slot, the access point can obtain the global information of the environment, that is, the information of each data packet in the sending queue of each data stream and the traffic model information of each data stream. The access point decides which data stream to schedule based on the global information, and then sends the data packets in the sending queue corresponding to the data stream. Since a link is formed between the access point and the user node, scheduling a data stream is equivalent to scheduling a link. Assuming that there are K data streams in the network, each data stream belongs to a general traffic model, that is, data packets arrive in the sending queue periodically, and the data packets in each data stream have a unified deadline. The general traffic model of the data stream can be represented by the following vector:

[0050] (offset k ,ped k , D k , B k , pro k ), kεK

[0051] Its offset k prd represents the offset between the arrival time of the first data packet of data stream k and the network start time; k represents the arrival period of the data packet in the kth data stream; Dk represents the deadline of the data packet in data stream k; B k ∈[1, 0] represents the arrival probability of data packets in data stream k. Data packets generated by the data stream of the application layer will be affected by the arrival probability when entering the sending queue and thus cannot enter the sending queue; k Indicates the link quality of link k. When data packets are transmitted on the link, they will be affected by wireless channel interference and noise, resulting in packet loss. Then the destination node will not be able to successfully receive the data packet.

[0052] Establish a real-time wireless network scheduling method framework based on deep reinforcement learning, including two modules: the agent and the environment of deep reinforcement learning, and design the content of the interaction between each module;

[0053] Furthermore, the two modules of deep reinforcement learning, agent and environment, are designed as follows:

[0054] (1) Access points in real-time wireless networks are considered as intelligent agents for deep reinforcement learning;

[0055] (2) The abstract environment is a real-time wireless network, including all data flow information in the network and the status information of the corresponding data packets in the sending queue;

[0056] (3) For the centralized scheduling method, there is only one access point in the network, that is, one intelligent agent. The access point obtains global information at each moment, that is, the state of the environment, and performs actions. The environment receives the action and feeds back a reward value to the intelligent agent, thereby realizing the interaction between the access point and the environment. The interaction process between the access point and the environment is modeled as a Markov decision process.

[0057] (4) For the distributed scheduling method, there are multiple access points in the network at the same time, that is, multiple intelligent agents. Each access point obtains local information at each moment, that is, part of the environment state, and performs actions. After receiving the action, the environment will feedback the reward value, thereby realizing the interaction between the access point and the environment. The interaction process between the access point and the environment is modeled as a partially observable Markov decision process.

[0058] Furthermore, in the centralized scheduling method, the state of the Markov decision process is defined as follows: the state of the data flow at time slot t is represented by a two-dimensional vector (q t (k),l t (k)) represents the state s of the environment at time slot t t is the set of states of all data in time slot t, that is: s t =((q t (1),l t (1)),...,(q t (k),l t(k)),...,(q t (K),l t (K))) indicates.

[0059] Among them, l t (k) represents the packet arrival factor of data stream k at time slot t. If data stream k may have new packets arriving at the sending queue in the next time slot t+1, then l t (k)=1, otherwise l t (k) = 0. t (k) can be calculated from the traffic model of the data flow. That is:

[0060]

[0061]

[0062] q t (k) represents the urgency of data flow k to be scheduled at time slot t:

[0063] Where G is the number of unexpired packets in the send queue of data stream k at time slot t. g is the arrival order of the packets in G data. The arrival time and expiration time of the gth packet in data stream k can be expressed as and The expiration time of the data packet can be calculated according to the traffic model of data flow k by the following formula:

[0064]

[0065] The remaining time before the data packet expires It can be expressed as:

[0066]

[0067] Furthermore, in the centralized scheduling method, the actions in the Markov decision process are defined as follows: the action a of the access point in time slot t is t It selects a certain sending queue and sends the data packet with the shortest expiration time in the queue. t is the number of the scheduled data stream. The action space of the access point can be expressed as A = {0, 1, 2, ..., K} and a t ∈A.

[0068] Furthermore, in the centralized scheduling method, the reward function of the Markov decision process is defined as follows. There are two reward functions to choose from. The first reward function is designed based on the result of data packet transmission. When the data packet sent by the sending node is successfully received by the destination node, r t =1, otherwise r t= 0. The second reward function is designed based on the link quality of the scheduled data flow. If there are data packets in the sending queue of the scheduled data flow, then r t =pro k , otherwise r t = 0. In practical applications, one of the reward functions can be selected based on whether the quality of the link is known.

[0069] Furthermore, the structure of the centralized real-time wireless network scheduling method based on deep reinforcement learning is as follows: the scheduling strategy based on the double deep Q network DDQN (Double Deep Q-Network) is executed at the access point, and at the same time, DDQN is centrally trained by a pre-set optimizer. The input of DDQN is the traffic model information of the data flow and the environmental state information, and the output is the value of taking different actions (i.e., Q value), which can be expressed as follows:

[0070] q(env feas,s,a;θ)

[0071] Among them, env feas represents the traffic model information of all data flows, also known as the environmental characteristics of the current environment. Each time slot access point obtains the current state s t The traffic model information env feas of all data flows is used as the input of the DDQN network to obtain the Q values ​​of different actions.

[0072] The centralized real-time wireless network scheduling method is structured into two main parts: the training process and the execution process.

[0073] During the training process, the access point interacts with multiple training environments, and different training environments are described by different environmental characteristics. In each time slot, the access point obtains the environmental state s t , then perform action a t , the access point will randomly select an action in the action space to execute or execute the action corresponding to the maximum Q value output by the DDQN network. t After that, the environment will give feedback r t and end t The environment status will be updated to s t+1 . The interactive information between the access point and the environment (s t ,a t ,r t ,s t+1 ,end t ) will be stored in the experience replay pool. t is the termination indicator. The specific location of the experience replay pool can be specified as needed. In the optimal embodiment, it is located at the access point, i.e., the agent. When t≥L*Prd and t%Prd=0, endt =true, otherwise end t =false. L is a constant positive integer that satisfies:

[0074] L*Prd≥max k∈p1,K] (offset k +D k )

[0075] Where Prd = Least Common Multiple (prd 1 ,prd 2 ,…,prd K ), that is: Prd is the least common multiple of the periods of all data streams, when end t = true, the access point ends its interaction with the current training environment and starts interacting with another new training environment.

[0076] The experience replay pool of the access point will be divided into multiple sub-areas, each of which stores the interaction information between the access point and a specific environment. In each time slot, a certain amount of interaction information will be randomly extracted from all experience replay pools. Each interaction information belongs to a specific environment, which is described by environmental features. The interaction information and environmental features are used together to train and update the parameters of the neural network. The amount of information extracted should not be too large or too small. If it is too large, it will lead to insufficient memory space and the neural network may converge to a local optimum. If it is too small, the training time will be too long and the network parameters between two adjacent trainings will have a large oscillation amplitude. In this experiment, the number of interaction information extracted, that is, batch_size, is set to 64.

[0077] Two neural networks with the same structure are established, namely the current network and the target network. The current network is used for training, and the target network is used to select actions. At fixed intervals, the parameter θ of the target network is target The current network parameters θ current in the Copy.

[0078] During the training process, a certain amount of interactive information (batch_size=64) is randomly extracted from the experience replay pool, and the extracted interactive information is used (s j ,a j ,r j ,s′ j ,end j ), where j∈[1,batch_size] represents the sequence number of each interactive information, s′ j Indicates j Take action a j The next state after that is: the s mentioned abovet+1 . Update the parameters of the current network by minimizing a preset loss function:

[0079]

[0080] Among them, y j is the Q value of the target network, when end t = true, y j =R j ; when end t = false, y j =R j +γQ(s j+1 ,a′;θ target ).

[0081] During the test, the access point receives the state of the environment s in each time slot. t and the traffic model information of all data flows as the input of the target network. The target network outputs the Q values ​​of different actions. The access point selects the action with the largest Q value to execute. After the environment receives the action, it updates to the next state s t+1 .

[0082] The distributed scheduling method comprises the following steps:

[0083] (2) The distributed scheduling method is applicable to the following scenarios: there are multiple access points (APs) and multiple user nodes in a single-hop network.

[0084] The characteristics of data flow are the same as those described in the centralized scheduling method. The difference is that in the centralized scheduling method, there is only one access point in the network, that is, one intelligent agent. This access point can obtain the status information of all data flows, while in the distributed scheduling method, there are multiple access points in the network, that is, multiple intelligent agents, and each access point can only obtain partial information about the environment state, that is, obtain the status information of part of the data flow, and the data flow information that each access point can obtain does not intersect with each other and is independent of each other.

[0085] The access point selects the sending queue corresponding to the data stream in each time slot and sends one of the data packets. Since there are multiple access points, and each access point makes scheduling decisions independently, if multiple access points simultaneously select the managed data stream to send data packets in the same time slot, a collision will occur, resulting in data packet transmission failure. The present invention assumes that in the distributed scheduling method, a complete data packet transmission process includes sending RTS, CTS, DATA (i.e., data packet) and ACK packet, and the transmission of control data packets RTS, CTS, ACK is not affected by link quality. The transmission of DATA, i.e., data packet will be affected by link quality, resulting in packet loss.

[0086] Furthermore, in the distributed scheduling method, the state definition in the partially observable Markov decision process is the same as the state definition in the centralized scheduling method. The difference between the distributed scheduling method and the centralized scheduling method is that in the distributed scheduling method, each access point cannot obtain the state information of all data flow sending queues in the network, but can only obtain the state information of the data flow managed by itself and the traffic model information of the data flow.

[0087] Furthermore, in the distributed scheduling method, the observation information in the partially observable Markov decision process is defined as follows: In each time slot, the access point can monitor the transmission of the data packet through wireless sharing information, which can be divided into three cases:

[0088] (1) No data packets are transmitted. In this case, a conflict may occur or no data stream managed by the access point sends data packets.

[0089] (2) A sending node successfully sends a data packet, but the data packet is lost due to link quality, resulting in the destination node not receiving it;

[0090] (3) A sending node successfully sends a data packet, which is successfully received by the destination node.

[0091] In the above three cases, nodes can distinguish them by the received RTS, CTS, DATA and ACK. The information observed by access point i (assuming that the total number of access points in the distributed scheduling method is I) in time slot t is defined as For case (1), For cases (2) and (3), if the sending node of the data packet is node i, then Otherwise, for case (2), For case (3), Historical Information It is defined as the set of observation information of the sending node i from time slot t-Prd to time slot t-1. That is: historical information It can be expressed as

[0092] Furthermore, in the distributed scheduling method, the action definition of the partially observable Markov decision process is similar to the Markov decision process definition in the centralized scheduling method. It schedules one data stream or does not schedule any data stream. i data streams, then the action space A of access point i i The size of N i+1. In each time slot, each access point makes a scheduling decision independently, and the actions of all access points are combined into a joint action. The size of the joint action space is

[0093] Furthermore, a distributed real-time wireless network scheduling method based on deep reinforcement learning is proposed: a centralized training and distributed execution framework is used. In the centralized training phase, the agent is allowed to obtain global information, and in the execution phase, the agent makes scheduling decisions independently based on local information. The specific details are as follows:

[0094] In the distributed scheduling algorithm, an apprentice network is created on each access point, and each experience replay pool is created, and an expert network is created in the entire network. In the centralized training phase, the expert network can collect the status information of the sending queues of all data streams and give the optimal scheduling decision action for each access point. Each access point will execute the optimal scheduling decision action given by the expert network during the training phase, and the interactive information composed of local information and the corresponding optimal scheduling decision action will be used. Put them into their respective experience replay pools.

[0095] For each access point, in each time slot, a certain amount of interaction information is randomly extracted from all experience replay pools to update the apprentice network parameters on the access point. As in the centralized setting, the number of extracted interactive information, i.e., batch_size, is set to 64. For the extracted interactive information, (h j ,s j ,opt_a j ), where j∈[1,batch_size] represents the sequence number of each interactive information, and the parameters of the apprentice network are updated by minimizing a preset loss function:

[0096]

[0097] Among them, p jc is the output of the apprentice network, which means that when the input is h j ,s j The probability of taking action c when y jc When action c is the optimal action, y jc =1, otherwise y jc =0.

[0098] According to a second aspect of the present invention, a computer-readable storage medium is provided, wherein one or more computer programs are stored. When the computer programs are executed, the real-time wireless network scheduling method of the present invention is implemented.

[0099] According to a third aspect of the present invention, there is provided a network coding computing system, comprising a storage device and one or more processors; wherein the storage device is used to store one or more computer programs, and the computer programs are used to implement the real-time wireless network scheduling method of the present invention when executed by the processor.

[0100] In order to make the above features and effects of the present invention more clearly understood, embodiments are given below and described in detail with reference to the accompanying drawings.

[0101] The centralized real-time wireless network scheduling method based on deep reinforcement learning of the present invention is implemented as follows: Figure 1 As shown, it mainly includes the following steps:

[0102] Before the sending node interacts with a new environment, it creates a new sub-area in the experience replay pool to store the interaction information between the sending node and the environment.

[0103] When the access point interacts with the environment, in each time slot, the access point converts the queue information of the sending queue corresponding to all data flows. The state of each data flow is composed of the urgency of the data flow to be scheduled and the data packet arrival factor of the data flow. The collection of the states of all data flows constitutes the environment state s of the current time slot. t At the same time, the access point can obtain environmental feature information, namely: traffic model information and link quality information (offset, prd, D, B, pro) of all data flows 1～K , the agent performs scheduling decisions based on the environment state and environment feature information a t The agent performs scheduling decisions a t After that, we will receive feedback from the environment, including the reward value r t and the end mark t The status of all data flows will be updated to s t+1 . Interaction information (s t ,a t ,r t ,s t+1 ,end t ) will be stored in the corresponding area of ​​the experience replay pool.

[0104] In each time slot, the access point randomly extracts a certain amount of interaction information from the experience replay pool. Each interaction information belongs to a specific environment, which is described by environmental features. In this experiment, the number of extracted information is set to 64. The interaction information and environmental features are used together to train and update the network parameters. The neural network parameters are updated using the dual Q learning (DDQN) method.

[0105] Specifically, the access point creates two neural networks with the same structure, namely the current network and the target network. The current network is used to update network parameters, while the target network is used to select actions. Whether in the training phase or the test phase, when the access point needs to make a scheduling decision, it will obtain the environmental state of the current time slot and the environmental characteristics of the current environment as the input of the target network. After the target network obtains the input, it outputs the value of taking different actions. The access point then adopts different action selection strategies according to whether the current phase is the training phase or the test phase. If the current phase is the training phase, the access point randomly selects an action from the action space with a probability of ε to execute, and selects the scheduling action corresponding to the maximum value in the output of the target network with a probability of 1-ε to execute. Among them, ε is a decimal between 0 and 1, and ε is used to balance the ratio between utilization and exploration. Utilization means that the agent selects the action with the greatest value when selecting an action, and exploration means that the agent randomly executes actions in the action space when selecting an action. From a short-term perspective, utilization can maximize the expected benefit at the current moment, while exploration may maximize the total benefit from a long-term perspective. In the early stages of the agent's training, a larger ε value will be set so that the agent can fully obtain the value information of different actions. As the training process progresses, in order to ensure that the actions taken by the agent can converge to the actions with the greatest value, ε will gradually decrease to a decimal very close to 0. Therefore, the calculation formula for ε is as follows:

[0106]

[0107] Among them, ε 0 =0.99,ε min = 0.001, train_time refers to the number of environments that the access point has interacted with. The initial value is 0. When the access point completes each interaction with an environment, the value increases by 1. If the current phase is the test phase, the access point selects the scheduling action corresponding to the maximum value in the output of the target network. The parameter θ of the target network tar get At regular intervals, the current network parameter θ current covered.

[0108] The distributed real-time wireless network scheduling method based on deep reinforcement learning of the present invention is implemented as follows: Figure 2 As shown, it mainly includes the following steps:

[0109] The distributed implementation framework adopts centralized training and distributed execution, as follows.

[0110] In the centralized training phase, there is an expert network that can obtain global information, including: the status information of all data flows, the traffic model information of all data flows, and the link quality information. The expert network makes the optimal scheduling decision based on the global information. Since the expert network can obtain global information, it can be pre-trained in advance using the centralized scheduling method. In each time slot, the optimal scheduling decision output by the expert network Transmitted to each access point, each access point executes the received optimal scheduling decision action, and then the local information that the access point can obtain in the time slot: including historical information and status information of the scheduled data streams and optimal scheduling decision action Combined into interactive information, stored in the experience replay pool, each access point has an apprentice network, the apprentice network consists of two types of neural networks, namely the long time series network LSTM and the fully connected neural network FCN. The LSTM network is used to process historical information The FCN network is used to process the status information of the scheduled data flow In each time slot, the access point randomly extracts batches of experience from the experience replay pool to update the parameters of the apprentice network.

[0111] During the distributed execution phase, the expert network will be removed. When the access point makes a scheduling decision, it will no longer be guided by the optimal scheduling decision output by the expert network. The apprentice network of each access point outputs the value of each scheduling decision based on local information, and the access point will select the scheduling decision with the greatest value for execution.

[0112] The following will use specific examples to simulate and illustrate the real-time wireless network scheduling method based on deep reinforcement learning described in the present invention.

[0113] Figure 3 Two typical network topologies are shown: a network with a single-hop single access point serving multiple user nodes and a network with a single-hop multiple access points serving multiple user nodes.

[0114] In this experiment, the centralized scheduling method in the example of the present invention is compared with the existing LDF, RAC and RAC-Approx methods, and the distributed scheduling method is compared with the existing DB_DP method. At the same time, the real-time throughput obtained by the two methods is compared with the theoretical optimal value. At the same time, the adaptability of different algorithms in dynamic environments is compared, and the importance of certain improvement measures in the algorithms is also compared. The above experiment is used to evaluate the real-time wireless network scheduling method based on deep reinforcement learning described in the present invention.

[0115] Figure 4The functional configuration block diagram of the device for real-time wireless network scheduling based on deep reinforcement learning according to an embodiment of the present invention is shown. The device includes: a network state conversion unit, configured to convert the current network state into a specific format and transmit it to the neural network; an intelligent scheduling unit, configured to select the optimal scheduling strategy according to the network state; and a data packet forwarding unit, configured to forward data packets in the data queue corresponding to the scheduled data flow.

[0116] Figure 5 The real-time throughput of the centralized scheduling method and the existing methods under different numbers of data flows is shown. The number of data flows K∈[2,10], the traffic model of each data flow (offset k ,prd k ,D k ,B k ,pro k ) is set as follows: When k is an odd number, offset k = k-1, B k =0.8, pro k =0.9, D k =5. When k is an even number, offset k = k-1, B k =0.8, pro k =0.7, D k =4. Two different experiments were set up according to different arrival periods. In one experiment, the arrival period of the data flow was greater than the deadline of the data flow, and in the other experiment, the arrival period of the data flow was less than the deadline of the data flow. For the first experiment, if k is an odd number, then prd k =6, if k is an even number, then prd k =7. For the second set of experiments, if k is an odd number, then prd k =2, if k is an even number, then prd k =3. Figure 5 The results show that the real-time throughput of the centralized scheduling method and the RAC method can both achieve the optimal real-time throughput. However, as the number of data streams increases, due to the exponential increase in the number of states, RAC cannot give a scheduling strategy when the number of data streams is 6 and 8, while the centralized scheduling method can still give a scheduling strategy. At the same time, the real-time throughput of the centralized scheduling method under different numbers of data streams is also higher than that of the heuristic methods RAC-Approx method and LDF method.

[0117] Figure 6 The real-time throughput of the centralized scheduling method and the existing methods under different deadlines is shown. The number of data flows K = 4, the range of deadlines is D∈[2,6], and the traffic model of each data flow (offset k ,prok ,D k ,B k ,pro k ) is set as follows: When k is an odd number, offset k = k-1, B k =0.8, pro k =0.9, D k =D+1. When k is an even number, offset k = k-1, B k =0.8, pro k =0.7, D k =D. Two different experiments were set up according to different arrival periods. In one experiment, the arrival period of the data flow was greater than the deadline of the data flow, and in the other experiment, the arrival period of the data flow was less than the deadline of the data flow. For the first experiment, if k is an odd number, then prd k =6, if k is an even number, then prd k =7. For the second set of experiments, if k is an odd number, then prd k =2, if k is an even number, then prd k =3. Figure 5 The results show that the real-time throughput of the centralized scheduling method and the RAC method can both achieve the optimal real-time throughput. At the same time, the real-time throughput of the centralized scheduling method under different numbers of data flows is also higher than that of the heuristic methods RAC-Approx method and LDF method.

[0118] Figure 7 The real-time throughput of the distributed scheduling method and the existing methods under different numbers of data flows is shown. The experimental setup is similar to Figure 5 However, each data stream has a B k =1. Figure 7 The results show that the real-time throughput of the distributed scheduling method can achieve the optimal real-time throughput. At the same time, the real-time throughput of the distributed scheduling method under different numbers of data streams is also higher than that of the DB-DP method.

[0119] Figure 8 The real-time throughput of the distributed scheduling method and existing methods under different deadlines is demonstrated. The experimental setup is similar to Figure 6 However, each data stream has a B k =1. Figure 8 The results show that the real-time throughput of the distributed scheduling method can achieve the optimal real-time throughput. At the same time, the real-time throughput of the distributed scheduling method under different deadlines is also higher than that of the DB-DP method.

[0120] Fig. 9The role of state transition in the centralized scheduling method and the impact of using different reward function definitions are demonstrated. Two state definition methods are compared, namely, the state definition using a binary string sequence (represented by type 1) and the state transition definition in the present invention (represented by type 2). Two reward functions are compared, namely, the reward function set according to the transmission result of the data packet (represented by type 1) and the reward function set according to the link quality of the scheduled link (represented by type 2). Fig. 9 It shows that when the same reward function is used, the method cannot converge to the optimal real-time throughput when the state definition of type 1 is used, but the method can converge to the optimal real-time throughput when the state definition of type 2 is used. This shows that the state representation proposed in the present invention is better than the existing state representation method. At the same time, according to Fig. 9 ,When using the same state definition, using different reward functions will not affect the ,realtime throughput of the method after convergence, but it will affect the convergence time of the ,method. Using type 1 reward function design will prolong the convergence time of the ,method. In actual situations, if the sending node can know the link quality of the ,link, type 2 reward function can be used, otherwise type 1 reward function can be used.

[0121] Fig.10 The importance of expert networks in distributed scheduling methods is demonstrated, and the changes in real-time throughput of distributed scheduling methods during training are compared under two conditions. Fig.10 It can be seen that the use of expert networks can make the real-time throughput of the distributed scheduling method gradually increase to the optimal real-time throughput as the number of training times increases. Without the use of expert networks, the apprentice network of each sending node is trained independently, which will cause the real-time throughput of the distributed scheduling method to change unstably as the number of training times increases, and eventually become 0.

[0122] Fig.11 and Fig.12 The changes of real-time throughput of centralized scheduling method and distributed scheduling method in dynamic wireless network environment are demonstrated. Fig.11 and Fig.12 There are two different network environments in two different stages. Fig.11 It shows that the centralized scheduling method can adjust the scheduling strategy in time when the network environment changes, so that the real-time throughput can be restored to the optimal real-time throughput in the new environment. Although the other methods LDF and RAC-Approx can schedule the scheduling strategy in a shorter time, they cannot achieve the optimal real-time throughput. After the environment changes, the RAC method needs more time to calculate the optimal scheduling strategy in the new environment.

[0123] Therefore, for a period of time, the real-time throughput of the RAC method is 0. Fig.12It is shown that although the distributed scheduling method cannot quickly recover to the optimal real-time throughput after the environment changes, it can continue to train on the original network parameters so that the real-time throughput gradually reaches the optimal real-time throughput. Although the DB-DP method can quickly adjust the scheduling strategy, it cannot achieve the optimal real-time throughput.

[0124] The following is a system embodiment corresponding to the above method embodiment. This embodiment can be implemented in conjunction with the above embodiment. The relevant technical details mentioned in the above embodiment are still valid in this embodiment. In order to reduce repetition, they are not repeated here. Accordingly, the relevant technical details mentioned in this embodiment can also be applied in the above embodiment.

[0125] The present invention also proposes a real-time distributed wireless network scheduling system based on deep reinforcement learning, which includes:

[0126] An initial module is used to obtain a wireless network consisting of multiple access points and multiple user nodes interconnected, each access point schedules part of the data flow in the network, and each access point has an experience playback pool locally;

[0127] Scheduling module, used when each access point i interacts with the network environment, in each time slot t of the training phase, the global expert network in the network collects the status information, traffic model information and link quality information of all data flows, and gives the optimal scheduling decision action for each access point, and each access point i executes the optimal scheduling decision action given by the global expert network Each access point i obtains the state of the data stream scheduled by access point i in time slot t according to the sending queue information corresponding to some data streams At the same time, obtain the historical observation information of the wireless channel before the cutoff time slot t Interactive Information Stored in the local experience replay pool of access point i, at each time slot t in the test phase, each access point i will replay the state of part of the data flow and historical observation information Input to the decision model, the access point executes the scheduling decision corresponding to the output of the decision model

[0128] The training module is used to enable each access point i to extract experience from a local experience replay pool to train and update the decision model of the access point i.

[0129] The real-time distributed wireless network scheduling system based on deep reinforcement learning, wherein the training of the decision model includes:

[0130] An expert network for acquiring global information is constructed. The global information includes the status information, traffic model information and link quality information of all data flows in the wireless network. In each time slot, the expert network makes the optimal scheduling decision for each access point based on the global information. Each access point executes the received optimal scheduling decision action. At the same time, the access point stores the historical observation information, local environmental status and optimal scheduling decision action obtained in the current time slot into the local experience replay pool of the access point. Each access point is equipped with an apprentice network. In each time slot, the access point randomly extracts batches of experience from the local experience replay pool to update the parameters of the apprentice network until the scheduling decision output by the apprentice network reaches the preset standard or reaches the preset number of updates, and the current apprentice network is saved as the decision model.

[0131] The real-time distributed wireless network scheduling method based on deep reinforcement learning, wherein the apprentice network includes a long time series network and a fully connected neural network, the long time series network is used to process historical observation information, and the fully connected neural network is used to process the state of the scheduled data stream;

[0132] The historical information is the data packet transmission status monitored by the access point in each historical time slot. The data packet transmission status includes the following three types:

[0133] No data packets are transmitted, the sending node successfully sends a data packet but the data packet is not received by the destination node, and the sending node successfully sends a data packet and is successfully received by the destination node.

[0134] The real-time distributed wireless network scheduling method based on deep reinforcement learning, wherein the loss function is based on the following Update the apprentice network parameters of the i-th access point:

[0135]

[0136] Among them, p jc is the output of the apprentice network, indicating that when the input is h j ,s j The probability of taking scheduling action c when j ,s j are the historical information and environmental state of the jth experience in the batch experience respectively; y jc When action c is the optimal action, y jc =1, otherwise y jc =0.

[0137] The present invention also proposes a computer-readable storage medium for storing one or more computer programs, which, when executed, are used to implement any of the real-time distributed wireless network scheduling methods based on deep reinforcement learning.

[0138] The present invention also proposes a client for use in any of the real-time distributed wireless network scheduling systems based on deep reinforcement learning.

[0139] It should be noted that not all steps described in the above embodiments are necessary, and those skilled in the art may make appropriate choices, replacements, modifications, etc. according to actual needs.

[0140] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention is described in detail with reference to the embodiments above, it should be understood by those skilled in the art that any modification or equivalent replacement of the technical solutions of the present invention does not depart from the spirit and scope of the technical solutions of the present invention and should be included in the scope of the claims of the present invention.

Claims

1. A real-time distributed wireless network scheduling method based on deep reinforcement learning, characterized in that: include: Step 1: obtain a wireless network consisting of multiple access points and multiple user nodes interconnected, each access point schedules part of the data flow in the network, and each access point has an experience playback pool locally; Step 2: When each access point i interacts with the network environment, in each time slot t during the training phase, the global expert network in the network collects the status information, traffic model information, and link quality information of all data flows, and gives the optimal scheduling decision action for each access point. Each access point i executes the optimal scheduling decision action given by the global expert network. Each access point i obtains the state of the data stream scheduled by access point i in time slot t according to the sending queue information corresponding to some data streams At the same time, obtain the historical observation information of the wireless channel before the cutoff time slot t Interactive Information Stored in the local experience replay pool of access point i, at each time slot t in the test phase, each access point i will replay the state of part of the data flow and historical observation information Input to the decision model, the access point executes the scheduling decision corresponding to the output of the decision model Step 3: Each access point i extracts experience from the local experience replay pool to train and update the decision model of access point i; The training process of the decision model includes: Construct an expert network for acquiring global information, which includes the status information, traffic model information and link quality information of all data flows in the wireless network. In each time slot, the expert network makes the optimal scheduling decision for each access point based on the global information. Each access point executes the received optimal scheduling decision action. At the same time, the access point stores the historical observation information, local environmental status and optimal scheduling decision action obtained by it in the current time slot in the local experience replay pool of the access point. Each access point is equipped with an apprentice network. In each time slot, the access point randomly extracts batches of experience from the local experience replay pool to update the parameters of the apprentice network until the scheduling decision output by the apprentice network reaches the preset standard or reaches the preset number of updates, and the current apprentice network is saved as the decision model. The apprentice network includes a long time series network and a fully connected neural network, the long time series network is used to process historical observation information, and the fully connected neural network is used to process the state of the scheduled data stream; The historical information is the data packet transmission status monitored by the access point in each historical time slot. The data packet transmission status includes the following three types: No data packets are transmitted, the sending node successfully sends a data packet but the data packet is not received by the destination node, and the sending node successfully sends a data packet and is successfully received by the destination node; Based on the following loss function Update the apprentice network parameters of the i-th access point: Among them, p jc is the output of the apprentice network, indicating that when the input is h j ,s j The probability of taking scheduling action c when j ,s j The distribution is the jth historical information and environmental state in the batch experience; y jc When action c is the optimal action, y jc =1, otherwise y jc =0.

2. A real-time distributed wireless network scheduling system based on deep reinforcement learning, characterized in that: include: An initial module is used to obtain a wireless network consisting of multiple access points and multiple user nodes interconnected, each access point schedules part of the data flow in the network, and each access point has an experience playback pool locally; Scheduling module, used when each access point i interacts with the network environment, in each time slot t of the training phase, the global expert network in the network collects the status information, traffic model information and link quality information of all data flows, and gives the optimal scheduling decision action for each access point, and each access point i executes the optimal scheduling decision action given by the global expert network Each access point i obtains the state of the data stream scheduled by access point i in time slot t according to the sending queue information corresponding to some data streams At the same time, obtain the historical observation information of the wireless channel before the cutoff time slot t Interactive Information Stored in the local experience replay pool of access point i, at each time slot t in the test phase, each access point i will replay the state of part of the data flow and historical observation information Input to the decision model, the access point executes the scheduling decision corresponding to the output of the decision model A training module, used to enable each access point i to extract experience from a local experience replay pool to train and update the decision model of access point i; The training of the decision model includes: Construct an expert network for acquiring global information, which includes the status information, traffic model information and link quality information of all data flows in the wireless network. In each time slot, the expert network makes the optimal scheduling decision for each access point based on the global information. Each access point executes the received optimal scheduling decision action. At the same time, the access point stores the historical observation information, local environmental status and optimal scheduling decision action obtained by it in the current time slot in the local experience replay pool of the access point. Each access point is equipped with an apprentice network. In each time slot, the access point randomly extracts batches of experience from the local experience replay pool to update the parameters of the apprentice network until the scheduling decision output by the apprentice network reaches the preset standard or reaches the preset number of updates, and the current apprentice network is saved as the decision model. The apprentice network includes a long time series network and a fully connected neural network, the long time series network is used to process historical observation information, and the fully connected neural network is used to process the state of the scheduled data stream; The historical information is the data packet transmission status monitored by the access point in each historical time slot. The data packet transmission status includes the following three types: No data packets are transmitted, the sending node successfully sends a data packet but the data packet is not received by the destination node, and the sending node successfully sends a data packet and is successfully received by the destination node; Based on the following loss function Update the apprentice network parameters of the i-th access point: Among them, p jc is the output of the apprentice network, indicating that when the input is h j ,s j The probability of taking scheduling action c when j ,s j are the historical information and environmental state of the jth experience in the batch experience respectively; y jc When action c is the optimal action, y jc =1, otherwise y jc =0.

3. A computer-readable storage medium for storing one or more computer programs, which, when executed, are used to implement the real-time distributed wireless network scheduling method based on deep reinforcement learning as described in claim 1.

4. A client for use in the real-time distributed wireless network scheduling system based on deep reinforcement learning as described in claim 2.

Images