A multi-user scheduling method and system for 5G Internet of Things system based on reinforcement learning

Abstract

A 5G Internet of Things multi-user scheduling method based on reinforcement learning includes the following steps: according to a communication scenario model, calculating the actual achievable rate of each user in the set; generating an initial scheduling user set according to the actual achievable rate of each user; evaluating the action value estimate of each user in the current scheduling period by a Q-learning method according to the actual achievable rate of each user and the number of times each user is called; determining the upper limit of the confidence interval of the action value of each user; determining the scheduling user set in the current scheduling period according to the upper limit of the confidence interval of the action value of each user; and calculating the actual achievable rate of each selected user in the current scheduling period again according to the scheduling user set in the current scheduling period. Using the algorithm of the present invention, the base station can directly select M optimal users without trying different user combinations. This reduces the amount of calculation, and the system performance after the algorithm converges is not lower than that of the existing method.

Description

Technical Field

[0001] The present invention belongs to the field of wireless communication technology, and more specifically, relates to a multi-user scheduling method for a 5G Internet of Things system based on reinforcement learning. Background Art

[0002] With the continuous development of 5G, more IoT applications have been supported. However, due to the massive number of nodes in the IoT, the 5G network is required to support a large number of access users. Massive multi-user MIMO technology is one of the effective ways to solve the above problems. Its core idea is to configure a large number of antennas at the base station end and support multi-user communication at the same time by dividing spatial resources, which enables the system to serve multiple users on the same time-frequency resources. Since most IoT services are not continuous in time, in order to improve resource utilization, the system will allow the number of accessed users to exceed the number of users that can communicate simultaneously. In each scheduling cycle, the system will select a group of users from all accessed users and allocate resources to these users, that is, multi-user scheduling. Through multi-user scheduling technology, the utilization of system resources and system performance can be greatly improved. There are currently two problems with 5G IoT node scheduling: First, for IoT systems, due to the limited computing power of nodes and the strict latency requirements of most industrial IoTs, reducing the computational complexity of IoT node scheduling algorithms is one of the key points for large-scale application of 5G IoT; second, it is necessary to determine the user scheduling target, which is usually to maximize the total system throughput (sum of user throughput) or energy efficiency (throughput / energy) without exceeding the maximum number of scheduled users. While ensuring system performance, it is necessary to take into account some nodes with poor channel conditions to avoid these nodes being difficult to schedule, resulting in the loss of basic data transmission opportunities, causing "extreme unfairness".

[0003] Regardless of whether the scheduling criteria is to maximize the total system throughput or to maximize energy efficiency, the calculation of the user's achievable rate is involved, which involves a large number of matrix singular value decomposition (SVD) calculations, which is very computationally intensive. At the same time, the user's achievable rate is related to the scheduled user combination (and the user's joint channel matrix), so the optimal user combination needs to be exhaustively calculated. Currently, there are two methods to reduce the computational complexity of the scheduling algorithm. One is to simplify the calculation of the achievable rate, and the other is to replace the exhaustive algorithm. Existing literature focuses on reducing the amount of calculation through the former. Its scheduling method is essentially a greedy algorithm, which reduces the complexity of the algorithm by adopting a low-complexity representation of the user's achievable rate. Existing methods need to pre-calculate the achievable rate of each user when selecting users, and the achievable rate is related to the user channel matrix and the joint channel matrix. Therefore, different user combinations need to be considered, and multiple matrix SVDs are required to calculate the achievable rate, and the computational complexity reaches O(M0T 3). In the 5G IoT system, with the huge increase in the number of access users, the number of base station antennas, and the maximum number of scheduled users, in order to ensure the scheduling delay and performance of the system, the computational complexity of the user scheduling algorithm needs to be further reduced. Summary of the invention

[0004] In order to solve the deficiencies in the prior art, the purpose of the present invention is to achieve lower computational complexity in solving the user scheduling problem of the 5G Internet of Things system and ensure that the long-term performance of the system is not inferior to the existing scheduling algorithm. Furthermore, a multi-user scheduling method for a 5G Internet of Things system based on reinforcement learning is proposed.

[0005] The present invention adopts the following technical solution.

[0006] A 5G Internet of Things multi-user scheduling method based on reinforcement learning comprises the following steps:

[0007] Step 1: Calculate the actual achievable rate of each user in the set according to the communication scenario model;

[0008] Step 2: Generate an initial scheduling user set based on the actual achievable rate of each user;

[0009] Step 3: Based on the actual reachable rate of each user and the number of times each user is called, the estimated action value of each user in the current scheduling period is evaluated by the Q-learning method;

[0010] Step 4, determine the upper limit of the confidence interval of each user's action value;

[0011] Step 5: Determine the set of scheduled users in the current scheduling period according to the upper limit of the confidence interval of each user's action value;

[0012] Step 6, according to the scheduled user set in the current scheduling period, the actual achievable rate of each selected user in the current scheduling period is calculated again;

[0013] Step 7, return to step 3 until the action value estimate of each user converges to the actual achievable rate of each user in the current scheduling cycle, and output the scheduled user set of the last executed step 5 as the result of the next scheduling cycle.

[0014] Furthermore, the step 1 specifically includes:

[0015] The actual achievable rate R of user m m for:

[0016]

[0017] Among them, P m,n is the transmission power of user m in parallel channel n, σ2 is the additive white Gaussian noise power, λ m,n for The non-zero singular values ​​of are:

[0018]

[0019]

[0020] in, is the joint matrix of the channel matrices of other users except user m, and the above formula is The singular value decomposition of .

[0021] Furthermore, the step 2 specifically includes:

[0022] Step 21: Define the scheduled user set The set of all users in the environment A = {M0};

[0023] Step 22, for each user x in the set A-Ψ, define Ψ x =Ψ+{x}, calculate each Ψ x Rate for:

[0024]

[0025] Among them, R m Actual achievable rate as a user;

[0026] Step 23: Update the scheduled user set Ψ=Ψ+{x *},in, User x * ;

[0027] Step 24, return to step 22 until |Ψ|=M, then the initial scheduled user set π0=Ψ, where the maximum number of users that can be scheduled simultaneously M is determined by the following formula:

[0028]

[0029] Where T is the number of antennas, N i The number of antennas for each user.

[0030] Furthermore, the step 3 specifically includes:

[0031] The estimated value of any action of user i for:

[0032]

[0033] in represents the number of times user i is scheduled in the scheduling period 0, 1, …, t, I(·) is the indicator function, the function value is 1 if the event in the brackets is true, otherwise it is 0; β∈(0,1) is the discount factor; r i,τ is the actual achievable rate of user i in the τth scheduling period.

[0034] Furthermore, the estimated action value of any user i is Simplified to:

[0035]

[0036] Among them, t≥1.

[0037] Furthermore, the step 4 specifically includes:

[0038] The upper limit of the confidence interval for any user i for:

[0039]

[0040] Among them, M is the maximum number of users that can be scheduled simultaneously.

[0041] Furthermore, the action value of each user in step 7 converges to the actual achievable rate of each user, specifically including:

[0042]

[0043] Furthermore, the step 5 specifically includes:

[0044] The set of scheduled users in the current scheduling period π t for:

[0045]

[0046] in, is the upper limit of the confidence interval of any user i, π t for The maximum value of the sum.

[0047] Furthermore, if the current scheduling period is the first scheduling period, let the action value estimate of each user be

[0048] A 5G Internet of Things multi-user scheduling system based on reinforcement learning, comprising: a performance calculation module and a logic calculation module;

[0049] The performance calculation module is used to calculate the actual achievable rate of each user in the scheduled user set, and use these rates as the historical rates of the user to provide a basis for the system to evaluate the value of its actions;

[0050] The logic operation module generates an initial scheduling user set, evaluates the action value estimate of each user in the current scheduling cycle, determines the upper limit of the confidence interval of each user's action value, and determines the scheduling user set in the current scheduling cycle.

[0051] The beneficial effects of the present invention are that, compared with the prior art, the present invention has the following advantages:

[0052] (1) Using the algorithm of the present invention, the base station can directly select M optimal users without trying different user combinations, thereby reducing the amount of calculation, and the system performance after the algorithm converges is not lower than the existing method.

[0053] (2) The algorithm adopts the “exploration” operation, which not only improves the accuracy of each user’s estimated action value and the long-term throughput of the system, but also solves the “extreme unfairness” problem. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 Schematic diagram of a multi-user scheduling method for 5G Internet of Things system based on reinforcement learning.

[0055] Figure 2 This is a problem model diagram of the communication scenario model established by the present invention based on the characteristics of the 5G Internet of Things system.

[0056] Figure 3 It is a flow chart of the user scheduling method of the present invention.

[0057] Figure 4 This is the convergence time of the present invention.

[0058] Figure 5 The figure is a comparison of the system throughput after convergence of the algorithm of the present invention and other scheduling methods.

[0059] Figure 6 The user scheduling delay of the algorithm of the present invention is compared with other scheduling methods. DETAILED DESCRIPTION

[0060] The present application is further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present application.

[0061] The purpose of the present invention is to propose a multi-user scheduling method for a 5G Internet of Things system based on reinforcement learning. The method uses Q-learning to estimate the user's action value estimate from the user's past rate samples, and selects the user according to the Q value (the upper limit of the confidence interval of the action value estimate). After the user selection is completed, the base station calculates the reachable rate of each selected user. Compared with the traditional user scheduling algorithm, this method needs to collect the historical reachable rate of each user as a sample for evaluating the action value. These samples should be as many as possible so that the action value can be more accurate, so it takes time for the algorithm to converge. However, the proposed user scheduling method does not need to try a variety of different user combinations before scheduling, which can reduce a lot of unnecessary calculations of reachable rates, so the computational complexity is lower. At the same time, after the algorithm converges, the optimal scheduling user set can still be obtained to ensure the optimal system performance.

[0062] like Figure 1 As shown, the technical solution adopted by the present invention is: a 5G Internet of Things multi-user scheduling method based on reinforcement learning. In layman's terms, the method may include the following steps:

[0063] (1) According to the characteristics of the 5G IoT communication system, a corresponding communication scenario model is established, and an initial scheduling user set is generated. The actual achievable rate of each user in the set is calculated as the historical data rate sample of the user;

[0064] (2) Based on the historical data rate samples of each user, the estimated action value of each user in the current scheduling period is evaluated through the Q-learning method;

[0065] (3) The confidence upper bound algorithm is used to balance exploration and utilization, calculate the confidence interval upper bound of the user's action value estimate, and select the user based on the confidence interval upper bound. After selecting the user, the base station recalculates the actual achievable rate of each selected user. It should be noted that the achievable rate is used to re-update the action value estimate of each user in step (2).

[0066] (4) Repeat steps (2) and (3). After the algorithm converges, the estimated value of the user's action value is equal to the actual achievable rate. At this time, the user selected by the base station scheduling step (3) can achieve the highest system performance.

[0067] Furthermore, in the communication scenario model of the 5G IoT system in step (1), the received signal y of user m m It can be expressed as:

[0068]

[0069] where s m is the signal sent by user m, H mis the channel matrix of user m, T m is the precoding matrix of user m, n m is additive white Gaussian noise. Signal interference from other users needs to be eliminated through BD precoding. The main principle of BD precoding is to set the precoding matrix of each user to the null space matrix of the joint channel matrix (which is formed by concatenating the channel matrices of other users except the user). The multiplication of this null space matrix and the channel matrix of other users is equal to 0. is the joint matrix of the channel matrices of other users except user m, and performs singular value decomposition (SVD) on it:

[0070]

[0071] in is the joint matrix The left and right singular vectors corresponding to zero singular values ​​are composed of, Depend on The left and right singular vectors corresponding to the non-zero singular values ​​are composed of diagonal matrices The main diagonal elements of The non-zero singular values ​​of . Exists in the joint matrix is in the null space, so we have:

[0072]

[0073] So use By precoding the transmitted signal of user m, the interference between users can be completely eliminated. To ensure that there is a non-zero solution in the null space, The number of rows cannot be greater than The number of columns, so:

[0074]

[0075] Formula (4) is the constraint expression of the maximum number of users that can be scheduled simultaneously, M, which can be used to solve the maximum number of users that can be scheduled simultaneously, M. Where T is the number of antennas, N i The number of antennas for each user.

[0076] Let the precoding matrix in Depend on Obtained by singular value decomposition:

[0077]

[0078] in, is a diagonal matrix whose main diagonal elements are The non-zero singular values ​​of m，n Indicates that there are N mnon-zero singular values.

[0079] T m Substituting into (1), we get:

[0080]

[0081] Therefore, the channel of each user can be equivalent to multiple parallel channels. The achievable rate R of user m m for:

[0082]

[0083] Where P m,n is the transmission power of user m in parallel channel n. In the present invention, the transmission power of all parallel channels is preset to 0.1 W. m,n yes The non-zero singular values ​​of m σ 2 is the additive white Gaussian noise power.

[0084] In order to improve the utilization of space resources, the system can allow the number of connected users M0 to exceed the maximum number of users that can communicate simultaneously M. In each scheduling cycle, the system needs to select M users from M0 users based on the achievable rate. Let the set of scheduled users be Ψ, then the system throughput R Ψ It can be expressed as:

[0085]

[0086] It should be noted that the system throughput R Ψ It is equal to the total data rate of the system multiplied by the time length of the scheduling period. When the time length remains unchanged, maximizing the system throughput is equivalent to maximizing the total data rate of the system. The present invention regards the time length as a unit time, so R Ψ It can also represent the system throughput.

[0087] The present invention uses the learning ability of reinforcement learning for unknown environments, finds the optimal scheduling user set through iteration of Q value evaluation and scheduling set update, and achieves the goal of maximizing the total system throughput. The proposed reinforcement learning model consists of three main parts: agent, action, and reward, which are mapped to the user scheduling of the 5G IoT system:

[0088] Agent

[0089] The agent is the executor of reinforcement learning. In the present invention, the agent is the base station. There are M0 users in the environment where the agent is located, and M users (M is the maximum number of scheduled users) need to be selected for service. The state of the agent is represented by the users scheduled in this period.

[0090] Action

[0091] Action a i Indicates that the base station selects user i to access and performs action a in scheduling period t i After that, the agent state is transferred: the scheduled user set Ψ t Add user i in the scheduling cycle. The same action cannot be repeated in a scheduling cycle. A total of M actions need to be performed, and the action set A = {M0}. We record the set of users scheduled in cycle t as π t , whose modulus |π t |=M, and there is

[0092] Reward

[0093] After executing an action, the environment will give the agent a benefit. Since the user's reachable rate is affected by the joint channel matrix, the subsequent user selection will affect the reachable rate of the previous user. Therefore, we calculate the action benefit, that is, the user's reachable rate, after a round of scheduling is completed. We specify that the period t executes action a i The actual revenue obtained by the base station i (t) is the achievable rate of the selected user i in this round of scheduling, which is obtained by formula (7). The total revenue obtained by the base station after the scheduling period ends

[0094] Generate the initial scheduling user set. The specific steps are as follows:

[0095] Step (1.1): Scheduled user set The set A of all users in the environment = {M0}.

[0096] Step (1.2): For each user x in the set A-Ψ, define Ψ x =Ψ+{x}, calculate each Ψ according to formula (8) x Rate It should be noted that the rate here It can be understood as the system throughput, Ψ and Ψ x They are essentially the same, they are all collections.

[0097] Find the conditions that meet User m * , update the scheduled user set Ψ=Ψ+{m* It should be noted that x∈A-Ψ, the formula means to find the user m that maximizes the system throughput from A-Ψ * .

[0098] Step (1.3): Repeat step (1.2) until |Ψ| = M, and the initial scheduled user set π0 = Ψ. The achievable rate of user i is Total revenue from base stations

[0099] It should be noted that if there are not enough historical data rate samples at the beginning, the initial action value estimate of each user can be set Initial selection times

[0100] Furthermore, the step (2) specifically includes:

[0101] The estimated action value of a user is defined as the expectation of the user's achievable rate. In the Q-learning method, the state transition probability is required to calculate the action value. However, it is difficult to obtain the state transition probability in practice. The present invention uses the weighted average of the historical achievable rates of the user terminal as the action value estimate. The action value estimate of user i for:

[0102]

[0103] in represents the number of times user i is scheduled in the scheduling period 0, 1, …, t. I(·) is an indicator function. The function value is 1 if the event in the brackets is true, otherwise it is 0. i,τ is the reachable rate of user i in period τ (see formula (13)). β∈(0,1) is a discount factor that can reduce the weight of sample data obtained earlier to ensure the timeliness of the data.

[0104] Formula (9) is further simplified to reduce the storage pressure of the base station, as follows:

[0105]

[0106] Where t≥1, a i It should be noted that since steps (2) and (3) need to be iterated repeatedly, the variable a i and the variable π τ The meaning is approximately equal to the variable i and π τ In other words, the former is understood as a temporary variable in a certain iteration under the scheduling period t.

[0107] The advantage of using formula (10) is that it does not need to store all the historical reachable rates generated by the user, and the action value of the previous cycle is used to update the action value of the next cycle, thereby improving the calculation efficiency.

[0108] Furthermore, the step (3) specifically includes:

[0109] In the proposed scheduling method, the base station cannot obtain the true value of the action value when selecting users, and can only estimate the action value of all users through the previously reachable rate of the selected users. M user terminals with the maximum estimated action value and are selected. This operation is called "exploitation" because it utilizes existing information. However, since the action value of each user terminal is estimated by past data rate samples, the number of samples affects the accurate estimation of the action value. When the sample size is small, the inaccuracy of the action value makes it impossible for us to guarantee that there are no users among the other M0-M users who can obtain higher system throughput. At the same time, due to mobility or other reasons, changes in data rate may also cause user terminals outside the current scheduled user set to have higher action values. Therefore, it is necessary to schedule users outside the current scheduled user set that have been scheduled less times before, calculate the data rates of these users, increase the sample space of these users, and make the estimation of the action value of these users more accurate. This operation is called "exploration" because it tries to explore more unknown information. When the balance between "exploration" and "exploitation" is reached, that is, after accurately estimating the action value of each user terminal, the system will find M users who can truly enable the system to obtain the highest total throughput. In order to balance exploration and exploitation, we adopt the upper confidence limit algorithm (UCB) and define the Q value of the user terminal as the upper bound of the confidence interval of its action value.

[0110] The upper bound of the confidence interval of the action value of user i Defined as:

[0111]

[0112] The set of scheduled users π in the tth scheduling period t (t≥1) is:

[0113]

[0114] Among them, π t The selection of has nothing to do with the size of the user's Q value in the previous cycle, because after the Q value calculation in the previous cycle, a user selection will be performed. The rate calculation after selecting the user will be used to update the Q value to optimize the selection. The order of each scheduling cycle is: 1. Calculate the action value and Q value based on the historical rate and number of selections; 2. Select the user based on the Q value; 3. Calculate the rate of the selected user.

[0115] The π in the formula has been replaced by πt Avoid misunderstanding. The main goal of this formula is to select the π with the largest “sum of user Q values” as π t .

[0116] When the scheduling cycle begins, all the scheduling users of this scheduling cycle can be directly selected using formula (12), and then the reachable rates of these users are calculated according to (13) to update the action value and optimize the user selection for the next scheduling cycle. The agent uses (7) to calculate the scheduling user set π t The actual achievable data rate of each user in the scheduling period t, the achievable rate of user i (that is, the revenue generated by user i) is:

[0117]

[0118] The newly generated data rate can be used to update the action value estimates and the upper bounds of the confidence intervals of the action values ​​of all users for user selection in the t+1th period.

[0119] The difference between this algorithm and the common greedy algorithm is that it can select users first and then calculate performance (greedy is to calculate performance first and then select users based on performance). Therefore, this algorithm needs to select users and then calculate their actual performance to optimize the next user selection, and through continuous optimization, the final user selection can achieve the best performance.

[0120] From (11) and (12), it can be seen that the algorithm can traverse all users in the initial stage, avoiding the situation where some users are never selected. After multiple rounds of user scheduling, the number of selections for each user continues to increase, the confidence interval gradually converges, and the user's Q value is equal to the action value. The base station mainly selects users with higher action values ​​and maximizes the system throughput. From a long-term perspective, the algorithm can be used to balance exploration and utilization and ensure long-term system performance. In addition, the Q value of users who have not been scheduled for a long time can continue to grow without an upper bound. These users will eventually be scheduled by the system, so the algorithm can avoid "extreme unfairness".

[0121] Furthermore, the specific contents of step (4) are as follows:

[0122] Starting from the tth scheduling period (t≥1), repeat steps (2) and (3) until for any user When π is equal to π, the algorithm converges and obtains the optimal scheduling user set π * .

[0123] In summary, the difference between this algorithm and the common greedy algorithm is that it can select users first and then calculate performance (greedy is to calculate performance first and then select users based on performance). Therefore, this algorithm needs to select users and then calculate their actual performance to optimize the next user selection, and through continuous optimization, the final user selection can achieve the best performance.

[0124] Therefore, the idea of ​​the algorithm is to first use formula (12) to directly select all the scheduling users in this scheduling period, and then calculate the reachable rates of these users according to (13) to update the action value and optimize the user selection in the next scheduling period.

[0125] This method of selecting users based on historical data cannot guarantee that the optimal scheduling set can be selected every time before convergence, that is, π t ≠π * This is because there is not enough data support to build an accurate user data rate model. A certain number of samples are needed to obtain accurate data (evaluated by Q-learning). In order to obtain accurate data for all users, we use the confidence upper bound algorithm The control system is able to select those users with fewer selections before convergence, so that these users can be accurately evaluated.

[0126] So π t is obtained by comparing After reaching convergence, π t =π * .

[0127] The algorithm can be supplemented by selecting users first and then calculating the system performance that can be achieved by these users. Different from the greedy algorithm that first calculates the performance and then selects users, the present invention can greatly reduce the scheduling delay (the time from all users initiating scheduling applications to the base station completing user selection).

[0128] The specific implementation modes of the present invention are described in detail below with reference to the accompanying drawings.

[0129] A method for dividing cell users based on an improved support vector machine algorithm, the method comprising the following steps:

[0130] (1) Based on the characteristics of the 5G communication system, a communication scenario model from mobile users in a cell to the base station is established;

[0131] Figure 2 The communication system model diagram of the present invention is shown in FIG. The total number of base station transmitting antennas is T, and the maximum number of users that the system can schedule in a single scheduling cycle is M. The system selects M users from M0 (M0 ≥ M) users to provide services in each scheduling cycle. User m is equipped with N mThere are receiving antennas, m = 1, 2, ..., M0, and there are is the channel state matrix of user m. i,j represents the spatial channel fading coefficient from the jth transmitting antenna to the ith receiving antenna, rank(H m )=N m . Let s m Represents the transmission signal vector sent by the base station to user m. Each value contains the information to be transmitted T m is the precoding matrix of user m, then the received signal y of user m m for:

[0132] y m =H m T m s m +n m (1)

[0133] The precoding design is is the joint matrix The right singular vectors corresponding to the zero singular values ​​of are composed of, is the matrix The right singular vectors corresponding to the non-zero singular values ​​of are composed. At this time, the signal interference of other users is eliminated, and the achievable rate of user m can be obtained:

[0134]

[0135] (2) The base station records the historical reachable rate of each user and uses these rates as samples to estimate the action value of each user through the Q-learning method;

[0136] (3) The confidence upper bound algorithm is used to balance exploration and utilization. The Q value of a user is defined as the upper bound of the confidence interval of its action value. Users are selected based on the Q value. After selecting the user, the base station calculates the achievable rate of each selected user.

[0137] (4) Repeat steps 2) and 3). After the algorithm converges, the user's Q value is equal to the true action value. At this time, the user selected by the base station scheduling step 3) can achieve the highest system performance.

[0138] Correspondingly, the present invention also discloses a 5G Internet of Things multi-user scheduling system based on reinforcement learning, including: a performance calculation module and a logic calculation module.

[0139] The performance calculation module is used to calculate the actual achievable rate of each user in the scheduled user set, and use these rates as the historical rates of the user to provide a basis for the system to evaluate the value of its actions.

[0140] The logic operation module generates an initial scheduling user set, evaluates the action value estimate of each user in the current scheduling cycle, determines the upper limit of the confidence interval of each user's action value, and determines the scheduling user set in the current scheduling cycle.

[0141] Figure 3 The figure is a flow chart of the user scheduling method of the present invention. At the beginning of the algorithm, the initial scheduling user set is generated by step (1). After that, each scheduling cycle will first execute step (2) to estimate the action value of each user. The present invention uses the weighted average of the historical reachable rates of the user terminals as the estimated value of the action value. The estimated value of the action value of user i is for:

[0142]

[0143] Since the action value estimate of each user terminal may be inaccurate, in order to prevent the algorithm from falling into a local optimum, the present invention proposes step (3) to balance exploration and utilization, defining the Q value of a user as the upper bound of the user action value confidence interval, and using the Q value as the criterion for user selection. The Q value of user i is:

[0144]

[0145] The scheduling user set of the tth scheduling period is:

[0146]

[0147] For user i∈{M0}, the achievable rate

[0148] Repeat steps (2) and (3) for any user When the algorithm converges, the optimal scheduling user set π is obtained * .

[0149] Figure 4 This is the convergence time of the present invention. In a single base station MU-MIMO system. The number of transmitting antennas of the base station is T = 80, the total number of users is M0 = 20, and the number of receiving antennas of user m is N m =4(m=

[0150] 1,2,…M0). In the downlink of this system, the transmit power of each antenna is P avg Indicates that its upper limit is 10W, and the noise power σ 2 =10 -10W, discount factor β = 0.8. As can be seen from the figure, in the IoT scenario with relatively fixed nodes, the algorithm can converge with a high probability within 9000 scheduling cycles, while in the scenario where users move frequently, it is difficult for the algorithm to converge quickly. At this time, if new users frequently appear in the environment or users switch base stations due to movement, the convergence time of the algorithm will be further extended. Therefore, the present invention is more suitable for 5G IoT systems with relatively fixed nodes.

[0151] Figure 5 The system throughput after convergence of the algorithm of the present invention is compared with other scheduling methods. The scheduling method for comparison is the greedy algorithm, the number of transmitting antennas T is set to 40, N m =4(m=1,2,…M0). When M0≤10, the greedy algorithm and this algorithm have the same total throughput, because the total number of system users does not exceed the maximum number of schedulable users, and the system can schedule all users. When M0>10, the total throughput obtained by this algorithm can be higher than the throughput obtained by the greedy algorithm, because the greedy algorithm itself is a suboptimal user selection algorithm, and the highest throughput may not be obtained by using this algorithm. Although this algorithm has the problem of long convergence time, the total throughput obtained after convergence can be significantly higher than the greedy algorithm, and the long-term performance is higher than that of the greedy algorithm, which can ensure that the system performance is not affected.

[0152] Figure 6 The user scheduling delay of the algorithm of the present invention is compared with other scheduling methods, and the number of transmitting antennas is T=40. In the comparison method, the user selection method adopts a greedy algorithm. However, when characterizing the achievable data rate of the user, the Frobenius norm or other methods are used to reduce the amount of calculation. In fact, the algorithm proposed in the present invention does not prevent the use of the Frobenius norm or other methods to further reduce the amount of calculation. In the algorithm proposed in the present invention, each scheduling period base station needs to go through three stages: action value evaluation, scheduling set update, and calculation of achievable rate.

[0153] 1) In the action value evaluation phase, the base station estimates the action values ​​of M0 users using the method of formula (10), and the computational complexity is O(M0).

[0154] 2) In the scheduling set update phase, the base station calculates the Q values ​​of M0 users and finds the M users with the largest sum of Q values ​​to form the scheduling user set. The calculation complexity is O(M0).

[0155] 3) In the achievable rate calculation stage, the base station calculates the rates of each of the M users. The amount of computation required to calculate the joint channel matrix SVD is O(T 3 ), T is the number of antennas at the base station, so the computational complexity of this stage is O(MT 3 ).

[0156] Therefore, when the algorithm of the present invention is used, the computational complexity of a scheduling cycle is O(MT 3 +M0). Table 1 shows the computational complexity of various scheduling algorithms.

[0157] Table 1 Algorithm complexity

[0158]

[0159] When the total number of users is large, the computational complexity of the proposed algorithm is much lower than that of the greedy algorithm. At the same time, as the total number of users increases, the computational complexity of the proposed algorithm does not increase too much. Therefore, the proposed algorithm can ensure that the system maintains a low computational load even if there are a large number of IoT nodes in the 5G network.

[0160] The applicant of the present invention has made a detailed explanation and description of the implementation examples of the present invention in conjunction with the drawings in the specification. However, those skilled in the art should understand that the above implementation examples are only preferred implementation schemes of the present invention, and the detailed description is only to help readers better understand the spirit of the present invention, but not to limit the scope of protection of the present invention. On the contrary, any improvements or modifications based on the inventive spirit of the present invention should fall within the scope of protection of the present invention.

Claims

1. A multi-user scheduling method for a 5G Internet of Things system based on reinforcement learning, characterized in that: The steps include: Step 1: Calculate the actual achievable rate of each user in the set according to the communication scenario model. The actual achievable rate R m for: Among them, P m,n is the transmission power of user m in parallel channel n, σ 2 is the additive white Gaussian noise power, N m is the number of antennas of user m, λ m,n for The non-zero singular values ​​of , Hm is the channel matrix of user m, is the joint matrix The right singular vectors corresponding to the zero singular values ​​of are composed of, is the joint matrix of the channel matrices of other users except user m; Step 2: Generate an initial scheduling user set based on the actual achievable rate of each user, including: Step 21: Define the scheduled user set The set of all users in the environment A = {M0}; Step 22, for each user x in the set A-Ψ, define Ψ x =Ψ+{x}, calculate each Ψ x Rate for: Among them, R m Actual achievable rate as a user; Step 23: Update the scheduled user set Ψ=Ψ+{x * }, to find the conditions that meet Users; Step 24, return to step 22 until |Ψ|=M, then the initial scheduled user set π0=Ψ, where the maximum number of users that can be scheduled simultaneously M is determined by the following formula: Where T is the number of antennas, N i The number of antennas for each user; Step 3: Based on the actual achievable rate of each user and the number of times each user is called, the estimated action value of each user in the current scheduling period is evaluated by the Q-learning method. The estimated action value of the user is the expectation of the user's achievable rate. Step 4, determine the upper limit of the confidence interval of each user's action value; Step 5: Determine the set of scheduled users in the current scheduling period according to the upper limit of the confidence interval of each user's action value; Step 6, according to the scheduled user set in the current scheduling period, the actual achievable rate of each selected user in the current scheduling period is calculated again; Step 7, return to step 3 until the action value estimate of each user converges to the actual achievable rate of each user in the current scheduling cycle, and output the scheduled user set of the last executed step 5 as the result of the next scheduling cycle.

2. According to a 5G Internet of Things system multi-user scheduling method based on reinforcement learning according to claim 1, it is characterized in that: The step 1 specifically includes: Among them, the above formula is The singular value decomposition of ; is the joint matrix The left and right singular vectors corresponding to zero singular values; Depend on It is composed of left and right singular vectors corresponding to non-zero singular values; M is the maximum number of users that can be scheduled in a single scheduling cycle.

3. According to a 5G Internet of Things system multi-user scheduling method based on reinforcement learning according to claim 1, it is characterized in that: The step 3 specifically includes: The estimated value of any action of user i for: Among them, a i Represents the user, represents the number of times user i is scheduled in the scheduling period 0, 1, …, t, I(·) is the indicator function, the function value is 1 if the event in the brackets is true, otherwise it is 0; β∈(0,1) is the discount factor; r i,τ is the actual achievable rate of user i in the τth scheduling period.

4. According to a 5G Internet of Things system multi-user scheduling method based on reinforcement learning according to claim 3, it is characterized in that: The estimated value of any action of user i Simplified to: Among them, t≥1.

5. According to a 5G Internet of Things system multi-user scheduling method based on reinforcement learning according to claim 3, it is characterized in that: The step 4 specifically includes: The upper limit of the confidence interval for any user i for: Among them, M is the maximum number of users that can be scheduled simultaneously.

6. According to a 5G Internet of Things system multi-user scheduling method based on reinforcement learning according to claim 5, it is characterized in that: The action value of each user in step 7 converges to the actual achievable rate of each user, specifically including:

7. According to a 5G Internet of Things system multi-user scheduling method based on reinforcement learning according to claim 1, it is characterized in that: The step 5 specifically includes: The set of scheduled users in the current scheduling period π t for: in, is the upper limit of the confidence interval of any user i, π t for The maximum value of the sum.

8. According to a 5G Internet of Things system multi-user scheduling method based on reinforcement learning according to claim 1, it is characterized in that: The step 3 of evaluating the action value estimate of each user in the current scheduling period specifically includes: If the current scheduling period is the first scheduling period, let the action value estimate of each user be 9. A 5G Internet of Things system multi-user scheduling system based on reinforcement learning, characterized in that: include: Performance operation module and logic operation module; The performance calculation module is used to calculate the actual achievable rate of each user in the scheduled user set, and use these rates as the historical rates of the user to provide a basis for the system to evaluate the value of its actions. The actual achievable rate R of the user m for: Among them, P m,n is the transmission power of user m in parallel channel n, σ 2 is the additive white Gaussian noise power, N m is the number of antennas of user m, λ m,n for The non-zero singular values ​​of H m is the channel matrix of user m, is the joint matrix The right singular vectors corresponding to the zero singular values ​​of are composed of, is the joint matrix of the channel matrices of other users except user m; The logic operation module generates an initial scheduling user set, evaluates the action value estimate of each user in the current scheduling period, wherein the action value estimate of the user is the expectation of the user's achievable rate, determines the upper limit of the confidence interval of the action value of each user, and determines the scheduling user set in the current scheduling period, specifically including: Defining a scheduled user collection The set of all users in the environment A = {M0}; For each user x in the set A-Ψ, define Ψ x =Ψ+{x}, calculate each Ψ x Rate for: Among them, R m Actual achievable rate as a user; Update the scheduled user set Ψ=Ψ+{x * }, to find the conditions that meet Users; Until |Ψ|=M, then the initial scheduled user set π0=Ψ, where the maximum number of users that can be scheduled simultaneously M is determined by the following formula: Where T is the number of antennas, N i The number of antennas for each user; According to the actual achievable rate of each user and the number of times each user is called, the estimated action value of each user in the current scheduling cycle is evaluated by the Q-learning method. The estimated action value of the user is the expectation of the user's achievable rate; the upper limit of the confidence interval of each user's action value is determined; according to the upper limit of the confidence interval of each user's action value, the set of scheduled users in the current scheduling cycle is determined; According to the scheduled user set under the current scheduling cycle, the actual achievable rate of each selected user under the current scheduling cycle is calculated again; until the action value estimate of each user converges to the actual achievable rate of each user under the current scheduling cycle, and the last executed scheduling user set is output as the result of the next scheduling cycle.

Images