An orderly competitive large-scale access learning method for machine type communication systems

Abstract

The present invention discloses an orderly competitive large-scale access learning method for a machine type communication system, including: the newly accessed device uses a multi-agent reinforcement learning algorithm to collaboratively select a channel that meets its own needs; the device sends a preamble on a physical random access channel, and the base station sends a response after receiving the request, and the response includes a specific number of the device under the selected preamble, which is determined according to the priority of the device itself, and the device with the smallest number sends data on the physical random access channel, and the devices with the remaining numbers are in a waiting state, and the number of the waiting devices is automatically reduced after each unit time. The present invention weakens the randomness of the competition process, and the newly accessed device can collaboratively select a suitable preamble through a multi-agent reinforcement learning algorithm and divide it according to priority, while ensuring the minimum delay requirement of the device that has not yet successfully accessed.

Description

Technical Field

[0001] The present invention belongs to the technical field of wireless communication, and in particular relates to an orderly competitive large-scale access learning method in large-scale machine-to-machine (M2M) communication of the Internet of Things. Background Art

[0002] With the rapid development of communication technology, communication services have gradually evolved from traditional human-to-human communication to object-to-object communication. This communication method is called the Internet of Things (IoT). It is estimated that by 2023, the number of devices that can achieve M2M (Machine-to-Machine) communication will exceed 35 billion worldwide, which poses a severe challenge to the existing cellular network. Therefore, the fifth generation mobile communication (5G) technology has become the focus of research and development, among which mMTC (massive Machine Type Communication) is one of the three major application scenarios of 5G. In order to meet the massive access needs of machine-type devices, it is necessary to further optimize the control on the basis of the current congestion control mechanism. At the same time, considering the differences in the quality of service (QoS) of machine type communication devices (MTCD) of different service types, the access requirements of MTCDs of different service types are different. Therefore, the problem of preamble resource allocation when MTCDs of multiple service types access the cellular network should also be solved. For scenarios where MTCDs of multiple service types perform random access at the same time, the access delay, number of collisions and access fairness of MTCDs of various service types must be considered while improving the system throughput. However, most traditional access solutions are based on random competition, and usually use dynamic adjustment of the ACB (Access Class Barring) factor to optimize collisions, that is, backoff prediction. Although this can effectively alleviate the collision problem, collisions still occur. Therefore, a new orderly competition access solution is needed to solve the collision problem. Summary of the invention

[0003] Technical problem solved: This invention proposes a new orderly competition access solution for the collision problem of traditional random access based on competition. Unlike the traditional solution, this solution is not blind competition, but targeted competition, thus weakening the randomness in the competition process. Each device has different priorities and minimum latency requirements. When a collision occurs, the device will enter a queue state. The newly connected device can collaborate through a multi-agent reinforcement learning algorithm to select a suitable preamble code and divide it according to priority. At the same time, it is necessary to ensure the minimum latency requirements of the devices that have not yet successfully accessed.

[0004] Technical solution:

[0005] An orderly competitive large-scale access learning method for a machine type communication system, the orderly competitive large-scale access learning method comprising the following steps:

[0006] S1, before random access begins, the base station allocates radio resources of the physical random access channel and the physical uplink shared channel and broadcasts them to all mobile devices; each physical random access channel corresponds to a unique preamble code;

[0007] S2, at the beginning of random access, the base station broadcasts the number of devices waiting under each preamble, and the newly connected devices use a multi-agent reinforcement learning algorithm to collaboratively select a channel that meets their needs; specifically, the device sends a preamble on the physical random access channel, and the base station sends a response after receiving the request. The response includes a specific number of the device under the selected preamble, which is determined according to the priority of the device itself. The device with the smallest number sends data on the physical random access channel, and the devices with other numbers are in a waiting state. The number of waiting devices is automatically reduced after each unit time;

[0008] The multi-agent reinforcement learning algorithm combines the number of devices waiting for the preamble sequence, its own channel requirements, delay tolerance, and device priority, so that each device is within its maximum delay tolerance and each preamble sequence preamble i The length distribution is uniform.

[0009] Furthermore, in step S2, the number of each newly connected device is determined according to the following device priority function:

[0010] priorityfunc(t)=P MTCD +k(t-t0)

[0011] Where, t≥t0, constant P MTCD Indicates the priority level of the device itself; the symbol k represents the growth rate of the priority, and different devices have different growth rates; the symbol t represents the current time, and the symbol t0 represents the time when the device enters the team.

[0012] Furthermore, in step S3, assuming that there are m preambles, the device queue corresponding to the i-th preamble is preamble i , i = 1, 2, 3, ..., m; the total number of devices corresponding to all preamble queues at time t is expressed as n t,i It is the preamble queue preambl i The number of devices waiting in e, n t,acces is the number of newly connected devices; preamble queue preambli The maximum tolerable delay of the device in e is expressed as

[0013] The objective function of the multi-agent reinforcement learning algorithm is expressed as:

[0014]

[0015]

[0016] Among them, x t,i It indicates the number of devices that select the i-th preamble code among the newly connected devices at time t. p reambl ei,last Indicates the last device under the i-th preamble sequence after the new device decision is made. It represents the maximum tolerable delay of the last device under the i-th preamble sequence after the new device makes a decision; It represents the variance of the queue length under m preamble queues at time t, that is, the objective function is to minimize the variance.

[0017] Furthermore, in step S3, the process in which the newly connected device uses a multi-agent reinforcement learning algorithm to collaboratively select a channel that meets its own needs includes the following steps:

[0018] S31, construct state set S: The state set S is used to represent the state of the entire access environment, and consists of t+1 states, S = {s0, s1, ...s t}; Each state includes the device information s under each preamble sequence t ={preamble 1,t ,preamble 2,t ,...,preamble m,t};

[0019] S32, build action set A: Action set A is used to represent each agent according to the current state s t and the action a taken by its own decision strategy π t ; a t is a one-dimensional array of length m, where a i,t ={0,1}, if it is 1, it means that the device selects the i-th preamble code, if it is 0, it means that the device does not select the i-th preamble code;

[0020] S33, construct reward R: After the agent takes an action, the state of the current environment will change, and environmental benefits will be generated, corresponding to the reward r j,t It is expressed as:

[0021] S34, uses deep reinforcement learning to build a neural network, the input of which is action a t , status t , the output is the Q value Q of the action k (s t ,a t ), the target neural network is used to calculate the next state s t+1 Q value Q k (s t+1 ,a t ):

[0022]

[0023] where α k and γ are the learning rate and discount factor respectively, s t+1 and r t+1 Indicates the next state and in state s t The reward obtained after taking the action, a' represents the state s t+1 The executable actions under A are the executable action set. Indicates state s t+1 The maximum Q value in the next action set A is obtained by using the ε greedy strategy in the process of finding the maximum value; the loss function E is expressed as:

[0024] S35, uses the gradient descent method to update the weight θ of the neural network.

[0025] Furthermore, in step S35, the process of updating the weight θ of the neural network using the gradient descent method includes the following steps:

[0026] S351, randomly initialize the weights θ of the neural network and the actions of agent j Each preamble sequence i Set to 0;

[0027] S352, calculate the priority of the newly connected device, set the convergence threshold of the loss function E, and initialize α k ,γ,ε;

[0028] S353, each agent makes decisions based on the current state information and uses the ε greedy strategy;

[0029] S354, update the state of the environment s t+1 And reward r t+1 ;

[0030] S355, storage t 、a t 、s t+1 、r t+1 Parameter values ​​for experience playback;

[0031] S356, repeating steps S353 to S355, accumulating experience; randomly extracting a certain number of samples from the accumulated experience, calculating the loss function E based on these samples, and updating the weight θ;

[0032] S357, repeat steps S353 to S356 until the loss function E reaches the convergence condition or the maximum number of iterations T is reached.

[0033] Beneficial effects:

[0034] (1) Different from the traditional random contention access method, the ordered contention large-scale access learning method for machine type communication system of the present invention adopts the ordered contention access to solve the collision problem and allow more mobile devices (MTCD) to access at the same scale.

[0035] (2) The orderly competitive large-scale access learning method for machine type communication systems of the present invention adopts a multi-agent reinforcement learning algorithm to collaboratively select a suitable preamble code when the mobile device (MTCD) makes a decision. The use of this learning algorithm can better adapt to environmental changes and improve the convergence speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a diagram of an access model based on ordered competition according to an embodiment of the present invention.

[0037] Figure 2 This is a model diagram based on multi-agent reinforcement learning in an embodiment of the present invention.

[0038] Figure 3 It is a model diagram of each preamble code sequence in an embodiment of the present invention.

[0039] Figure 4 It is a neural network structure diagram of a multi-agent according to an embodiment of the present invention. DETAILED DESCRIPTION

[0040] The following examples will enable those skilled in the art to more fully understand the present invention, but are not intended to limit the present invention in any way.

[0041] This embodiment proposes an orderly competitive large-scale access learning method for a machine type communication system, and the orderly competitive large-scale access learning method includes the following steps:

[0042] S1, before random access begins, the base station allocates wireless resources of the physical random access channel and the physical uplink shared channel and broadcasts them to all mobile devices; each physical random access channel corresponds to a unique preamble code.

[0043] S2, at the beginning of random access, the base station broadcasts the number of devices waiting under each preamble code, and the newly connected devices use a multi-agent reinforcement learning algorithm to collaboratively select a channel that meets their own needs; specifically, the device sends a preamble code on the physical random access channel, and the base station sends a response after receiving the request. The response includes a specific number for the device under the selected preamble code, which is determined according to the priority of the device itself. The device with the smallest number sends data on the physical random access channel, and the devices with other numbers are in a waiting state. The number of waiting devices is automatically reduced after each unit time.

[0044] The multi-agent reinforcement learning algorithm combines the number of devices waiting for the preamble sequence, its own channel requirements, delay tolerance, and device priority, so that each device is within its maximum delay tolerance and each preamble sequence preamble i The length distribution is uniform.

[0045] In the scenario of a single base station (Base Station BS), there are several mobile devices MTCD. In the traditional solution, users can be divided into newly accessed devices and devices that have collided and backed off. In the present invention, the devices that have collided and backed off will not randomly compete for the preamble code again but will be in a queue state. Before the random access (Random Access RA) starts, the base station allocates the wireless resources of the physical random access channel (Physical Random Access Channel PRACH) and the physical uplink shared channel (Physical Uplink Shared Channel PUSCH) and broadcasts them to all MTCDs.

[0046] See also Figure 1 At the beginning of RA, the base station will broadcast the number of devices waiting under each preamble, so that the newly connected devices can decide to select the appropriate channel according to their own needs. Different from the traditional random access method, the device sends a preamble on PRACH, and the base station will send a response after receiving the request. The response includes a specific number for the device under the selected preamble, and the number is determined according to the priority of the device itself. The device with the smallest number sends data on PUSCH, and the device number will decrease automatically after each unit time. The devices with other numbers are in a waiting state. The priority function of the device is as follows:

[0047] priorityfunc(t)=P MTCD +k(t-t0);

[0048] Where t≥t0; the priority function consists of two parts, where the constant P MTCDIndicates the priority level of the device itself; the symbol k represents the growth rate of the priority level, and different devices have different growth rates; the symbol t represents the current time, and the symbol t0 represents the time when the device enters the team. After each unit time, the priority level of the waiting device in the team will increase to varying degrees.

[0049] Compared with the traditional random access method, since the device sends data on PUSCH alone in the time dimension, there is no collision and no conflict avoidance step. Assume that the arrival model of MTCD follows the Beta distribution, as shown below:

[0050]

[0051] Among them, α=3 and β=4.

[0052] See also Figure 2 In terms of device decision-making, a multi-agent reinforcement learning algorithm is used to collaborate with each other. Each device can be regarded as an agent, and the decision of each agent will affect the rest of the agents, because the length of the preamble will change with the addition of new devices. Device decision-making will take into account the number of devices waiting for the preamble sequence, its own channel requirements, delay tolerance, and device priority.

[0053] See also Figure 3 , assuming there are m preambles, which can be represented by numbers 1, 2, 3, ..., i, ..., m, and each preamble has a certain number of queues, represented by preamble i To indicate; each preamble queue has n t,i Waiting devices, plus the number of newly connected devices n t,access , then the total number of devices can be expressed as The maximum tolerable delay of each device can be expressed as During the device waiting process, it is necessary to ensure that the device's waiting time does not exceed this threshold. If the number of devices exceeds a certain scale, it cannot be ensured that each device is within its maximum delay tolerance. Assuming that the device at the head of the queue can successfully send data, the MTCD access success rate at time t can be expressed as:

[0054]

[0055] When a newly connected device uses multi-agent collaborative decision-making, it is also necessary to make each preamble sequence as uniform as possible. i The length of the following, that is, in summary, the objective function can be expressed as:

[0056]

[0057]

[0058] Among them, x t,i It indicates the number of devices that select the i-th preamble code among the newly connected devices at time t. p reamblei,last represents the last device under the i-th preamble sequence after the new device decision.

[0059] State set S: used to represent the state of the entire access environment, including the device information s under each preamble sequence t ={preamble 1,t ,preamble 2,t ,...,preamble m,t}, consisting of t+1 states, namely S={s0,s1,...s t}.

[0060] Action set A: Based on the current state, each agent takes action a according to its own decision strategy π t , a t is a one-dimensional array of length m, where a i,t ={0,1}, if it is 1, it means that the device selects the i-th preamble code, if it is 0, it means that the device does not select the i-th preamble code.

[0061] Reward R: After the agent takes an action, the state of the current environment will change, and environmental benefits or rewards will be generated. To simplify the calculation, we hope that each device at the end of the queue can meet its own delay tolerance during its waiting time, so the reward r j,t It can be expressed as:

[0062]

[0063] Where 1≤j≤n t,access Since the state set and action set are large in size, deep reinforcement learning is used to construct a neural network whose input is action a t , status t , the output is the Q value of the action, i.e. Q k (s t ,a t ), and the target neural network is used to calculate the next state s t+1 The Q value is Q k (s t+1 ,a t ), it can be updated by the following expression:

[0064]

[0065] where α kand γ are the learning rate and discount factor respectively, s t+1 and r t+1 Indicates the next state and in state s t The reward obtained after taking the action, a' represents the state s t+1 The executable actions under A are the executable action set. Indicates state s t+1 The maximum Q value in the next action set A is obtained, and the ε greedy strategy is used in the process of finding the maximum value.

[0066] Loss function E: To make Q k+1 (s t ,a t With Q k (s t ,a t ) is the smallest, which can make tends to 0, then the loss function E can be expressed as:

[0067]

[0068] The gradient descent method is used to update the weights θ of the neural network.

[0069] See also Figure 4 , according to the content mentioned in the previous technical solution, the specific implementation steps are as follows:

[0070] Step 1: Randomly initialize the weights θ of the neural network and the actions of agent j Each preamble sequence i Set to 0.

[0071] Step 2: Calculate the priority of the newly connected device, set the convergence threshold of the loss function E, and initialize α k ,γ,ε.

[0072] Step 3: Each agent makes decisions based on the current state information and uses the ε-greedy strategy.

[0073] Step 4: Update the status of the environment t+1 And reward r t+1 .

[0074] Step 5: Place the t 、a t 、s t+1 、r t+1 These parameter values ​​are stored for easy replay of experience.

[0075] Step 6: Randomly select a certain number of samples from the accumulated experience, calculate the loss function E based on these samples, and update the weight θ, repeating step 3 until the loss function E reaches the convergence condition or the program itself reaches the maximum number of iterations T.

[0076] The above is only a preferred embodiment of the present invention. It should be noted that, for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered as the protection scope of the present invention. All components not specified in this embodiment can be implemented by existing technologies.

Claims

1. An orderly competitive large-scale access learning method for a machine type communication system, characterized in that: The orderly competitive large-scale access learning method comprises the following steps: S1, before random access begins, the base station allocates radio resources of the physical random access channel and the physical uplink shared channel and broadcasts them to all mobile devices; each physical random access channel corresponds to a unique preamble code; S2, at the beginning of random access, the base station broadcasts the number of devices waiting under each preamble, and the newly connected devices use a multi-agent reinforcement learning algorithm to collaboratively select a channel that meets their needs; specifically, the device sends a preamble on the physical random access channel, and the base station sends a response after receiving the request. The response includes a specific number of the device under the selected preamble, which is determined according to the priority of the device itself. The device with the smallest number sends data on the physical random access channel, and the devices with other numbers are in a waiting state. The number of waiting devices is automatically reduced after each unit time; The multi-agent reinforcement learning algorithm combines the number of devices waiting for the preamble sequence, its own channel requirements, delay tolerance, and device priority, so that each device is within its maximum delay tolerance and each preamble sequence preamble i The length distribution is uniform. In step S2, the number of each newly connected device is determined according to the following device priority function: priorityfunc(t)=P MTCD +k(t-t0) Where, t≥t0, constant P MTCD Indicates the priority level of the device itself; the symbol k indicates the growth rate of the priority, and different devices have different growth rates; the symbol t indicates the current time, and the symbol t0 indicates the time when the device enters the team; Assume there are m preambles, and the device queue corresponding to the i-th preamble is preamble i , i = 1, 2, 3, ..., m; the total number of devices corresponding to all preamble queues at time t is expressed as n t,i It is the preamble queue i The number of devices waiting in t,access is the number of newly connected devices; preamble queue preamble i The maximum tolerable delay of the device in is expressed as The objective function of the multi-agent reinforcement learning algorithm is expressed as: Among them, x t,i Indicates the number of devices that select the i-th preamble code among the newly connected devices at time t, preamble i，last Indicates the last device under the i-th preamble sequence after the new device decision is made. It represents the maximum tolerable delay of the last device under the i-th preamble sequence after the new device makes a decision; It represents the variance of the queue length under m preamble queues at time t.

2. The method for learning ordered competitive large-scale access for a machine type communication system according to claim 1, characterized in that: The process by which a newly connected device uses a multi-agent reinforcement learning algorithm to collaboratively select a channel that meets its needs includes the following steps: S31, construct state set S: The state set S is used to represent the state of the entire access environment, and consists of t+1 states, S = {s0, s1, ...s t }; Each state includes the device information s under each preamble sequence t ={preamble 1,t ,preamble 2,t ,...,preamble m,t }; S32, build action set A: Action set A is used to represent each agent according to the current state s t and the action a taken by its own decision strategy π t ; a t is a one-dimensional array of length m, where a i,t ={0,1}, if it is 1, it means that the device selects the i-th preamble code, if it is 0, it means that the device does not select the i-th preamble code; S33, construct reward R: After the agent takes an action, the state of the current environment will change, and environmental benefits will be generated, corresponding to the reward r j,t It is expressed as: S34, uses deep reinforcement learning to build a neural network, the input of which is action a t , status t , the output is the Q value Q of the action k (s t ,a t ), the target neural network is used to calculate the next state s t+1 Q value Q k (s t+1 ,a t ): where α k and γ are the learning rate and discount factor respectively, s t+1 and r t+1 Indicates the next state and in state s t The reward obtained after taking the action, a′ represents the state s t+1 The executable actions under A are the executable action set. Indicates state s t+1 The maximum Q value in the next action set A, the ε greedy strategy is used in the process of finding the maximum value; the loss function E is expressed S35, uses the gradient descent method to update the weight θ of the neural network.

3. The method for learning ordered competitive large-scale access for a machine type communication system according to claim 2, characterized in that: In step S35, the process of updating the weight θ of the neural network using the gradient descent method includes the following steps: S351, randomly initialize the weights θ of the neural network and the actions of agent j Each preamble sequence i Set to 0; S352, calculate the priority of the newly connected device, set the convergence threshold of the loss function E, and initialize α k ,γ,ε; S353, each agent makes decisions based on the current state information and uses the ε greedy strategy; S354, update the state of the environment s t+1 And reward r t+1 ; S355, storage t 、a t 、s t+1 、r t+1 Parameter values ​​for experience playback; S356, repeating steps S353 to S355, accumulating experience; randomly extracting a certain number of samples from the accumulated experience, calculating the loss function E based on these samples, and updating the weight θ; S357, repeat steps S353 to S356 until the loss function E reaches the convergence condition or the maximum number of iterations T is reached.

Images