A method for uplink NOMA-URLLC resource allocation based on multi-agent reinforcement learning

Abstract

The present invention relates to an uplink NOMA-URLLC resource allocation method based on multi-agent reinforcement learning, including: generating a DQN network with each user as an agent, and setting the optimization target to maximize the number of successful decoding times; obtaining the channel state information of each user as the current state; inputting the current state of each user into the DQN network to generate the action of each user; generating rewards and the next moment state according to the action of each user; storing the current state, subchannel allocation action and power allocation action, rewards, and the next moment state as experience samples, and extracting and training the DQN network after reaching a certain number until the optimization target is reached. The optimized DQN network is used to perform subchannel allocation and power allocation for each user. The method of the present invention uses the DQN network to output the subchannel allocation and power allocation actions of each user, and then adjusts the allocation scheme according to the access rate fed back by the system, thereby maximizing the probability of successful user transmission.

Description

Technical Field

[0001] The present invention belongs to the technical field of wireless communications, and in particular relates to an uplink NOMA-URLLC resource allocation method based on multi-agent reinforcement learning. Background Art

[0002] Ultra-Reliable Low Latency Communication (URLLC) is one of the three major application scenarios of 5G. URLLC refers to services that require high reliability and low latency connections, such as driverless cars, industrial automation, and telemedicine. How to use limited system resources to maximize the needs of URLLC services has become an urgent problem to be solved in wireless communication networks. Therefore, it is of great significance to study the resource allocation problem of URLLC systems.

[0003] Non-Orthogonal Multiple Access (NOMA) uses power allocation technology to allocate the signal power of different users on the same subchannel according to a related algorithm, and then uses serial interference cancellation for decoding at the receiving end. This multiplexing technology in the power domain can reduce the transmission delay of each user and provide a new solution to the strict requirements of low latency in URLLC services.

[0004] However, non-orthogonal multiple access technology brings certain reliability issues. On the same channel, different users may choose the same sub-channel, which may lead to access failure due to power conflict. Therefore, a method is needed to balance delay, reliability and throughput. While using non-orthogonal multiple access methods to reduce transmission delay and improve throughput, it can also ensure the reliability requirements of user access. Summary of the invention

[0005] Based on the above-mentioned shortcomings and deficiencies in the prior art, one of the objects of the present invention is to at least solve one or more of the above-mentioned problems in the prior art. In other words, one of the objects of the present invention is to provide an uplink NOMA-URLLC resource allocation method based on multi-agent reinforcement learning that meets one or more of the aforementioned needs.

[0006] In order to achieve the above-mentioned object of the invention, the present invention adopts the following technical solutions:

[0007] A method for allocating uplink NOMA-URLLC resources based on multi-agent reinforcement learning, the method comprising the following steps:

[0008] S1. Treat each URLLC user as an intelligent agent and generate a corresponding DQN network.

[0009] The optimization goal of the DQN network is set to maximize the number of successful decodings of URLLC users within T time slots;

[0010] S2. Obtain channel state information of each URLLC user on different sub-channels as the current state of the URLLC user;

[0011] S3. Input the current state of each URLLC user into the corresponding DQN network to generate sub-channel allocation actions and power allocation actions for each URLLC user;

[0012] S4. Generate rewards and next moment status according to the subchannel allocation action and power allocation action of each URLLC user;

[0013] Store the current state, subchannel allocation action and power allocation action, reward, and next moment state in the experience replay buffer of the DQN network;

[0014] Return to step S2 until the number of samples in the experience playback buffer reaches a certain number, and then proceed to step S5;

[0015] S5. Extract a certain number of samples from the experience replay buffer to train the DQN network;

[0016] Return to step S2 until the optimization target is reached, and then proceed to step S6;

[0017] S6. Use the DQN network to perform sub-channel allocation and power allocation for each URLLC user.

[0018] As a preferred solution, the constraints of the optimization target in step S1 include:

[0019] The demodulation order constraints of users;

[0020] Power constraints for each subchannel;

[0021] Rate constraints for each URLLC user;

[0022] Decoding error rate constraints for each URLLC user;

[0023] Constraint on the number of sub-channels for each URLLC user.

[0024] As a preferred solution, each DQN network consists of three fully connected hidden layers, each hidden layer contains 250 neurons; the learning rate of each DQN network is 0.001, and the discount factor is 0.9.

[0025] As a preferred solution, step S3 specifically includes:

[0026] S31, input the current state of each URLLC user into the corresponding DQN network;

[0027] S32, generating an action space, where the actions in the action space are jointly composed of subchannel allocation actions and power allocation actions;

[0028] S33. Each DQN network randomly selects an action from the action space with probability ε, or selects the action with the maximum estimated Q value in the action space with probability 1-ε.

[0029] As a preferred solution, the transmission power levels of the power allocation action are discretized into four power levels.

[0030] As a preferred solution, the reward in step S4 is calculated as follows:

[0031]

[0032] Among them, r(t) is the reward, N u is the number of active URLLC users in the current time slot, N s is the number of subchannels, A Boolean value indicating whether the decoding was successful.

[0033] As a preferred solution, the experience replay buffer stores samples in a first-in-first-out manner.

[0034] As a preferred solution, in step S5, the DQN network is trained by minimizing the square error and updating the parameters by stochastic gradient descent.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] The method of the present invention utilizes the DQN network to output the subchannel allocation and power allocation actions of each user, and then adjusts the allocation scheme according to the access rate fed back by the system, thereby maximizing the probability of successful user transmission and realizing dynamic adjustment according to the number of users and the network environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 is a training flow chart of a DQN network according to an embodiment of the present invention;

[0038] Figure 2 It is a transmission success rate comparison chart of an uplink NOMA-URLLC resource allocation method based on multi-agent reinforcement learning in an embodiment of the present invention. DETAILED DESCRIPTION

[0039] In order to more clearly illustrate the embodiments of the present invention, the specific implementation methods of the present invention will be described below with reference to the accompanying drawings. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those of ordinary skill in the art, other accompanying drawings and other implementation methods can be obtained based on these accompanying drawings without creative work.

[0040] Embodiment: This embodiment provides an uplink NOMA-URLLC resource allocation method based on multi-agent reinforcement learning, the method comprising the following steps:

[0041] First, a conventional NOMA-assisted uplink URLLC network is constructed.

[0042] In the uplink URLLC network, there is a base station and N u users, users through N S The NodeB communicates with the base station through orthogonal sub-channels, and each sub-channel is a power-multiplexed NOMA cluster.

[0043] Based on this uplink URLLC system, the method of this embodiment is implemented. First, step S1 is performed to treat each URLLC user in the network as an intelligent agent and generate a corresponding DQN network. Since there are N u users, so create the corresponding N u A DQN network.

[0044] The optimization target model of the DQN network is set as:

[0045]

[0046] Where T is the number of time slots, The following formula takes value:

[0047]

[0048] Therefore, the optimization goal can be interpreted as maximizing the total number of successful decoding of URLLC users within T time slots.

[0049] In order to ensure the latency and reliability of URLLC, the following constraints are imposed on the above optimization target model:

[0050]

[0051]

[0052]

[0053]

[0054]

[0055] Constraint (a) constrains the demodulation order of users; constraint (b) constrains the power applied to each subchannel; constraint (c) constrains the rate of each user. Greater than the target data rate Constraint (d) constrains the decoding error rate of each user Less than the target error rate Constraint (e) constrains each user to only select one subchannel.

[0056] Each of the above DQN networks consists of three fully connected hidden layers, each hidden layer contains 250 neurons, the learning rate α = 0.001, and the discount factor γ = 0.9.

[0057] S2. Obtain channel state information of each URLLC user on different sub-channels as the current state of the URLLC user;

[0058] Specifically, suppose that the signal received by the base station in the uplink phase is Y=HS+N, where S is the pilot signal and N is the additive white Gaussian noise. The channel estimation

[0059] The user's current state s k (t) is defined as s k (t) = {s cha}∈S, where s cha Represents the channel state information, and the state space dimension is equal to the number of active URLLC users N in the current time slot u .

[0060] S3: Take the current state of each URLLC user as input to the DQN network corresponding to the user, and then output the subchannel allocation action and power allocation action a of each URLLC user. k (t).

[0061] Specifically, step S3 can be implemented as follows:

[0062] S31. At the beginning of the current time slot, the current state s obtained by each URLLC user in step S2 is k (t) Input to the DQN network corresponding to the URLLC user.

[0063] S32: Generate an action space for subsequent selection, where the action space includes actions formed by combining multiple subchannel allocation actions and power allocation actions. Define action a k (t) = {m, p}∈A, indicating that each URLLC user can decide which subchannel to select and which level of transmission power to be allocated, where the subchannel selection m∈{1, 2, ... N s}, the level of transmit power is selected as p∈{50, 150, 300, 500}, in mW. In this embodiment, since the transmit power is discretized into four power levels, the dimension of the action space is the number of channels * the number of power levels N s ×4.

[0064] S33, each DQN network uses the ε-greedy strategy to select subchannel allocation actions and power allocation actions from the action space a k (t). Specifically, the ε-greedy strategy is to randomly select an action a from the action space with probability ε k (t), or select the action a with the maximum estimated Q value in the action space with probability 1-ε k (t).

[0065] Finally, all sub-channel allocation actions and power allocation actions output by each DQN network are combined as follows:

[0066]

[0067] S4: Jointly train the DQN network according to the above assigned actions. The flowchart is as follows Figure 1 As shown, the subchannel allocation action and power allocation action of each URLLC user in step S3 are combined with a(t) and input into the DQN network to obtain the state s at the next moment k (t+1) and reward r(t).

[0068] The reward r(t) is calculated as follows:

[0069]

[0070] in Same as above,

[0071] After calculating the state and reward, collect experience samples e k (t) = {s k (t), a k (t), r(t), s k (t+1)}, the experience samples are stored in the experience replay buffer in a first-in-first-out manner, and after a certain number of experience samples are stored in the experience replay buffer, the DQN network is trained using these experience samples.

[0072] When the number of experience samples in the experience revisit buffer exceeds the set number, a batch of experience samples are randomly drawn from it and the Q estimate is calculated using these experience samples:

[0073]

[0074] Then calculate the square error L(θ) = (y k (t)-Q(s k (t), a k (t); θ(t))) 2 , train the Q network of the DQN network by minimizing the squared error:

[0075] Finally, stochastic gradient descent is used to update the parameters θ of the DQN network until the optimized target model reaches the maximum value.

[0076] After the optimization goal is achieved, the DQN network can be used to allocate subchannels and power to each URLLC active user. The comparison results of the transmission success probability achieved by the method of this embodiment with the ideal SIC scenario, random allocation action, and OMA solution are as follows: Figure 2 As shown in the figure, as the number of URLLC users increases, the method of this embodiment has a significantly better transmission success probability than the OMA solution and the random resource allocation solution, especially when the total number of users is large; the figure also provides the transmission success probability of the solution of this embodiment in an ideal SIC scenario. The results show that compared with the existing solutions, the present invention not only effectively guarantees the URLLC service performance indicators in actual scenarios, but also significantly improves the transmission success probability of the URLLC service, thereby effectively alleviating network pressure.

[0077] It should be noted that the above embodiments are only detailed descriptions of the preferred embodiments and principles of the present invention. For ordinary technicians in this field, there will be changes in the specific implementation methods based on the ideas provided by the present invention, and these changes should also be regarded as the scope of protection of the present invention.

Claims

1. A method for uplink NOMA-URLLC resource allocation based on multi-agent reinforcement learning, characterized in that: The method comprises the following steps: S1. Treat each URLLC user as an intelligent agent and generate a corresponding DQN network. The optimization goal of the DQN network is set to maximize the number of successful decoding of URLLC users within T time slots; S2. Obtain channel state information of each URLLC user on different sub-channels as the current state of the URLLC user; S3. Input the current state of each URLLC user into the corresponding DQN network to generate a subchannel allocation action and a power allocation action for each URLLC user; S4. Generate a reward and a next moment state according to the subchannel allocation action and power allocation action of each URLLC user; The current state, the subchannel allocation action and the power allocation action, the reward, and the next moment state are stored in the experience replay buffer of the DQN network; Return to step S2 until the number of experience samples in the experience playback buffer reaches a certain number, and then proceed to step S5; S5. Extract a certain number of experience samples from the experience replay buffer to train the DQN network; Return to step S2 until the optimization target is reached, and then proceed to step S6; S6. Use the DQN network to perform subchannel allocation and power allocation for each URLLC user; The optimization target model of the DQN network is set as: Where T is the number of time slots, The following formula takes value: Therefore, the optimization objective is interpreted as maximizing the total number of successful decodings of URLLC users within T time slots; In order to ensure the latency and reliability of URLLC, the following constraints are imposed on the above optimization target model: Constraint (a) constrains the demodulation order of users; constraint (b) constrains the power applied to each subchannel; constraint (c) constrains the rate of each user. Greater than the target data rate Constraint (d) constrains the decoding error rate of each user Less than the target error rate Constraint (e) constrains each user to only select one subchannel; Step S3 is implemented by the following method: S31. At the beginning of the current time slot, the current state s obtained by each URLLC user in step S2 is k (t) input to the DQN network corresponding to the URLLC user; S32, generating an action space for subsequent selection, wherein the action space includes actions formed by a combination of multiple subchannel allocation actions and power allocation actions; defining action a k (t) = {m, p}∈A, indicating that each URLLC user decides which subchannel to select and which level of transmission power to allocate, where the subchannel selection m∈{1, 2, ... N s }, the level of transmit power is selected as p∈{50, 150, 300, 500}, unit is mW; in this embodiment, since the transmit power is discretized into four power levels, the dimension of the action space is the number of channels * the number of power levels N s ×4; S33, each DQN network uses the ε-greedy strategy to select subchannel allocation actions and power allocation actions from the action space a k (t); Specifically, the ε-greedy strategy is to randomly select an action a from the action space with probability ε k (t), or select the action a with the maximum estimated Q value in the action space with probability 1-ε k (t); The reward r(t) is calculated as follows: Where r(t) is the reward, N u is the number of active URLLC users in the current time slot, N s is the number of subchannels, A Boolean value indicating whether the decoding is successful; in Same as above, After calculating the state and reward, collect experience samples e k (t) = {s k (t), a k (t), r(t), s k (t+1)}, store the experience samples in the experience replay buffer in a first-in-first-out manner, and start using these experience samples to train the DQN network after a certain number of experience samples are stored in the experience replay buffer; When the number of experience samples in the experience revisit buffer exceeds the set number, a batch of experience samples are randomly drawn from it and the Q estimate is calculated using these experience samples: Then calculate the square error L(θ) = (y k (t)-Q(s k (t), a k (t); θ(t))) 2 , train the Q network of the DQN network by minimizing the squared error: Finally, the parameters θ of the DQN network are updated using stochastic gradient descent until the optimized target model reaches the maximum value; When the optimization goal is reached, the DQN network is used to perform sub-channel allocation and power allocation for each URLLC active user.

2. The uplink NOMA-URLLC resource allocation method based on multi-agent reinforcement learning as claimed in claim 1, characterized in that: The constraints of the optimization objective in step S1 include: The demodulation order constraints of users; Power constraints for each subchannel; Rate constraints for each URLLC user; Decoding error rate constraints for each URLLC user; Constraint on the number of sub-channels for each URLLC user.

3. The uplink NOMA-URLLC resource allocation method based on multi-agent reinforcement learning as claimed in claim 1, characterized in that: Each of the DQN networks consists of three fully connected hidden layers, each of which contains 250 neurons; the learning rate of each of the DQN networks is 0.001, and the discount factor is 0.9.

Images