A method for improving air interface spectrum efficiency based on multi-agent reinforcement learning
By employing a Transformer-based deep reinforcement learning method, the high computational complexity and data explosion problems of resource allocation in multi-beam satellite communication are solved. This enables rapid learning of resource allocation strategies and improves spectrum efficiency, avoids co-channel interference, and enhances the intelligence and adaptability of the system.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHINA ACADEMY OF SPACE TECHNOLOGY
- Filing Date
- 2022-10-18
- Publication Date
- 2026-05-12
Smart Images
Figure CN115665878B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of high-throughput communication system technology, and in particular to a method for improving air interface spectrum efficiency based on reinforcement learning and multi-agent communication. Background Technology
[0002] To avoid interference from adjacent beams, traditional multi-beam satellites use the four-color theorem to allocate the frequency range of each beam, ensuring that adjacent beams do not use the same frequency and reducing co-channel interference. To achieve gigabit-level high-throughput satellite systems and maximize available data rates and spectrum utilization, a full-frequency reuse scheme can be adopted; however, this scheme introduces severe co-channel interference problems. Dynamic resource allocation is considered an effective approach to interference management. Currently, the main resource allocation methods in multi-beam satellite communications are as follows:
[0003] (1) Traditional base station-level radio resource allocation methods. The core idea of this method is to divide the cell into a central area and an edge area, and allocate specific radio resources to the corresponding area. For example, soft frequency reuse and partial frequency reuse methods better adapt to the distribution of services within and at the edge of the cell by adjusting the power threshold ratio of the subcarrier to the primary carrier. Although this scheme improves the throughput of users at the cell edge, the power threshold ratio needs to be readjusted after changes in service distribution, making it difficult to adapt to dynamic wireless network environments.
[0004] (2) Traditional user-level radio resource allocation methods include the round-robin algorithm, the maximum carrier-to-interference ratio (MCI) algorithm, and the proportional fairness algorithm. The round-robin algorithm is an algorithm that aims to maximize fairness by periodically allocating resources to users in a certain order. This method is simple to implement but does not consider factors such as service characteristics and user priorities. The MCI algorithm is an algorithm that aims to maximize performance by allocating all resources to the user with the best signal quality within the scheduling period. This method has the highest resource utilization rate but does not consider fairness factors at all. The proportional fairness algorithm is a compromise between the round-robin and MCI algorithms. This method comprehensively considers fairness and system performance, but it requires tracking the channel state, resulting in high algorithm complexity.
[0005] (3) Wireless Resource Allocation Method Based on Deep Reinforcement Learning. Deep reinforcement learning integrates the perception capabilities of deep learning with the decision-making capabilities of reinforcement learning, solving the problems of insufficient dynamism and intelligence in traditional resource allocation methods. Deep reinforcement learning technology models the wireless resource allocation problem as a continuous dynamic interaction between an agent and the wireless network environment. By learning the dynamics of the wireless environment through feedback information from the environment, it can make optimal resource allocation decisions. However, this method usually suffers from problems such as data explosion and large data demand, so it is difficult to achieve ideal results when there are many users and complex business scenarios.
[0006] While existing wireless resource allocation methods can mitigate interference to some extent and thus improve system spectral efficiency, some shortcomings still exist:
[0007] (1) Traditional resource allocation methods have high computational complexity in the optimization process, long time spent on iterative algorithms, and lack dynamism and intelligence, making them unsuitable for dynamic wireless network environments.
[0008] (2) Resource allocation methods based on deep reinforcement learning rely on a large amount of interactive data, which may cause problems such as the curse of dimensionality and data explosion in large-scale networks.
[0009] The purpose of this invention is to solve the problems of data dimensionality explosion and low sample efficiency in wireless resource allocation schemes based on deep reinforcement learning. Summary of the Invention
[0010] The purpose of this invention is to propose a method for improving the air interface spectrum efficiency of a multi-agent system based on reinforcement learning in order to solve the above-mentioned problems.
[0011] To achieve the above objectives, the present invention adopts the following technical solution:
[0012] A multi-agent air interface spectrum efficiency improvement method based on reinforcement learning models the resource allocation problem in multi-user cellular networks as a two-sequence decision process and solves it using deep reinforcement learning tools combined with Transformer, including:
[0013] By utilizing the attention mechanism in the Transformer architecture, we can mine the correlation between user location distribution and the allocation relationship between various resources to obtain multi-user resource allocation decisions within a single transmission time interval.
[0014] Also includes:
[0015] By leveraging the dynamic interaction between the agent and the cellular network environment in deep reinforcement learning, a resource allocation scheme can be obtained over multiple consecutive transmission time intervals.
[0016] Preferably, the wireless resource allocation method based on deep reinforcement learning with a Transformer architecture includes the following steps:
[0017] S1: Construct a resource allocation strategy model based on deep reinforcement learning with a Transformer architecture;
[0018] S2: The agent collects the observation status in a multi-user cellular network;
[0019] S3: Maps the user's multidimensional request information to a one-dimensional user tag;
[0020] S4: The user sequence consisting of user tags is input into the Transformer network, which outputs the allocation decision for each resource block;
[0021] S5: Executes resource allocation decisions and obtains reward information from multi-user cellular networks;
[0022] S6: The agent evaluates the value of the current environmental state and resource allocation actions;
[0023] S7: Repeat steps S2 to S6 above to collect data and calculate advantages;
[0024] S8: Train the resource allocation strategy network model offline using the collected data;
[0025] S9: Fine-tune the policy network model trained in step S8 above;
[0026] S10: Output the optimal resource allocation scheme based on the state at subsequent time steps.
[0027] Preferably, in step S1, a resource allocation strategy model based on deep reinforcement learning with a Transformer structure is constructed at the central controller. Within a single transmission time interval, a resource allocation model based on a Transformer structure is constructed; for multiple consecutive transmission time intervals, a resource allocation model based on deep reinforcement learning is constructed.
[0028] Preferably, in step S2, the central controller acts as an intelligent agent, collecting the observed states in the multi-user cellular network. These states primarily include the states of each user, the states of each resource block, and the request information of each user. This information collectively constitutes the state of the multi-user cellular network, and is represented as follows:
[0029] Preferably, in step S3, some information is selected from user requests as key factors affecting resource allocation, and user tags are extracted from them to avoid the resource allocation problem falling into the curse of dimensionality. The set of user tags constitutes a user sequence, which is input into the Transformer network.
[0030] Preferably, in step S4, the resource allocation action is generated based on a Transformer network: the user label set is input into the encoder of the Transformer structure, and the starting position of the resource allocation is input into the decoder of the Transformer structure. An attention mechanism is used to explore the correlation between user requests and resource allocation, and the allocation result of the first resource is output after sampling. Then, start bit union Both are used as input to the decoder to obtain the allocation result of the second resource. This process is repeated until the allocation of all resource blocks is obtained and represented as...
[0031] Preferably, in step S5, based on the resource allocation scheme given in step S4, the user transmits data at a certain power on a given resource block, obtaining reward information regarding system spectral efficiency and user fairness. Among them, Ψ t It is the system's spectral efficiency, Ψ max It is the theoretical threshold of the system's spectral efficiency, while Γ t Representing fairness to users, α1 and α2 are the weighting coefficients assigned to both.
[0032] Preferably, in step S6, the value of the observed state is evaluated based on the Critic network as V(s). t In step S7, multiple {s} are collected. t ,a t ,r t ,V(s t The training data is stored in the data cache, and the advantage function can be calculated simultaneously.
[0033] Preferably, in step S8, the network parameters are updated using the training data in the data cache so that the resource allocation strategy gradually converges to the optimal value. The loss functions of the Actor network and the Critic network are as follows:
[0034] The loss function of the Actor network is:
[0035]
[0036] Where θ1 represents the parameters of the Actor network, ρ t (θ1) is the ratio of the probabilities of the new and old strategies. The specific form of the clip(·) function is clip(ρ). t (θ1), 1-∈, 1+∈);
[0037] The loss function of the Critic network is:
[0038]
[0039] θ2 represents the parameters of the Critic network.
[0040] Preferably, in step S9, the trained strategy model continues to interact with the multi-user cellular network, and the resource allocation strategy model is fine-tuned online every once in a while using newly collected data to ensure that the optimal resource allocation strategy is obtained in real time; in step S10, the central controller collects the status information at subsequent times and inputs it into the strategy model in step S9 to obtain the optimal resource allocation scheme.
[0041] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:
[0042] 1. This application models the resource allocation problem in multi-cell, multi-user cellular networks as a dual-sequence decision process. It not only considers the temporal correlation in the allocation process, but also analyzes the allocation correlation between resources. This avoids the occurrence of co-channel interference to a certain extent, and can accelerate the learning and convergence speed of the resource allocation strategy model.
[0043] 2. Considering the temporal correlation in the allocation process, this application adopts deep reinforcement learning technology to model the resource allocation problem as a Markov process decision process. Through the interaction and learning between the agent and the multi-user cellular network, the intelligence and foresight of the resource allocation strategy are enhanced, and the goal of improving the system's spectrum efficiency is achieved.
[0044] 3. To avoid interference from the same frequency, this application adopts a Transformer structure from user sequence to resource sequence, and uses the attention mechanism therein to analyze the correlation between user location distribution and resource allocation relationship. At the same time, the Transformer sequence structure can solve the problems of data explosion and dimensionality curse, and to a certain extent enhances the generalization ability of the resource allocation strategy model. Attached Figure Description
[0045] Figure 1 A schematic diagram of a wireless resource allocation process based on deep reinforcement learning using Transformer, according to an embodiment of the present invention, is shown.
[0046] Figure 2 A schematic diagram of a wireless resource allocation method based on deep reinforcement learning using Transformer, according to an embodiment of the present invention, is shown. Detailed Implementation
[0047] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0048] Please see Figure 1-2 The present invention provides a technical solution:
[0049] A multi-agent air interface spectrum efficiency improvement method based on reinforcement learning models the resource allocation problem in multi-user cellular networks as a two-sequence decision process and solves it using deep reinforcement learning tools combined with Transformer, including:
[0050] By utilizing the attention mechanism in the Transformer architecture, we can mine the correlation between user location distribution and the allocation relationship between various resources to obtain multi-user resource allocation decisions within a single transmission time interval.
[0051] Also includes:
[0052] By leveraging the dynamic interaction between the agent and the cellular network environment in deep reinforcement learning, a resource allocation scheme can be obtained over multiple consecutive transmission time intervals.
[0053] Specifically, such as Figure 1 and Figure 2 As shown, the wireless resource allocation method based on deep reinforcement learning with a Transformer architecture includes the following steps:
[0054] S1: Construct a resource allocation strategy model based on deep reinforcement learning with a Transformer architecture;
[0055] S2: The agent collects the observation status in a multi-user cellular network;
[0056] S3: Maps the user's multidimensional request information to a one-dimensional user tag;
[0057] S4: The user sequence consisting of user tags is input into the Transformer network, which outputs the allocation decision for each resource block;
[0058] S5: Executes resource allocation decisions and obtains reward information from multi-user cellular networks;
[0059] S6: The agent evaluates the value of the current environmental state and resource allocation actions;
[0060] S7: Repeat steps S2 to S6 above to collect data and calculate advantages;
[0061] S8: Train the resource allocation strategy network model offline using the collected data;
[0062] S9: Fine-tune the policy network model trained in step S8 above;
[0063] S10: Output the optimal resource allocation scheme based on the state at subsequent time steps.
[0064] In a multi-cell, multi-user cellular network, assume there exists a virtual central controller responsible for managing the following information:
[0065] (1) A multi-user cellular network is deployed with B base stations, U users and C resource blocks;
[0066] (2) Users have active / inactive states, and resources have available / unavailable states;
[0067] (3) Users in the same cell are served on orthogonal resource blocks, and each user can occupy multiple resources. All resources in any cell can be reused by neighboring cells.
[0068] Specifically, such as Figure 1 and Figure 2 As shown, step S1, based on the aforementioned multi-user cellular network environment, constructs a resource allocation strategy model based on deep reinforcement learning with a Transformer structure at the central controller. Within a single transmission time interval, a resource allocation model based on a Transformer structure is built, mainly consisting of an encoder and a decoder. The encoder captures the positional correlation between users in the input sequence, while the decoder represents the resource allocation relationship in the output sequence and, combined with the hidden representation generated by the encoder, predicts the allocation of available resources. For multiple consecutive transmission time intervals, a resource allocation model based on deep reinforcement learning is built, mainly consisting of an Actor network and a Critic network. The Actor network is primarily responsible for generating the probability distribution of actions and selecting the resource allocation action to be executed based on this distribution. The Critic network evaluates and estimates the value of the actions selected by the Actor network.
[0069] In step S2, the central controller acts as an intelligent agent, collecting observational states in the multi-user cellular network. This primarily includes the states of each user, each resource block, and each user's request information. Request information includes: request ID, user identifier, user's affiliated base station, distance from the user to the base station, transmission delay, request queue length, number of newly arrived bytes, number of bytes transmitted in the previous transmission time interval, average throughput, resource block demand, bandwidth CQI, and subband CQI. This information collectively constitutes the state of the multi-user cellular network, represented as:
[0070] In step S3, some information is selected from the user request as key factors affecting the resource allocation effect, such as the user's base station, the distance between the user and the base station, the queue length of the request, the channel quality and the transmission delay, and user tags are extracted from them to avoid the resource allocation problem falling into the curse of dimensionality. The set of user tags constitutes the user sequence and is input into the Transformer network.
[0071] In step S4, resource allocation actions are generated based on a Transformer network: the user label set is input into the encoder of the Transformer structure, and the starting bit of resource allocation is input into the decoder of the Transformer structure. An attention mechanism is used to explore the correlation between user requests and resource allocation. After sampling, the allocation result of the first resource is output. Then, start bit union Both are used as input to the decoder to obtain the allocation result of the second resource. This process is repeated until the allocation of all resource blocks is obtained and represented as...
[0072] Step S5, based on the resource allocation scheme given in step S4, involves the user transmitting data at a certain power on a given resource block, obtaining reward information regarding system spectral efficiency and user fairness. Among them, Ψ t It is the system's spectral efficiency, Ψ max It is the theoretical threshold of the system's spectral efficiency, while Γ t Representing fairness to users, α1 and α2 are the weighting coefficients given to both.
[0073] In step S6, based on the Critic network, the value of the observed state is evaluated as V(s). t In step S7, the agent interacts multiple times with the multi-user cellular network to collect data and calculate advantages, collecting multiple {s} t ,a t ,r t ,V(s t The training data is stored in the data cache, and the advantage function can be calculated simultaneously.
[0074] In step S8, the training data in the data cache is used to update the network parameters so that the resource allocation strategy gradually converges to the optimal value. The loss functions of the Actor network and the Critic network are as follows:
[0075] The loss function of the Actor network is:
[0076]
[0077] Where θ1 represents the parameters of the Actor network, ρ t (θ1) is the ratio of the probabilities of the new and old strategies. The specific form of the clip(·) function is clip(ρ). t (θ1), 1-∈, 1+∈);
[0078] The loss function of the Critic network is:
[0079]
[0080] Where θ2 represents the parameters of the Critic network;
[0081] In step S9, the trained policy model continues to interact with the multi-user cellular network. Every so often, the resource allocation policy model is fine-tuned online using newly collected data to ensure that the optimal resource allocation policy is obtained in real time. In step S10, the central controller collects the state information at subsequent times and inputs it into the policy model in step S9 to obtain the optimal resource allocation scheme.
[0082] The key aspect of this invention is to construct a resource allocation strategy model based on a deep reinforcement learning method using a Transformer structure, according to the dual-sequence decision-making process of multi-user resource allocation. The optimal resource allocation strategy is learned through offline training and online fine-tuning using interactive data with the multi-user cellular network environment.
[0083] The Transformer structure can be replaced by a recurrent neural network. Both take sequential data as input and recursively perform operations in the direction of sequence evolution to represent the relationship between input and output. Deep reinforcement learning techniques can be replaced by meta-learning techniques. Both acquire experience or knowledge by collecting data to complete specific goals or tasks.
[0084] The above description of the embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A method for improving air interface spectral efficiency based on reinforcement learning for multiple agents, characterized in that, The resource allocation problem in multi-user cellular networks is modeled as a two-sequence decision process and solved using deep reinforcement learning tools combined with Transformer, including: By utilizing the attention mechanism in the Transformer architecture, we can mine the correlation between user location distribution and the allocation relationship between various resources to obtain multi-user resource allocation decisions within a single transmission time interval. Also includes: By leveraging the dynamic interaction between the agent and the cellular network environment in deep reinforcement learning, a resource allocation scheme can be obtained over multiple consecutive transmission time intervals. The wireless resource allocation method based on deep reinforcement learning with Transformer architecture includes the following steps: S1: Construct a resource allocation strategy model based on deep reinforcement learning with a Transformer architecture; S2: The agent collects the observation status in a multi-user cellular network; S3: Maps the user's multidimensional request information to a one-dimensional user tag; S4: The user sequence consisting of user tags is input into the Transformer network, which outputs the allocation decision for each resource block; S5: Executes resource allocation decisions and obtains reward information from multi-user cellular networks; S6: The agent evaluates the value of the current environmental state and resource allocation actions; S7: Repeat steps S2 to S6 above to collect data and calculate advantages; S8: Train the resource allocation strategy network model offline using the collected data; S9: Fine-tune the policy network model trained in step S8 above; S10: Output the optimal resource allocation scheme based on the state at subsequent time steps.
2. The method for improving air interface spectral efficiency based on reinforcement learning for multiple agents according to claim 1, characterized in that, In step S1, a resource allocation strategy model based on deep reinforcement learning with a Transformer structure is constructed at the central controller. Within a single transmission time interval, a resource allocation model based on a Transformer structure is built; for multiple consecutive transmission time intervals, a resource allocation model based on deep reinforcement learning is built.
3. The method for improving air interface spectral efficiency based on reinforcement learning for multiple agents according to claim 1, characterized in that, In step S2, the central controller acts as an intelligent agent, collecting observational states in the multi-user cellular network. These states primarily include the status of each user, the status of each resource block, and the request information of each user. This information collectively constitutes the state of the multi-user cellular network, and is represented as follows: .
4. The method for improving air interface spectral efficiency based on reinforcement learning for multiple agents according to claim 1, characterized in that, In step S3, some information is selected from user requests as key factors affecting resource allocation, and user tags are extracted from them to avoid the resource allocation problem falling into the curse of dimensionality. The set of user tags constitutes a user sequence, which is input into the Transformer network.
5. The method for improving air interface spectral efficiency based on reinforcement learning for multiple agents according to claim 1, characterized in that, In step S4, the resource allocation action is generated based on the Transformer network: the user label set is input into the encoder of the Transformer structure, and the starting bit of the resource allocation is input into the decoder of the Transformer structure. The attention mechanism is used to explore the correlation between user requests and resource allocation. After sampling, the allocation result of the first resource is output. Then, start bit union Both are used as input to the decoder to obtain the allocation result of the second resource. This process is repeated until the allocation of all resource blocks is obtained and represented as... .
6. The method for improving air interface spectral efficiency based on reinforcement learning for multiple agents according to claim 1, characterized in that, In step S5, based on the resource allocation scheme given in step S4, the user transmits data at a certain power on a given resource block, obtaining reward information regarding system spectral efficiency and user fairness. ,in, It refers to the system's spectral efficiency. It is the theoretical threshold of the system's spectral efficiency, while Representing fairness to users, and These are the weighting coefficients assigned to both.
7. The method for improving air interface spectral efficiency based on reinforcement learning for multiple agents according to claim 1, characterized in that, In step S6, the value of the observed state is evaluated based on the Critic network. In step S7, multiple data points are collected. Training data is collected and stored in a data cache, while the advantage function can be calculated. .
8. The method for improving air interface spectral efficiency based on reinforcement learning for multiple agents according to claim 1, characterized in that, In step S8, the training data in the data cache is used to update the network parameters so that the resource allocation strategy gradually converges to the optimal value. The loss functions of the Actor network and the Critic network are as follows: The loss function of the Actor network is: in, These represent the parameters of the Actor network. It is the ratio of the probabilities of the new and old strategies. The specific form of the function is ; The loss function of the Critic network is: in This represents the parameters of the Critic network.
9. The method for improving air interface spectral efficiency based on reinforcement learning for multiple agents according to claim 1, characterized in that, In step S9, the trained strategy model continues to interact with the multi-user cellular network. Every once in a while, the resource allocation strategy model is fine-tuned online using newly collected data to ensure that the optimal resource allocation strategy is obtained in real time. In step S10, the central controller collects the state information at subsequent times and inputs it into the strategy model in step S9 to obtain the optimal resource allocation scheme.