Cell-free network user resource allocation method based on cvar and reinforcement learning

Abstract

This invention provides a method for allocating user resources in a cellless network based on CVaR and reinforcement learning. The method includes: acquiring the state information of the cellless network at the current time step and inputting it into a reinforcement learning network to determine the action information at the current time step; allocating resources to each user according to the action information at the current time step and transitioning to the next time step; the reinforcement learning network is trained based on a CVaR-constrained target risk-aware advantage function, which includes a penalty term that explicitly imposes fairness constraints on the communication rate of the tail users in each user group at the current time step, and the communication rate of each user at the current time step is determined by the resources allocated to that user at the current time step. This invention ensures fairness in resource allocation among users.

Description

Technical Field

[0001] This invention relates to the field of wireless communication technology, and in particular to a cellless network user resource allocation method based on CVaR and reinforcement learning. Background Technology

[0002] Mobile networks are rapidly evolving from enhanced mobile broadband to massive machine-type communications and emerging immersive services. While 5G (5th Generation Mobile Communication Technology) has improved peak speeds and reduced latency compared to 4G (4th Generation Mobile Communication Technology), significant spatial and temporal differences in user experience remain due to factors such as obstructions, uneven mobility loads, and irregular site geometries. This gap will become even more pronounced as networks evolve towards 5G and 6G (6th Generation Mobile Communication Technology), as service guarantees become more defined and operators increasingly prioritize consistent experience over peak throughput. This evolution requires consideration of heterogeneous operating conditions and a focus on consistent user-level performance, rather than solely concentrating on increasing throughput on a few advantageous links.

[0003] Currently, cellular massive multiple-input multiple-output (CF-mMIMO) offers a promising architectural solution to meet these needs. In cellular networks (i.e., cellless networks), a large number of distributed access points (APs) work together to serve all users under centralized or semi-centralized coordination, utilizing the same time and frequency resources. Under the CF-mMIMO architecture, a large number of distributed access points (APs) collaboratively serve users in a user-centric manner.

[0004] However, the aforementioned technologies have difficulty guaranteeing the fairness of resource allocation for users. Summary of the Invention

[0005] This invention provides a cellless network user resource allocation method based on CVaR and reinforcement learning, which addresses the shortcomings of existing technologies in ensuring the fairness of user resource allocation. It achieves this by training a reinforcement learning network with a CVaR-based target risk-aware advantage function that explicitly constrains the communication rates of the tail users in the cellless network, and then using the trained reinforcement learning network to determine the action information at the current time step. Based on this action information, the method aims to achieve dynamic and fair resource allocation for each user.

[0006] This invention provides a cell-free network user resource allocation method based on CVaR and reinforcement learning, comprising: The state information of the current time step of the cellless network is obtained and input into the reinforcement learning network to generate actions and determine the action information of the current time step; the cellless network mentioned above includes multiple distributed access points (APs) and multiple users communicating with each AP; Based on the action information of the current time step, resources are allocated to each user, and the process is moved to the next time step; The aforementioned reinforcement learning network is trained based on a target risk perception advantage function constrained by Conditional Value at Risk (CVaR). This target risk perception advantage function includes a penalty term that explicitly imposes fairness constraints on the communication rate of the tail users in each user at the current time step. The communication rate of each user at the current time step is determined by the resources allocated to the corresponding user at the current time step.

[0007] According to the present invention, a cellless network user resource allocation method based on CVaR and reinforcement learning is provided, wherein the above-mentioned objective risk-aware advantage function is constructed in the following ways: Get the communication rate of all users in the cellless network at the current time step; all users in the cellless network include tail users; Obtain a learnable communication rate threshold; A penalty term is constructed based on the communication rate threshold and the rate difference between the communication rates of each user at the current time step; the penalty term will reduce the value of the target risk perception advantage function when the communication rate of the tail users is lower than the communication rate threshold. Construct a target risk perception advantage function based on the penalty term.

[0008] According to the present invention, a cellless network user resource allocation method based on CVaR and reinforcement learning is provided, wherein the penalty term is constructed based on a communication rate threshold and the rate difference between the communication rates of each user at the current time step, including: Calculate the communication rate threshold and the rate difference between the communication rate of each user at the current time step; Obtain the percentage of tail users and the number of each user; the percentage mentioned above is the proportion of tail users in the total number of users. Penalty terms are constructed based on the rate difference, quantity ratio, and number of users for each user.

[0009] According to the present invention, a cellless network user resource allocation method based on CVaR and reinforcement learning is provided. The method constructs a penalty term based on the rate difference, quantity ratio, and number of each user, including: Constraints are applied to each rate difference to determine the rate constraint result for each user; the rate constraint result includes taking 0 when the rate difference is less than 0 or taking the rate difference when the rate difference is greater than 0. Based on the rate constraint results, the proportion of each user, and the number of each user, a penalty term is constructed.

[0010] According to the present invention, a cellless network user resource allocation method based on CVaR and reinforcement learning is provided, wherein the above-mentioned construction of a target risk-aware advantage function based on a penalty term includes: Obtain the initial risk perception advantage function; The target risk perception advantage function is obtained by subtracting the initial risk perception advantage function from the penalty term.

[0011] According to the present invention, a cellless network user resource allocation method based on CVaR and reinforcement learning is provided, the method further includes: Based on the action information at the current time step, calculate the product of the probability ratio of the new and old policies and the target risk perception advantage function, and combine the product after pruning and restriction to construct the policy gradient objective function. Then, use the policy gradient objective function to train the reinforcement learning network.

[0012] According to the present invention, a cellless network user resource allocation method based on CVaR and reinforcement learning is provided, wherein the action information of the current time step includes the antenna downtilt angle of each AP in the current time step, the user preference information of each AP in the current time step, and the power control information of each AP in the current time step. The user preference information of each AP at the current time step is used to characterize the overall preference weight of each AP for each user when allocating resources at the current time step.

[0013] According to the present invention, a cellless network user resource allocation method based on CVaR and reinforcement learning is provided, wherein the resource allocation to each user based on the action information of the current time step includes: Based on the antenna downtilt angle of each AP at the current time step and the power control information of each AP at the current time step, determine the channel quality of each AP on any subcarrier; Based on the user preference information of each AP at the current time step and the channel quality of each AP on any subcarrier, the association score corresponding to each user allocated on each subcarrier of each AP is determined; the higher the association score, the higher the priority of the user to obtain resources, and the association score of each user is inversely proportional to the resources already allocated to the user; Based on the association score of each user allocated on each subcarrier of each AP, the subcarriers of the AP allocated to each user at the current time step are determined and subcarrier allocation is performed.

[0014] According to the present invention, a cellless network user resource allocation method based on CVaR and reinforcement learning is provided. The method for determining the channel quality of each AP on any subcarrier based on the antenna downtilt angle and power control information of each AP at the current time step includes: Based on the antenna downtilt angle of each AP at the current time step, determine the antenna gain of each AP at the corresponding antenna downtilt angle and pointing towards any user; Based on the location information of each user, the location information of each AP, and the characteristics of the wireless environment, determine the instantaneous channel power from each AP to any user on any subcarrier. The channel quality of each AP on any subcarrier is determined by the antenna gain of each AP at the corresponding antenna downtilt angle and pointing to any user, and the instantaneous channel power from each AP to any user on any subcarrier.

[0015] According to the present invention, a cellless network user resource allocation method based on CVaR and reinforcement learning is provided. The method determines the association score corresponding to each user allocated on each subcarrier of each AP based on the user preference information of each AP at the current time step and the channel quality of each AP on any subcarrier. This includes: Get the number of subcarriers that each AP has allocated to any user at the current time step; Based on the number of subcarriers that each AP has allocated to any user at the current time step, determine the dynamic load attenuation factor between each AP and the user with allocated subcarriers; among them, the more subcarriers a user has been allocated, the lower the association score. Based on the user preference information of each AP at the current time step, the channel quality of each AP on any subcarrier, and the dynamic load attenuation factor between each AP and the users on the allocated subcarriers, the association score corresponding to each user allocated on each subcarrier of each AP is determined.

[0016] According to the present invention, a cellless network user resource allocation method based on CVaR and reinforcement learning is provided, wherein the method for determining the subcarriers of the APs allocated to each user at the current time step based on the association score of each user allocated on each subcarrier of each AP includes: For each subcarrier of each AP, determine the maximum association score based on the association score of each user allocated on the AP's subcarrier, and allocate the AP's subcarrier to the user corresponding to the maximum association score at the current time step.

[0017] The present invention also provides a cellless network user resource allocation device based on CVaR and reinforcement learning, comprising the following modules: The action generation module is used to obtain the state information of the current time step of the cellless network and input the state information of the current time step into the reinforcement learning network for action generation to determine the action information of the current time step; the aforementioned cellless network includes multiple distributed access points (APs) and multiple users communicating with each AP; The user resource allocation module is used to allocate resources to each user based on the action information of the current time step and transfer to the current time step; The aforementioned reinforcement learning network is trained based on a target risk perception advantage function constrained by Conditional Value at Risk (CVaR). This target risk perception advantage function includes a penalty term that explicitly imposes fairness constraints on the communication rate of the tail users in each user at the current time step. The communication rate of each user at the current time step is determined by the resources allocated to the corresponding user at the current time step.

[0018] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the cellless network user resource allocation method based on CVaR and reinforcement learning as described above.

[0019] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the cellless network user resource allocation method based on CVaR and reinforcement learning as described above.

[0020] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the cellless network user resource allocation method based on CVaR and reinforcement learning as described above.

[0021] The present invention provides a cellless network user resource allocation method based on CVaR and reinforcement learning. This method acquires the state information of the cellless network at the current time step and inputs it into a reinforcement learning network for action generation, determines the action information for the current time step, allocates resources to each user based on the action information, and then moves to the next time step. The cellless network includes multiple access points (APs) and multiple users communicating with each AP. The reinforcement learning network is trained based on a target risk perception advantage function constrained by CVaR. The target risk perception advantage function includes a penalty term that explicitly imposes fairness constraints on the communication rate of the tail users in each user at the current time step. The communication rate of each user at the current time step is determined by the resources allocated to that user at the current time step. In this method, a penalty term based on CVaR constraints is introduced during reinforcement learning network training to explicitly constrain the communication rate of tail users. This constrains the communication rate of tail users with excessively low communication rates in the cellless network during dynamic resource allocation, ensuring that their communication rates are not too low. Since the communication rate is determined by the resources allocated to the user based on the AP, it is equivalent to constraining the tail users to receive a sufficient amount of resources. This ensures fair allocation of resources among users in the cellless network, thereby achieving risk avoidance control and ensuring that the service quality of tail users in the cellless network is effectively improved even when channel uncertainty exists. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0023] Figure 1 This is a schematic diagram illustrating an application scenario of the cellless network user resource allocation method based on CVaR and reinforcement learning provided by this invention.

[0024] Figure 2 This is a flowchart illustrating the cellless network user resource allocation method based on CVaR and reinforcement learning provided by the present invention.

[0025] Figure 3 This is a schematic diagram illustrating the specific process of the cellless network user resource allocation method based on CVaR and reinforcement learning provided by the present invention.

[0026] Figure 4 This is a simulation diagram of population and equipment distribution in two real-world scenarios in City A, provided by the present invention.

[0027] Figure 5This is a schematic diagram of the learning curve of the reinforcement learning network provided by the present invention during the training process.

[0028] Figure 6 This is a schematic diagram illustrating the impact of decisions generated by the CVaR-MAPPO algorithm and the MAPPO algorithm provided in this invention on the distribution of user rates in the system.

[0029] Figure 7 This is a schematic diagram of the cellless network user resource allocation device based on CVaR and reinforcement learning provided by the present invention.

[0030] Figure 8 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0032] Currently, Cell-Free Massive MIMO (CF-mMIMO) offers a promising architectural solution to meet these needs. In cellular networks (i.e., cellless networks), a large number of distributed access points (APs) serve all users collectively using the same time and frequency resources under centralized or semi-centralized coordination. In the CF-mMIMO architecture, these distributed APs collaboratively serve users in a user-centric manner. The concept of fixed cell boundaries disappears, replaced by dynamically formed clusters of cooperating access points supporting users. This user-centric collaboration enhances macro-diversity, suppresses inter-cell interference, and produces a smoother spatial rate distribution compared to traditional cellular-based deployments. These characteristics make cellular architecture highly attractive in dense deployments and scenarios with strong spatial heterogeneity. However, the cellular paradigm brings new optimization challenges beyond those of traditional cellular systems.

[0033] In the CF-mMIMO architecture, a large number of distributed access points collaboratively serve users in a user-centric manner. Early research mainly relied on precise modeling and convex optimization to alleviate the coupling between multidimensional resources. However, as network size and state dimensions increase, this framework relying on precise modeling and offline optimization becomes increasingly limited in terms of complexity and environmental adaptability, prompting a shift towards deep reinforcement learning (DRL) methods. CVaR (Conditional Value at Risk) provides a unified risk metric for characterizing tail-end expected losses exceeding a given confidence level, and it is an important tool for designing risk-averse and risk-sensitive strategies. In safety-critical fields such as robotics and autonomous driving, CVaR constraints are used to characterize the safety risk of collisions with randomly moving obstacles, and two-stage planning and control schemes are employed to improve trajectory performance while controlling tail-end collision risk. In contrast, research on combining CVaR with DRL to explicitly constrain tail-end user rates during the learning process for CF-mMIMO systems remains limited.

[0034] In the field of reinforcement learning, early work on CVaR was primarily conducted within the framework of risk-sensitive Markov decision processes. Several studies replaced the traditional reward function R with a risk function C and proposed a series of gradient-based and value function-based methods aimed at minimizing cumulative loss while keeping the CVaR below a specified threshold. Some proposed DSAC (Distributed Reinforcement Learning Soft Actor-Critic) for distributed reinforcement learning, which combines soft actor-critic with distributed value function methods such as implicit quantile networks. Others proposed CPPO, incorporating trajectory reward CVaR constraints into PPO (Proximal Policy Optimization), simultaneously improving cumulative reward and robustness to both types of perturbations on various MuJoCo (open-source physics simulation engine) continuous control tasks.

[0035] Overall, while current technologies have made progress in CF-mMIMO optimization and DRL-based resource management, two shortcomings remain: 1. The inherent distributed nature of CF-mMIMO is often simplified, leading to insufficient multi-agent coordination; 2. User fairness is a concern, as it is difficult to guarantee fairness in resource allocation, particularly regarding the protection of marginal users and tail performance, which is rarely explicitly guaranteed during the learning process. CVaR measures the expected outcome of the worst-case scenario at a selected confidence level. It provides a tail-oriented criterion that goes beyond average rate maximization and offers a controllable approach to handling the worst user experience, thus enabling explicit prioritization of users with poorer experiences to enhance fairness.

[0036] It can be seen that the core objective of this invention is to construct a dynamic optimization framework in the CF-mMIMO system that takes into account both system throughput performance and user fairness, so as to solve the problem that traditional resource allocation strategies with average rate or total throughput as the target are insufficient in guaranteeing fairness to users with tail risk. To achieve the above objective, this invention faces the following key challenges: (1) It is difficult to explicitly model the fairness objective: User experience fairness is essentially related to the tail behavior of rate distribution. Traditional average rate or minimum rate indicators are difficult to characterize stability, robustness and extreme user experience at the same time; (2) The optimization problem is highly non-convex and combinatorially complex: Antenna downtilt angle, user association, subcarrier allocation and power control are coupled with each other, forming a high-dimensional, nonlinear, integer and continuous variable mixed optimization problem, which is difficult to solve using traditional convex optimization methods; (3) The distributed and cooperative characteristics of the CF-mMIMO system: A large number of distributed APs need to make cooperative decisions with only local observations, making it difficult to extend centralized or single-agent optimization methods; Uncertainty of dynamic wireless environment: User movement, channel fading and load changes lead to highly random system states, making it difficult for static or offline optimization strategies to adapt to long-term operation requirements.

[0037] Based on this, embodiments of the present invention provide a method for allocating user resources in cellless networks based on CVaR and reinforcement learning, which can solve the above-mentioned technical problems. Specifically, it can jointly optimize the antenna downtilt angle, user preference, power allocation and subcarrier allocation of distributed access points (APs) to achieve effective resource coordination and improve the fairness of resource allocation in cellless network (CF-mMIMO) systems.

[0038] The system framework used in the technical solution of this invention will be described below. See [link to relevant documentation]. Figure 1 The diagram shown illustrates an application scenario of the cellless network user resource allocation method based on CVaR and reinforcement learning provided by this invention. This application scenario is a typical CF-mMIMO downlink communication system. (See attached diagram.) Figure 1As shown in Figure a, this system includes multiple distributed access points (APs), multiple user equipment (UEs), and a central processing unit (CPU). Each UE can communicate with each AP, and the CPU is used for centralized coordination and management. All APs are connected to the CPU via a high-speed, low-latency fiber optic fronthaul link. The CPU coordinates antenna control, user association, and resource allocation decisions. This system employs Orthogonal Frequency Division Multiplexing (OFDM) technology to achieve parallel access for multiple users (i.e., multiple user equipment) in the frequency domain. The frequency domain is divided into N orthogonal subcarriers. Each AP serves at most one user on any subcarrier, while different APs can simultaneously serve the same or different users on the same subcarrier to obtain diversity gain and improve coverage performance. For the APs mentioned above, see [link to relevant documentation]. Figure 1 As shown in Figure b, assuming the AP side is equipped with an adjustable downtilt directional antenna, the elevation angle of its main lobe relative to the horizontal plane is denoted as... The beamwidth is As one of the important controllable physical parameters of the system, the downlink channel between the AP and the user in the CF-mMIMO system is composed of large-scale fading and small-scale fading. The large-scale fading includes path loss, shadowing fading and antenna directivity gain, while the small-scale fading adopts the Rayleigh fading model.

[0039] Based on the above system framework, the execution subject of the embodiments of the present invention can be a cellless network user resource allocation device based on CVaR and reinforcement learning, or it can be the above system, or it can be the above central processing unit CPU, or it can be an electronic device in the CPU, or it can be other devices and equipment. No specific limitation is made here. The following embodiments will be described using the above system as the execution subject as an example.

[0040] Figure 2 This is a flowchart illustrating the cell-free network user resource allocation method based on CVaR and reinforcement learning provided by the present invention, as shown below. Figure 2 As shown, the method includes the following steps: Step 202: Obtain the state information of the current time step of the cellless network, and input the state information of the current time step into the reinforcement learning network for action generation to determine the action information of the current time step; the aforementioned cellless network includes multiple distributed access points (APs) and multiple users communicating with each AP.

[0041] Step 204: Allocate resources to each user based on the action information of the current time step, and move to the next time step.

[0042] The cellless network includes M (M>1) geographically distributed access points, denoted as M={1,...,M}, and K (K>1) users (i.e., user equipment UEs) that communicate with each AP, denoted as K={1,...,K}.

[0043] In this embodiment, a reinforcement learning network is used to generate action information for the current time step based on the state information of the cellless network at the current time step. This reinforcement learning network can be, for example, a Deep Reinforcement Learning (DRL) network. The interval between each time step can be set according to the actual situation and is not specifically limited here. The reinforcement learning network adopts a centralized training and distributed execution architecture, including a value network and a policy network. The cellless network system includes multiple agents (i.e., each AP), and each agent shares the policy network. During training, the value network receives the global state information of the cellless network system for policy evaluation; the global state information is the state information observed by all APs. During action generation, each agent receives its corresponding local observation information through the policy network and outputs its action information for the corresponding time step.

[0044] Optionally, the action information for the current time step includes the antenna downtilt angle of each AP, the user preference information of each AP, and the power control information of each AP. The user preference information of each AP at the current time step is used to characterize the overall preference weight of each AP for each user when allocating resources at the current time step. In other words, the action information for the current time step includes the action information of each AP at the current time step, and the action information of each AP at the current time step includes the antenna downtilt angle of that AP at the current time step, the user preference information of that AP at the current time step, and the power control information of that AP. The antenna downtilt angle of the AP at the current time step reflects the antenna gain of the AP at the current time step. The user preference information at the current time step refers to the user preference information of each AP at the current time step. This user preference information for each AP at the current time step reflects the preference weight of each subcarrier allocated to each user at the current time step. For example, a subcarrier of this AP might have a preference weight of 50% allocated to user A, and a subcarrier of this AP might have a preference weight of 80% allocated to user B. The power control information of each AP at the current time step characterizes the transmit power allocated by each AP to each subcarrier at the current time step. These three parts of action information can be used to preliminarily determine the subcarrier allocation situation of the APs to users. Of course, the action information at the current time step can also include other information, such as the number of subcarriers allocated to each user at the current time step.

[0045] Furthermore, to improve the average user rate in cellless massive multiple-input multiple-output (CF-mMIMO) systems / cellless network systems while suppressing performance degradation of tail users, this embodiment introduces a risk model based on Conditional Value at Risk (CVaR) constraints. It also proposes a target risk-aware advantage function based on CVaR constraints to train the reinforcement learning network (also known as the policy gradient deep reinforcement learning algorithm), abbreviated as the CVaR-MAPPO algorithm. This policy gradient deep reinforcement learning algorithm employs an actor-critic framework: a centralized value network uses the global state to evaluate the long-term reward of the joint policy, while multiple agent-specific policy networks output each agent's actions based on their local observations. Simultaneously, by modeling the CVaR constraint penalty term for tail user communication rates, actions that lead to poor tail user performance are explicitly penalized during policy updates. In the CVaR-MAPPO algorithm, once tail users satisfy the fairness constraint, the reinforcement learning network / policy gradient deep reinforcement learning algorithm automatically shifts its training focus to maximizing the system's user communication rate, thereby achieving a dynamic balance between fairness and overall system efficiency.

[0046] Specifically, the aforementioned reinforcement learning network is trained based on a target risk perception advantage function constrained by Conditional Value at Risk (CVaR). This target risk perception advantage function includes a penalty term that explicitly imposes fairness constraints on the communication rate of the tail users among all users at the current time step. The communication rate of each user at the current time step is determined by the resources allocated to the corresponding user at the current time step. During the training of the reinforcement learning network, the communication rate of each user at each time step can be collected (specifically, the user's experience rate, i.e., the sum of the communication rates obtained by the user after each AP allocates subcarrier resources to the user). The communication rate of each user at each time step is determined by the resources (such as subcarriers) allocated to the user by the AP at each time step, and the resources allocated by the AP to each user at each time step are determined by the action information of the AP at each time step output by the reinforcement learning network. After obtaining the communication rate of each user at each time step, the tail users can be identified from the users at the current time step. The tail users refer to the users whose communication rates are at the bottom in the order of high to low. That is, the tail users are users with low communication rates. These tail users are allocated relatively few resources, while users with high communication rates are allocated more resources. Therefore, the allocation of resources among users is unfair. Based on this, this embodiment employs CVaR constraints to construct a penalty term that explicitly constrains the communication rate of the tail users among all users at the current time step, thus ensuring fairness. This penalty term constrains the reinforcement learning network to output action information that benefits the communication rate of tail users, i.e., it outputs action information that allocates more resources to tail users, thereby increasing the resources allocated to them and improving the fairness of resource allocation among all users. Then, based on this penalty term and the initial risk-aware advantage function, a target risk-aware advantage function constrained by CVaR can be constructed. This target risk-aware advantage function is then used to construct a target expectation function, and maximizing the value of the target expectation function is used as the objective to train the policy network in the reinforcement learning network, resulting in a trained policy network, i.e., a trained reinforcement learning network. The initial risk-aware advantage function can be calculated based on the basic reward function and the evaluation results of the value network. The basic reward function can be represented by the average communication rate of each user in a cellless network.

[0047] After training the reinforcement learning network, it can generate action information for the cellless network at the current time step based on the network's state information. This action information then guides each agent (i.e., each access point) to allocate resources to users. For example, a specific subcarrier from one AP might be allocated to user A, thus allocating resources to each user at the current time step. After resource allocation, each user communicates in the wireless communication environment based on their allocated resources, obtaining their communication rate at the current time step. They can then move to the next time step, obtain its state information, generate action information, and allocate resources again based on this information, achieving dynamic, real-time, and fair resource allocation for users.

[0048] The status information for the current time step may include: the communication rate of each user in the previous time step (which may be the experience rate of all users in the system), the action information of the previous time step, the location of each user in the previous time step, and the channel power of the current time step.

[0049] It is understandable that when training a reinforcement learning network using a target risk-aware advantage function based on CVaR constraints, this target risk-aware advantage function can include a penalty term that constrains the low communication rate of tail users caused by the action information output by the reinforcement learning network. This encourages the reinforcement learning network to output action information that increases the communication rate of tail users as much as possible. Increasing the communication rate of tail users is achieved by increasing the resources allocated to tail users, thereby encouraging more resources to be allocated to tail users, ensuring that tail users receive as many resources as possible, narrowing the gap in resource allocation between them and other users, and thus ensuring that the resources allocated to each user are as fair as possible at the system level.

[0050] In this embodiment, the state information of the current time step of the cellless network is obtained and input into the reinforcement learning network for action generation. The action information of the current time step is determined, and resources are allocated to each user according to the action information of the current time step, and the process is transferred to the next time step. The cellless network includes multiple APs and multiple users communicating with each AP. The reinforcement learning network is trained based on a target risk perception advantage function with CVaR constraints. The target risk perception advantage function includes a penalty term that applies explicit fairness constraints to the communication rate of the tail users among the users in the current time step. The communication rate of each user in the current time step is determined by the resources allocated to the corresponding user in the current time step. In this method, a penalty term based on CVaR constraints is introduced during reinforcement learning network training to explicitly constrain the communication rate of tail users. This constrains the communication rate of tail users with excessively low communication rates in the cellless network during dynamic resource allocation, ensuring that their communication rates are not too low. Since the communication rate is determined by the resources allocated to the user based on the AP, it is equivalent to constraining the tail users to receive a sufficient amount of resources. This ensures fair allocation of resources among users in the cellless network, thereby achieving risk avoidance control and ensuring that the service quality of tail users in the cellless network is effectively improved even when channel uncertainty exists.

[0051] The above embodiments briefly illustrate the construction process of the target risk perception advantage function. The following embodiments will explain the specific construction process of the target risk perception advantage function.

[0052] In one embodiment, the above-mentioned target risk perception advantage function is constructed as follows: Step A1: Obtain the communication rate of all users in the cellless network at the current time step; all users in the cellless network include tail users.

[0053] Among them, see Figure 3 The diagram shown illustrates the specific process of the cellless network user resource allocation method based on CVaR and reinforcement learning provided by this invention. The reinforcement learning network can generate action information for the current time step based on the state information of the current time step. This action information can include action information generated for each AP at the current time step, for example, denoted as a1, a2, ..., a M The action information for each AP at the current time step may include the antenna downtilt angle of that AP at the current time step. User preference information of the AP at the current time step (These represent the allocation of the AP's subcarriers to each user at the current time step) and the AP's power control information at the current time step. (These represent the transmit power allocated to each subcarrier by the AP at the current time step).

[0054] Afterwards, each AP can allocate resources (i.e., subcarriers) to each user based on the action information of the current time step, thus feeding back the resource allocation status to the communication environment. After receiving the allocated resources, the communication rate of each user in the communication environment will change, resulting in the communication rate (i.e., user rate) of each user at the current time step. The users at the end of the ranking, corresponding to the communication rates at the bottom, are the tail users of the current time step. In other words, the tail users refer to those with the lowest communication rates at the current time step. The proportion of tail users to all users can be set according to actual conditions; for example, setting it to 5% would mean that the users with the lowest communication rates in the last 5% of the ranking results are the tail users.

[0055] Step A2: Obtain the learnable communication rate threshold.

[0056] In this step, a learnable communication rate threshold can be set. This threshold is used to limit the user's communication rate from being too low. The specific value can be set according to the actual situation and can be denoted as [insert value here]. .

[0057] Step A3: Construct a penalty term based on the communication rate threshold and the rate difference between the communication rates of each user at the current time step; the penalty term will reduce the value of the target risk perception advantage function when the communication rate of the tail users is lower than the communication rate threshold.

[0058] In this step, after obtaining the communication rate threshold and the communication rate of each user at the current time step, the rate difference between the communication rate threshold and the communication rate of each user at the current time step can be calculated to obtain the rate difference corresponding to each user at the current time step. Then, a penalty term for the target risk perception advantage function is constructed based on each rate difference. These rate differences include values ​​greater than 0 and values ​​less than 0. Values ​​greater than 0 indicate that the corresponding user's communication rate at the current time step is less than the communication rate threshold, that is, the user's communication rate at the current time step is too low. This means that these users are allocated fewer resources at the current time step. Therefore, to ensure the fairness of resource allocation among users, the communication rate and the difference between the communication rates of these users can be constrained to reduce the value of the target risk perception advantage function, thereby penalizing the reinforcement learning network for actions that result in these users being allocated fewer resources. Conversely, values ​​less than 0 indicate that the corresponding user's communication rate at the current time step is greater than the communication rate threshold, that is, the user's communication rate at the current time step is relatively high. In this case, the system does not need to impose a penalty on the reinforcement learning network.

[0059] Optionally, the penalty term constructed in this step based on the communication rate threshold and the rate difference between the communication rates of each user at the current time step may include: Calculate the communication rate threshold and the rate difference between the communication rate of each user at the current time step; Obtain the percentage of tail users and the number of each user; the percentage mentioned above is the proportion of tail users in the total number of users. Penalty terms are constructed based on the rate difference, quantity ratio, and number of users for each user.

[0060] After calculating the rate difference between each user at the current time step, the preset proportion of the last few users out of all users can also be obtained, i.e., the percentage of users, such as the bottom 5%, denoted as . Simultaneously, the number of users at the current time step can be obtained, denoted as K. Then, a penalty term can be constructed based on the rate difference of each user at the current time step, the proportion of tail users, and the number of each user. For example, the rate differences of each user can be processed, summed, and the sum divided by the product of the proportion and the total number of users to obtain the penalty term.

[0061] Optionally, the penalty term is constructed based on the rate difference, the proportion of users, and the number of users, including: Constraints are applied to each rate difference to determine the rate constraint result for each user; the rate constraint result includes taking 0 when the rate difference is less than 0 or taking the rate difference when the rate difference is greater than 0. Based on the rate constraint results, the proportion of each user, and the number of each user, a penalty term is constructed.

[0062] After calculating the rate difference for each user at the current time step, including rate differences greater than 0 and rate differences less than 0, a ReLU activation function can be used to constrain each rate difference. Specifically, the rate difference can be set to 0 when it is less than 0 or taken as the rate difference when it is greater than 0, thus obtaining the rate constraint result for each user's rate difference at the current time step (i.e., the user rate after the fairness assessment in the diagram). Then, the rate constraint results for all users at the current time step can be summed to obtain the sum of the rate differences.

[0063] Simultaneously, the proportion of tail users can be obtained. The Lagrange multiplier is divided by the product of this proportion of tail users and the total number of all users in the system. The resulting ratio is multiplied by the above-mentioned rate difference sum to obtain the penalty term.

[0064] Step A4: Construct the target risk perception advantage function based on the penalty term.

[0065] In this step, after obtaining the penalty term, the inverse of the penalty term can be used as the target risk perception advantage function; alternatively, the target risk perception advantage function can be constructed by combining it with the original risk perception advantage function. In the latter case, optionally, constructing the target risk perception advantage function based on the penalty term includes: obtaining the initial risk perception advantage function; and subtracting the initial risk perception advantage function from the penalty term to obtain the target risk perception advantage function. The initial risk perception advantage function can be the original risk perception advantage function, which can be calculated based on the evaluation results of the state information by the value network (Critic) and the basic reward function (e.g., using generalized advantage estimation (GAE)). The basic reward function can be represented by the average communication rate of each user in a cellless network. Then, the penalty term can be subtracted from the initial risk perception advantage function to construct the target risk perception advantage function. The constructed target risk perception advantage function can be expressed as follows: ; in, This represents the penalty / risk term constructed based on CVaR constraints. This penalty term based on CVaR constraints can evaluate the fairness of communication rate / resource allocation for each user. By directly subtracting the penalty / risk term from the initial risk perception advantage function, a new target risk perception advantage function is constructed.

[0066] This represents the constructed target risk perception advantage function. This represents the initial risk perception advantage function. This indicates the percentage of users at the bottom of the leaderboard. Denotes Lagrange multipliers, greater than or equal to 0. Indicates the communication rate threshold. This represents the communication rate of the k-th user at the current time step t, where K represents the total number of users. This indicates the constraint handling for the ReLU activation function.

[0067] As can be seen from the above formula, when the communication rate of the tail users is significantly lower than the communication rate threshold at a certain time step, the value in the summation sign is a positive rate difference. This increases the corresponding penalty term based on CVaR constraints, significantly reducing the value of the target risk-aware advantage function, and potentially even making it negative. Consequently, actions detrimental to resource allocation for tail users are penalized in the gradient update of the policy network in the reinforcement learning network. Conversely, when the communication rates of all users are higher than the communication rate threshold, the value in the summation sign is 0. The corresponding penalty term based on CVaR constraints disappears, and the target risk-aware advantage function degenerates into the initial risk-aware advantage function.

[0068] Alternatively, the aforementioned communication rate threshold and Lagrange multipliers All of these are learnable parameters. During the training of the reinforcement learning network, a loss function containing the communication rate threshold and Lagrange multipliers is constructed, and gradient descent is used to jointly optimize and update it to adaptively adjust the penalty strength of the fairness constraint.

[0069] Furthermore, during the training of the policy network of the reinforcement learning network using the aforementioned target risk-aware advantage function, i.e., during the gradient update process of the policy network, the target risk-aware advantage function can be substituted into the truncated policy gradient objective function to obtain the action optimization objective function based on CVaR constraints, i.e., the policy gradient objective function / target expectation function. The reinforcement learning network can then be trained using this policy gradient objective function. Optionally, the construction method of this policy gradient objective function includes: Based on the action information at the current time step, calculate the product of the probability ratio of the new and old policies and the target risk perception advantage function, and combine the product after pruning and restriction to construct the policy gradient objective function. Then, use the policy gradient objective function to train the reinforcement learning network.

[0070] Specifically, after obtaining the action information at the current time step, for the same set of state and action information, the ratio of the probability of outputting the action information using the updated policy network (new policy) to the probability of outputting the action information using the unupdated policy network (old policy) is calculated. This ratio is used as the probability ratio of the new and old policies at the current time step, measuring the range of probability change before and after the policy update. Then, the clipping function can be used to clip this probability ratio, limiting it to [1-]. ,1+ Within the range of [1-], the cutting is then limited to [1- ,1+ The product of the probability ratio within the range and the target risk perception advantage function is used to obtain the product term after pruning restrictions.

[0071] Simultaneously, the probability ratio of the new and old strategies can be multiplied by the target risk perception advantage function to obtain the product term. Then, a function that minimizes this product term and the product term after pruning and constraints can be constructed, and its expected value can be taken to construct the policy gradient objective function. The specific constructed policy gradient objective function can be seen in the following formula: ; in, This represents the objective function for constructing the policy gradient; The expression represents the expected value; min() represents the function that takes the minimum value to prevent the policy update from being too large and to ensure the stability of training. The probability ratio can be expressed as: , This represents a new policy in the policy network. Indicates the old strategy; This represents the pruning hyperparameter, typically taking a value of 0.1 or 0.2, used to control the magnitude of policy updates; Represents the product term; This represents the product term after trimming and restriction.

[0072] During training, the reinforcement learning network can be trained using the policy gradient objective function, while the initial risk-aware advantage function can be replaced with the target risk-aware advantage function for policy gradient updates. The specific policy gradient update process is shown in the following formula: ; in, The parameters represent the policy network of the reinforcement learning network; the left arrow indicates an assignment or update operation; lr represents the learning rate; Indicates the expectation about the time step; Indicates about The gradient; a tThis indicates the action information executed at time step t; o t Represents the state information at time step t, | represents the conditional probability, and log represents taking the natural logarithm.

[0073] As can be seen from the above policy gradient update process, this update direction is equivalent to applying explicit penalties to action trajectories that cause the tail user communication rate to be too low while maximizing the expected return. This enables the reinforcement learning network to have risk perception capabilities and enhances system fairness.

[0074] In this embodiment, a penalty term is constructed using the difference between the communication rate of each user at the current time step and a communication rate threshold. This term reduces the value of the target risk-aware advantage function when the communication rate of tail users falls below the threshold. This adds an explicit penalty to the unfavorable actions of tail users with excessively low communication rates during the reinforcement learning network training process, making the allocation of communication resources among users more equitable. Furthermore, by incorporating the proportion of tail users and the number of each user when constructing the penalty term, and by constraining the rate difference between users, the constructed penalty term becomes more accurate. Additionally, the target risk-aware advantage function is constructed by subtracting the initial risk-aware advantage function from the penalty term, which explicitly penalizes actions unfavorable to tail users during the policy gradient update of the reinforcement learning network. Moreover, by pruning and limiting the product term of the target risk-aware advantage function and taking the minimum value before and after pruning, excessive policy updates in the reinforcement learning network are prevented, ensuring training stability.

[0075] The above embodiments briefly illustrate the process of determining the resource allocation strategy for the current time step based on the action information of the current time step. The following embodiments will describe the process in detail based on the specific action information of the current time step.

[0076] In one embodiment, the resource allocation to each user based on the action information of the current time step in step 204 above may include: Step B1: Determine the channel quality of each AP on any subcarrier based on the antenna downtilt angle of each AP at the current time step and the power control information of each AP at the current time step.

[0077] The action information for the current time step includes the antenna downtilt angle of each AP, the user preference information of each AP, and the power control information of each AP. The user preference information of each AP in the current time step is used to characterize the overall preference weight of each AP for each user when allocating resources in the current time step. In other words, the action information for the current time step includes the action information of each AP in the current time step, and the action information of each AP in the current time step includes the antenna downtilt angle of the AP in the current time step, the user preference information of the AP in the current time step, and the power control information of the AP.

[0078] Specifically, after obtaining the antenna downtilt angle and power control information of each AP at the current time step, the channel quality of each AP on any subcarrier at the current time step can be calculated in the communication environment by combining the AP's position at the current time step and the positions of each user at the current time step. This channel quality can be represented by antenna gain, instantaneous channel power, etc.

[0079] Taking channel quality as an example, which can be represented by antenna gain and instantaneous channel power, optionally, this step can determine the channel quality of each AP on any subcarrier based on the antenna downtilt angle of each AP at the current time step and the power control information of each AP at the current time step. This can include: Based on the antenna downtilt angle of each AP at the current time step, determine the antenna gain of each AP at the corresponding antenna downtilt angle and pointing towards any user; Based on the location information of each user, the location information of each AP, and the characteristics of the wireless environment, determine the instantaneous channel power from each AP to any user on any subcarrier. The channel quality of each AP on any subcarrier is determined by the antenna gain of each AP at the corresponding antenna downtilt angle and pointing to any user, and the instantaneous channel power from each AP to any user on any subcarrier.

[0080] After obtaining the antenna downtilt angle of each AP at the current time step, the antenna gain of the AP pointing to any user at its corresponding antenna downtilt angle can be calculated within the communication environment, taking into account the AP's position at the current time step and the positions of each user. Simultaneously, the instantaneous channel power from the AP to any user on any subcarrier can be calculated within the communication environment, taking into account the AP's position at the current time step, the positions of each user at the current time step, and the wireless environment characteristics (i.e., the environmental characteristics corresponding to the communication environment). Then, the antenna gain and instantaneous channel power of each AP are determined as the channel quality of that AP on the subcarrier.

[0081] Step B2: Based on the user preference information of each AP at the current time step and the channel quality of each AP on any subcarrier, determine the association score corresponding to each user allocated on each subcarrier of each AP; the higher the association score, the higher the priority of the user to obtain resources, and the association score of each user is inversely proportional to the resources already allocated to the user.

[0082] In this step, after obtaining the channel quality of each AP on any subcarrier at the current time step and the user preference information of each AP at the current time step, where the user preference information of each AP at the current time step refers to the preference weights of each subcarrier of the AP assigned to each user at the current time step, the user preference information of each AP at the current time step can be multiplied with the antenna gain of each AP at the corresponding antenna downtilt angle and pointing to any user, and the instantaneous channel power from each AP to any user on any subcarrier to determine the association score corresponding to each user assigned on each subcarrier of each AP. That is, for any AP and any subcarrier, the association score calculated for each user when any subcarrier of any AP is assigned to each user.

[0083] It should be noted that, for any AP and any subcarrier, after calculating the correlation score for each user, the user with the higher correlation score has a higher priority in obtaining the resources of that AP and that subcarrier. Furthermore, each user's correlation score is inversely proportional to their already allocated resources. That is, the more resources a user has already allocated at the current time step, the lower their correlation score will be at the current time step, thus reducing their priority in resource allocation at the current time step. Conversely, the fewer resources a user has already allocated at the current time step, the higher their correlation score will be at the current time step, thus increasing their priority in resource allocation at the current time step. This ensures that more resources are allocated to them at the current time step, guaranteeing fairness in resource allocation among users within the system.

[0084] In addition to calculating the association score using the channel quality of each AP on any subcarrier at the current time step and the user preference information of each AP at the current time step, the above can also combine other information to comprehensively calculate the association score. Optionally, the above determines the association score assigned to each user on each subcarrier of each AP based on the user preference information of each AP at the current time step and the channel quality of each AP on any subcarrier, including: Get the number of subcarriers that each AP has allocated to any user at the current time step; Based on the number of subcarriers that each AP has allocated to any user at the current time step, determine the dynamic load attenuation factor between each AP and the user with allocated subcarriers; among them, the more subcarriers a user has been allocated, the lower the association score. Based on the user preference information of each AP at the current time step, the channel quality of each AP on any subcarrier, and the dynamic load attenuation factor between each AP and the users on the allocated subcarriers, the association score corresponding to each user allocated on each subcarrier of each AP is determined.

[0085] Specifically, by statistically analyzing the number of subcarriers allocated to each user by each AP in the communication environment at the current time step and previous time steps, the number of subcarriers allocated to each user at the current time step can be obtained; that is, the number of subcarriers allocated to any user by any AP. Simultaneously, the coefficient controlling the attenuation strength can be obtained. Multiplying this coefficient by the number of subcarriers allocated to each user at the current time step, summing the product with 1, and taking the reciprocal of the sum, yields the dynamic load attenuation factor between each AP and the users with allocated subcarriers. The formula for calculating this dynamic load attenuation factor between each AP and the users with allocated subcarriers is as follows: ; in, D k,m This represents the dynamic load decay factor between the m-th AP and the k-th user; N k,m This represents the number of subcarriers that the m-th AP has allocated to the k-th user; β represents a coefficient that controls the attenuation intensity, which is greater than 1, for example, it can be 10.

[0086] Then, the user preference information of each AP at the current time step, the antenna gain of each AP at the corresponding antenna downtilt angle and pointing to any user, the instantaneous channel power from each AP to any user on any subcarrier, and the dynamic load attenuation factor between each AP and the users on the allocated subcarriers can be multiplied to determine the association score corresponding to each user allocated on each subcarrier of each AP. The specific calculation formula is as follows: ; in, q k,m,n This represents the association score calculated for the k-th user when the m-th AP allocates the n-th subcarrier to that user; weight k,m This represents the preference weight of the m-th AP for the k-th user, which is the preference weight of the k-th user in the user preference information of the m-th AP. It can reflect the probability that the m-th AP selects k users to allocate subcarriers. This indicates the downtilt angle of the m-th AP at the antenna. Antenna gain pointing down to the k-th user; This represents the instantaneous channel power from the m-th AP to the k-th user on subcarrier n; D k,mThis represents the dynamic load decay factor between the m-th AP and the k-th user.

[0087] In addition, before calculating the associated score using the action information of the current time step, the information in the action information of the current time step can be normalized and preprocessed to ensure that the information is reasonable in a physical sense.

[0088] As can be seen from the above correlation score calculation formula, the more subcarriers a user has been allocated, the lower their dynamic load attenuation factor, and the lower their corresponding correlation score. Consequently, the priority of subcarrier resources allocated to them in the current time step is lower. On the other hand, the fewer subcarriers a user has been allocated, the higher their dynamic load attenuation factor, and the higher their corresponding correlation score. Consequently, the priority of subcarrier resources allocated to them in the current time step is higher. This allows more resources to be allocated to tail users, improving the fairness of resource allocation within the system.

[0089] Step B3: Based on the association score of each user allocated on each subcarrier of each AP, determine the subcarriers of the APs allocated to each user at the current time step and perform subcarrier allocation.

[0090] In this step, after calculating the association score for each user assigned to each subcarrier of each AP, a user can be selected from the multiple users corresponding to each subcarrier of each AP and assigned that subcarrier of the AP to that user. This obtains the resource allocation of the AP on that subcarrier. In this way, the resource allocation of all APs on all subcarriers can be obtained, and subcarrier allocation can be performed quickly accordingly.

[0091] Optionally, this step determines the subcarriers of the APs allocated to each user at the current time step based on the association score of each user allocated on each subcarrier of each AP, including: For each subcarrier of each AP, determine the maximum association score based on the association score of each user allocated on the AP's subcarrier, and allocate the AP's subcarrier to the user corresponding to the maximum association score at the current time step.

[0092] Specifically, for each subcarrier of each AP, an association score can be obtained when allocating that subcarrier of the AP to each user. Then, the user with the highest association score is selected, and the user corresponding to the highest association score is determined as the user to be allocated to that subcarrier of the AP at the current time step. In other words, the optimal user corresponding to the highest association score for allocating that subcarrier of the AP at the current time step can be represented by the following formula: ; in, km,n This represents the target / optimal user allocated by the m-th AP on the n-th subcarrier; This represents the operation of selecting the user with the highest association score from among K users when the m-th AP allocates the n-th subcarrier to the k-th user.

[0093] It can be seen that the selection of the optimal user is jointly determined by the AP's user preference information at the current time step, the AP's channel quality at the current time step, and the AP's load attenuation factor at the current time step. Once a user is selected, the subcarrier count / number allocated to it will be updated by incrementing by one. The update process is represented as follows: ; Through the above subcarrier counting / quantity update process, the number of subcarriers allocated to the selected user at the current time step will increase, thereby reducing its dynamic load attenuation factor and consequently its association score. This will lower the user's priority in resource allocation in subsequent policy network decisions and increase the chances of other users obtaining resources, thus achieving basic fairness adjustment in the system.

[0094] In this embodiment, the channel quality of each AP on the subcarrier is calculated using the antenna downtilt angle and power control information of each AP at the current time step. Based on the channel quality, the association score of each user's allocated subcarrier is calculated, and the subcarriers allocated to each user are determined accordingly. Users with higher association scores have higher priority in obtaining resources. Each user's association score is inversely proportional to the resources already allocated to that user. This allows for the selection of the most suitable user for each AP on each subcarrier, improving fairness at the system architecture level. Furthermore, channel quality is characterized by calculating the antenna gain and instantaneous channel power of each AP at the current time step. This multi-dimensional information characterization of channel quality makes the association score calculated based on channel quality more accurate. Moreover, the association score for each user is calculated in conjunction with the number of subcarriers currently allocated to each user. This reduces the priority of users with excessively allocated resources in subsequent policy network decisions, increasing the chances of other users obtaining resources and achieving basic fairness adjustment within the system. In addition, by allocating each subcarrier of each AP to the user with the highest association score in the association score, it can be ensured that the user with the higher association score has a higher priority to obtain resources, so as to achieve fair allocation of resources within the system as soon as possible.

[0095] In summary, this invention proposes the CVaR-MAPPO algorithm, which aims to improve system user rates under fairness constraints by constructing a policy optimization guided by CVaR constraints. The overall workflow of the CVaR-MAPPO algorithm is summarized as follows: 1. First, the connection between the distributed access point (AP) and the user equipment (UE) is optimized using connection rules that ensure basic fairness, thus laying a structural foundation for subsequent fairness enhancements at the system level; 2. Subsequently, CVaR constraints are incorporated into the dynamic resource allocation process to achieve risk avoidance control, ensuring that the service quality of tail users is effectively improved even in the presence of channel uncertainty; 3. Furthermore, CVaR constraints are used to guide gradient information during the reinforcement learning network training process, endowing the reinforcement learning network with risk perception capabilities and constructing a fully risk-sensitive training framework, thereby improving the fairness performance of the learned decision-making policies.

[0096] Based on the technical solutions described in the above embodiments, a technical solution evaluation was conducted in city A. (See attached document.) Figure 4 The diagram shows a simulation of population and device distribution in two real-world scenarios in City A provided by this invention. Two real-world scenarios in City A were selected during the evaluation process, taking into account the actual building distribution, road structure, and population distribution information. The AP density and population density in scenario 1 (left diagram) are both higher than those in scenario 2 (right diagram). In each experimental real-world scenario, the system deploys one central processing unit (CPU) and M distributed access points (APs) to provide communication services to K users. Table 1 below shows the experimental results of the CVaR-MAPPO algorithm of this invention and various baseline methods. This invention mainly considers two performance indicators: tail user rate and system user rate, where higher values ​​indicate better performance. This indicates the performance improvement of the CVaR-MAPPO algorithm in this embodiment of the invention compared to other baseline methods. This indicates an average performance improvement.

[0097] Table 1 The experimental results above demonstrate that the CVaR-MAPPO algorithm of this invention (i.e., the experimental results in the "ours" row of the table above) achieves excellent performance in both real-world scenarios and significantly outperforms all baseline methods in terms of system fairness. MAPPO, a typical on-policy reinforcement learning algorithm, is trained by collecting data online through real-time interaction with the environment, exhibiting good performance in optimizing the system's average rate. However, because its primary goal is to improve system throughput without explicitly considering fairness constraints, MAPPO obtains extremely low tail user rates, resulting in a severe decrease in system fairness. MADDPG and MATD3 are typical off-policy reinforcement learning algorithms. Limited by relatively low training data quality, their convergence speed is slow, and they do not show significant advantages in either system rate or fairness. The genetic algorithm GA performs poorly on the highly non-convex and complex optimization problem considered in this invention, reflecting its inherent limitations in such scenarios. The Greedy method employs short-sighted rules to maximize system rate, thus exhibiting the lowest tail user rate, indicating that it sacrifices fairness for improved throughput performance. In contrast, the method proposed in this embodiment of the invention introduces a risk perception mechanism based on CVaR constraints, enabling the reinforcement learning network to learn risk sensitivity and actively maintain user fairness during training. Therefore, it achieves the best system fairness performance among all methods in two real-world scenarios, while still maintaining competitive system speed performance.

[0098] Furthermore, the convergence of the reinforcement learning network in the training process of this embodiment of the invention was analyzed, see [link to relevant documentation]. Figure 5 The diagram shown illustrates the learning curve of the reinforcement learning network provided by this invention during the training process. The reinforcement learning network being trained is the aforementioned CVaR-MAPPO algorithm. The left graph represents the system user rate, and the right graph represents the tail user rate. It can be seen that the reinforcement learning network reaches a learning curve of approximately 8 × 10⁻⁶. 5The system converges to a stable performance within a few training steps. In the early stages of training, the system user rate curve first decreases and then increases, while the tail user rate curve first increases and then decreases. This phenomenon indicates that under the CVaR constraint, the reinforcement learning network is forced to prioritize tail user rates in the early stages of training, thereby enhancing system fairness. In improving tail performance, the CVaR-MAPPO algorithm tends to allocate more network resources to weaker users, which reduces the overall system rate in the short term—behavior highly consistent with the theoretical expectation of the present invention. When the tail user rate reaches the minimum acceptable threshold, the reinforcement learning network adaptively shifts its training focus to maximizing the system rate. Therefore, as the system user rate increases, the tail user rate gradually decreases and eventually stabilizes near the target threshold, which also conforms to the theoretical analysis results. In the CVaR-MAPPO algorithm, the tail user rates eventually stabilize at 61.6 and 131.12, respectively. In contrast, as shown in Table 1 above, the tail performance of other baseline methods deteriorates further, resulting in poorer system fairness. Overall, the training curves of the reinforcement learning network in this embodiment clearly demonstrate that the CVaR-MAPPO algorithm in this embodiment first prioritizes the user experience of the tail user under fairness constraints, and then optimizes the system speed on this basis, which is consistent with the original design intention.

[0099] Furthermore, the fairness of resource allocation for each user in the reinforcement learning network of this invention embodiment was analyzed; see [link to relevant documentation]. Figure 6The diagram illustrates the impact of decisions made by the CVaR-MAPPO algorithm and the MAPPO algorithm provided by this invention on the distribution of user rates in the system. The CVaR-MAPPO algorithm is the reinforcement learning network trained based on explicit fairness constraints of CVaR in this embodiment of the invention (i.e., Ours in the diagram), while the MAPPO algorithm is a baseline algorithm that does not explicitly consider fairness. It can be seen that in the tail region (0%–5%), the CVaR-MAPPO algorithm of this embodiment of the invention achieves a significantly higher rate for the worst user than the MAPPO algorithm. Furthermore, in the distribution intervals of 5%–20%, 20%–40%, and 40%–60%, the user rates obtained by the CVaR-MAPPO algorithm of this embodiment of the invention are all higher than or at least not lower than the results of the MAPPO algorithm. This indicates that the CVaR-MAPPO algorithm of this embodiment of the invention imposes a strong constraint on the rate of tail users, and this constraint gradually propagates to users in the 5%–60% quantile interval, thereby improving the performance of relatively low-rate users to varying degrees. In contrast, within the 60%–100% quantile range, the MAPPO algorithm primarily focuses on maximizing system rate, thus prioritizing the allocation of network resources to users with better channel conditions. This results in high-rate users receiving more resources and further exacerbates system unfairness. This phenomenon aligns with the theoretical design expectations of the embodiments of this invention and verifies that the CVaR-MAPPO algorithm of these embodiments effectively improves overall system fairness by increasing the rates of tail-end users and other low-rate users.

[0100] The above experimental analyses demonstrate that the present invention explicitly incorporates CVaR constraints into non-cellular user rate optimization to enhance user fairness, providing a controllable improvement in service quality for tail users while optimizing the average user rate. A CVaR-MAPPO algorithm is proposed, where each distributed access point (AP) is modeled as an independent agent making cooperative decisions based on user preference information, antenna gain, and channel conditions / quality. The CVaR of tail user rates is incorporated into the risk-aware advantage function through a Lagrange penalty term, enabling the network learning process to adaptively balance fairness constraints with overall system performance. Furthermore, theoretical analyses of algorithm complexity and convergence properties are provided. The proposed CVaR-MAPPO algorithm is evaluated in two real-world scenarios of different scales, including realistic geographical layouts and user mobility patterns. Compared to a representative baseline, the method in this invention improves the average user rate by 8.77% and the tail user rate by 80.31%, demonstrating the effectiveness of the proposed method in designing for user resource allocation fairness.

[0101] Additionally, it should be noted that, Figures 4-6 The text or numbers in this document are for illustrative purposes only and do not affect the technical essence of the present invention.

[0102] The following describes the cellless network user resource allocation device based on CVaR and reinforcement learning provided by the present invention. The cellless network user resource allocation device based on CVaR and reinforcement learning described below can be referred to in correspondence with the cellless network user resource allocation method based on CVaR and reinforcement learning described above.

[0103] Figure 7 This is a schematic diagram of the cellless network user resource allocation device based on CVaR and reinforcement learning provided by the present invention. See [link / reference]. Figure 7 As shown, the device may include: The action generation module 710 is used to obtain the state information of the current time step of the cellless network, and input the state information of the current time step into the reinforcement learning network for action generation to determine the action information of the current time step; the aforementioned cellless network includes multiple distributed access points (APs) and multiple users communicating with each AP; The user resource allocation module 720 is used to allocate resources to each user based on the action information of the current time step and move to the next time step; The aforementioned reinforcement learning network is trained based on a target risk perception advantage function constrained by Conditional Value at Risk (CVaR). This target risk perception advantage function includes a penalty term that explicitly imposes fairness constraints on the communication rate of the tail users in each user at the current time step. The communication rate of each user at the current time step is determined by the resources allocated to the corresponding user at the current time step.

[0104] In one embodiment, the above-mentioned apparatus further includes: The function construction module is used to obtain the communication rate of all users in the cellless network at the current time step; all users in the cellless network include tail users; obtain a learnable communication rate threshold; construct a penalty term based on the communication rate threshold and the rate difference between each user's communication rate at the current time step; the above penalty term will reduce the value of the target risk perception advantage function when the communication rate of tail users is lower than the communication rate threshold; construct the target risk perception advantage function based on the penalty term.

[0105] Optionally, the above function construction module is specifically used to calculate the rate difference between the communication rate threshold and the communication rate of each user at the current time step; obtain the proportion of tail users and the number of each user; the above proportion is the ratio of the number of tail users to the total number of users; and construct a penalty term based on the rate difference, the proportion, and the number of each user.

[0106] Optionally, the above function construction module is specifically used to perform constraint processing on each rate difference to determine the rate constraint result corresponding to each user; the rate constraint result includes taking 0 when the rate difference is less than 0 or taking the rate difference when the rate difference is greater than 0; and constructing a penalty term based on the rate constraint result corresponding to each user, the proportion of the number, and the number of each user.

[0107] Optionally, the above function construction module is specifically used to obtain the initial risk perception advantage function; and to obtain the target risk perception advantage function by subtracting the initial risk perception advantage function from the penalty term.

[0108] Optionally, the above function construction module is also used to calculate the product of the probability ratio of the new and old policies and the target risk perception advantage function based on the action information of the current time step, and to construct the policy gradient objective function by combining the product term after pruning and restriction, and to train the reinforcement learning network using the policy gradient objective function.

[0109] In one embodiment, the action information of the current time step includes the antenna downtilt angle of each AP in the current time step, the user preference information of each AP in the current time step, and the power control information of each AP in the current time step; the user preference information of each AP in the current time step is used to characterize the overall preference weight of each user when each AP allocates resources in the current time step.

[0110] Optionally, the user resource allocation module 720 is specifically used to determine the channel quality of each AP on any subcarrier based on the antenna downtilt angle and power control information of each AP at the current time step; to determine the association score corresponding to each user allocated on each subcarrier of each AP based on the user preference information and channel quality of each AP at the current time step; the user with the higher association score has a higher priority in obtaining resources, and the association score of each user is inversely proportional to the resources already allocated to the user; and to determine the subcarriers of the APs allocated to each user at the current time step and perform subcarrier allocation based on the association scores of each user allocated on each subcarrier of each AP.

[0111] Optionally, the user resource allocation module 720 is specifically used to determine the antenna gain of each AP pointing to any user at the corresponding antenna downtilt angle based on the antenna downtilt angle of each AP at the current time step; to determine the instantaneous channel power from each AP to any user on any subcarrier based on the location information of each user, the location information of each AP, and the wireless environment characteristics; and to determine the antenna gain of each AP pointing to any user at the corresponding antenna downtilt angle and the instantaneous channel power from each AP to any user on any subcarrier as the channel quality of each AP on any subcarrier.

[0112] Optionally, the user resource allocation module 720 is specifically used to obtain the number of subcarriers that each AP has allocated to any user at the current time step; determine the dynamic load attenuation factor between each AP and the users with allocated subcarriers based on the number of subcarriers that each AP has allocated to any user at the current time step; wherein, the more subcarriers a user has been allocated, the lower its association score; and determine the association score corresponding to each user allocated on each subcarrier of each AP based on the user preference information of each AP at the current time step, the channel quality of each AP on any subcarrier, and the dynamic load attenuation factor between each AP and the users with allocated subcarriers.

[0113] Optionally, the user resource allocation module 720 is specifically used to determine the maximum association score for each subcarrier of each AP based on the association score of each user allocated on the subcarrier of the AP, and to allocate the subcarrier of the AP to the user corresponding to the maximum association score at the current time step.

[0114] Figure 8 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 8 As shown, the electronic device may include: a processor 810, a communications interface 820, a memory 830, and a communications bus 840, wherein the processor 810, the communications interface 820, and the memory 830 communicate with each other through the communications bus 840. The processor 810 can call logic instructions in the memory 830 to execute a cellless network user resource allocation method based on CVaR and reinforcement learning. This method includes: acquiring the state information of the cellless network at the current time step, inputting the state information of the current time step into a reinforcement learning network for action generation, and determining the action information of the current time step; the cellless network includes multiple distributed access points (APs) and multiple users communicating with each AP; allocating resources to each user according to the action information of the current time step, and transferring to the next time step; wherein the reinforcement learning network is trained based on a target risk perception advantage function constrained by conditional value at risk (CVaR), the target risk perception advantage function including a penalty term for explicit fairness constraints on the communication rate of tail users in the current time step, and the communication rate of each user in the current time step is determined by the resources allocated to the corresponding user in the current time step.

[0115] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0116] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the CVaR-based and reinforcement learning-based user resource allocation method for cellless networks provided by the above methods. The method includes: acquiring the state information of the cellless network at the current time step, inputting the state information of the current time step into a reinforcement learning network for action generation, and determining the action information of the current time step; the cellless network includes multiple distributed access points (APs) and multiple users communicating with each AP; allocating resources to each user according to the action information of the current time step, and transferring to the next time step; wherein the reinforcement learning network is trained based on a target risk perception advantage function constrained by conditional value at risk (CVaR), the target risk perception advantage function including a penalty term for the communication rate of the tail users in each user at the current time step with explicit fairness constraints, and the communication rate of each user at the current time step is determined by the resources allocated to the corresponding user at the current time step.

[0117] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the CVaR-based and reinforcement learning-based user resource allocation method for cellless networks provided by the above methods. The method includes: acquiring the state information of the cellless network at the current time step, inputting the state information of the current time step into a reinforcement learning network for action generation, and determining the action information of the current time step; the cellless network includes multiple distributed access points (APs) and multiple users communicating with each AP; allocating resources to each user according to the action information of the current time step, and transferring to the next time step; wherein the reinforcement learning network is trained based on a target risk perception advantage function constrained by conditional value at risk (CVaR), the target risk perception advantage function including a penalty term for explicitly fairing the communication rate of the tail users among the users at the current time step, and the communication rate of each user at the current time step is determined by the resources allocated to the corresponding user at the current time step.

[0118] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0119] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0120] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A cell-free network user resource allocation method based on CVaR and reinforcement learning, characterized in that, The method includes: The state information of the current time step of the cellless network is obtained, and the state information of the current time step is input into the reinforcement learning network for action generation to determine the action information of the current time step; the cellless network includes multiple distributed access points (APs) and multiple users communicating with each AP; Based on the action information of the current time step, resources are allocated to each user, and the process is moved to the next time step; The reinforcement learning network is trained based on a target risk perception advantage function constrained by conditional value at risk (CVaR). The target risk perception advantage function includes a penalty term that applies explicit fairness constraints to the communication rate of the tail users among the users at the current time step. The communication rate of each user at the current time step is determined by the resources allocated to the corresponding user at the current time step.

2. The cellless network user resource allocation method based on CVaR and reinforcement learning according to claim 1, characterized in that, The construction methods for the target risk perception advantage function include: Obtain the communication rate of all users in the cellless network at the current time step; the users in the cellless network include the tail users; Obtain a learnable communication rate threshold; The penalty term is constructed based on the communication rate threshold and the rate difference between the communication rates of each user at the current time step; the penalty term reduces the value of the target risk perception advantage function when the communication rate of the tail users is lower than the communication rate threshold. Construct the target risk perception advantage function based on the penalty term.

3. The cellless network user resource allocation method based on CVaR and reinforcement learning according to claim 2, characterized in that, The step of constructing the penalty term based on the communication rate threshold and the rate difference between the communication rates of each user at the current time step includes: Calculate the rate difference between the communication rate threshold and the communication rate of each user at the current time step; Obtain the percentage of the number of the tail users and the number of each user; the percentage is the proportion of the number of tail users in the total number of users. The penalty term is constructed based on the rate difference corresponding to each user, the proportion of the number, and the number of each user.

4. The cellless network user resource allocation method based on CVaR and reinforcement learning according to any one of claims 2 to 3, characterized in that, The step of constructing the target risk perception advantage function based on the penalty term includes: Obtain the initial risk perception advantage function; The target risk perception advantage function is obtained by subtracting the initial risk perception advantage function from the penalty term.

5. The cellless network user resource allocation method based on CVaR and reinforcement learning according to any one of claims 1 to 3, characterized in that, The method further includes: Based on the action information at the current time step, the product of the probability ratio of the new and old policies and the target risk perception advantage function is calculated. Combined with the product after pruning and restriction, a policy gradient objective function is constructed, and the reinforcement learning network is trained using the policy gradient objective function.

6. The cellless network user resource allocation method based on CVaR and reinforcement learning according to any one of claims 1 to 3, characterized in that, The action information of the current time step includes the antenna downtilt angle of each AP in the current time step, the user preference information of each AP in the current time step, and the power control information of each AP in the current time step. The user preference information of each AP at the current time step is used to characterize the overall preference weight of each AP for each user when allocating resources at the current time step; The step of allocating resources to each user based on the action information at the current time step includes: Based on the antenna downtilt angle of each AP at the current time step and the power control information of each AP at the current time step, determine the channel quality of each AP on any subcarrier; Based on the user preference information of each AP at the current time step and the channel quality of each AP on any subcarrier, the association score corresponding to each user allocated on each subcarrier of each AP is determined; the higher the association score, the higher the priority of the user to obtain resources, and the association score of each user is inversely proportional to the resources allocated to the user; Based on the association score of each user assigned to each subcarrier of each AP, the subcarriers of the AP assigned to each user at the current time step are determined and subcarrier allocation is performed.

7. The cellless network user resource allocation method based on CVaR and reinforcement learning according to claim 6, characterized in that, The step of determining the channel quality of each AP on any subcarrier based on the antenna downtilt angle of each AP at the current time step and the power control information of each AP at the current time step includes: Based on the antenna downtilt angle of each AP at the current time step, determine the antenna gain of each AP at the corresponding antenna downtilt angle and pointing towards any user; Based on the location information of each user, the location information of each AP, and the wireless environment characteristics, determine the instantaneous channel power from each AP to any user on any subcarrier; The antenna gain of each AP at the corresponding antenna downtilt angle and pointing to any user, and the instantaneous channel power from each AP to any user on any subcarrier, are determined as the channel quality of each AP on any subcarrier.

8. The cellless network user resource allocation method based on CVaR and reinforcement learning according to claim 6, characterized in that, The step of determining the associated score for each user assigned on each subcarrier of each AP based on the user preference information of each AP at the current time step and the channel quality of each AP on any subcarrier includes: Obtain the number of subcarriers that each AP has allocated to any user at the current time step; Based on the number of subcarriers that each AP has allocated to any user at the current time step, determine the dynamic load attenuation factor between each AP and the user with allocated subcarriers; wherein, the more subcarriers a user has been allocated, the lower the association score. Based on the user preference information of each AP at the current time step, the channel quality of each AP on any subcarrier, and the dynamic load attenuation factor between each AP and the users on the allocated subcarriers, the association score corresponding to each user allocated on each subcarrier of each AP is determined.

9. The cellless network user resource allocation method based on CVaR and reinforcement learning according to claim 6, characterized in that, The step of determining the subcarriers of the APs allocated to each user at the current time step based on the association score of each user allocated on each subcarrier of each AP includes: For each subcarrier of each AP, the maximum association score is determined based on the association score of each user allocated on the subcarrier of the AP, and the subcarrier of the AP is allocated to the user corresponding to the maximum association score at the current time step.

10. A cell-free network user resource allocation device based on CVaR and reinforcement learning, characterized in that, The device includes: An action generation module is used to obtain the state information of the current time step of the cellless network, and input the state information of the current time step into the reinforcement learning network for action generation to determine the action information of the current time step; the cellless network includes multiple distributed access points (APs) and multiple users communicating with each AP; The user resource allocation module is used to allocate resources to each user based on the action information of the current time step and move to the next time step; The reinforcement learning network is trained based on a target risk perception advantage function constrained by conditional value at risk (CVaR). The target risk perception advantage function includes a penalty term that applies explicit fairness constraints to the communication rate of the tail users among the users at the current time step. The communication rate of each user at the current time step is determined by the resources allocated to the corresponding user at the current time step.

Images