Strategy optimization algorithm for non-cellular large-scale MIMO system

By using multi-agent reinforcement learning algorithm with graph attention in a large-scale MIMO system without cellular, we optimize content cache and resource allocation, the problem of difficult content cache and resource allocation strategies in a multi-user environment to adapt to dynamic changes, and the network is improved Rate and energy efficiency.

CN119095076AActive Publication Date: 2024-12-06CHONGQING UNIV OF POSTS & TELECOMM

Patent Information

Application Number
CN202411149227.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-21
Publication Date
2024-12-06
Estimated Expiration
2044-08-21

AI Technical Summary

Technical Problem

In the multi-user environment, the existing large-scale cellular MIMO system is difficult to adapt to the dynamically changing network environment, resulting in increased user content acquisition delays and low service quality.

Method used

Using multi-agent reinforcement learning algorithm based on graph attention, a user association model, downlink signal model and system energy consumption model are constructed, abstracted into a distributed part to observe Markov decision-making process, and optimize content cache, user association and power distribution strategies. .

Benefits of technology

It realizes adaptive content cache deployment and resource management in a dynamic network environment, improves network speed and energy efficiency, and meets different business needs.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119095076A_ABST
    Figure CN119095076A_ABST
Patent Text Reader

Abstract

The invention relates to a strategy optimization algorithm for a cellular-free large-scale MIMO system, and the algorithm comprises the steps: building a user association model which is used for modeling the association relationship between a mobile device MD and an access point AP in each time slot; constructing a downlink signal model, wherein the downlink signal model is used for modeling the network reachable rate of the cellular-free large-scale MIMO system in each time slot; constructing a system energy consumption model, wherein the system energy consumption model is used for modeling the total energy consumption of all access points (AP) providing services in each time slot; and constructing a target optimization problem model according to the constructed user association model, downlink signal model and system energy consumption model, and solving by using a multi-agent reinforcement learning algorithm based on graph attention to obtain an optimal strategy. The energy consumption of the system is reduced, and the user service quality is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the technical field of mobile communications, and in particular relates to a strategy optimization algorithm for a non-cellular large-scale MIMO system. Background Art

[0002] With the booming development of the Internet of Things and edge computing technologies, global communication traffic has exploded, posing new technical challenges to wireless network architecture. Due to the inherent structured boundaries and centralized management mechanisms of traditional cellular networks, it has become difficult to efficiently cope with the current complex and high-density communication business needs. To this end, the field of wireless communications has begun to turn to the research and exploration of new network architectures, among which cell-free massive MIMO (CF-mMIMO) is considered to be a potential solution. Unlike the fixed boundaries of traditional cellular MIMO systems, CF-mMIMO provides services to users through a large number of distributed node deployments, thereby achieving more uniform network coverage, and with its decentralized network characteristics, it has significantly improved the flexibility and adaptability of communication networks.

[0003] Most of the existing technologies are aimed at traditional CF-mMIMO network scenarios and consistent user service requirements, and do not fully consider the coupling relationship between content cache deployment, user association and resource allocation in multi-user CF-mMIMO network environments. In actual network scenarios, diversified user service requirements, decentralized resource deployment and dynamic network environment make the resource management problem of CF-mMIMO networks very complicated. First, the diversified user service requirements make different AP nodes have significant spatial differences in content caching, resource allocation and other aspects, which brings challenges to the efficient collaboration between AP nodes; secondly, the decentralized resource deployment of CF-mMIMO networks leads to uneven resource distribution, affecting the management performance and network resource utilization efficiency between APs; in addition, the dynamic nature of the network environment will also cause deep uncertainty in network status and resource availability. Traditional user association and resource allocation strategies have been difficult to adapt to highly dynamic changes in the network environment.

[0004] In summary, the problem of the prior art is that the dynamically changing needs of users in the CF-mMIMO content caching network affect the cache deployment and resource allocation of the AP, while the wireless AP cache capacity is limited and is not enough to store all the services that the user may request. Therefore, it is necessary to obtain the missing content from the fronthaul link or the backhaul link, which will increase the user content acquisition delay and the service quality is not high. Summary of the invention

[0005] In order to solve the problems existing in the background technology, the present invention proposes a policy optimization algorithm for a non-cellular massive MIMO system to learn the optimal content caching, user association and power allocation strategy. The system comprises: N access points AP with cache resources and M mobile devices MD, and is characterized in that it includes:

[0006] S1: constructing a user association model, wherein the user association model is used to model the association relationship between the mobile device MD and the access point AP in each time slot;

[0007] S2: construct a downlink signal model, where the downlink signal model is used to model the network achievable rate of the non-cellular massive MIMO system in each time slot;

[0008] S3: constructing a system energy consumption model, where the system energy consumption model is used to model the total energy consumption of all access points APs providing content services in each time slot;

[0009] S4: construct a target optimization problem model based on the constructed user association model, downlink signal model and system energy consumption model;

[0010] S5: The target optimization problem model is constructed as a partially observable Markov decision process model, and the multi-agent reinforcement learning algorithm based on graph attention is used to solve it to obtain the association strategy between the access point AP and the mobile device MD, the content caching strategy of the access point AP, and the power allocation strategy of the access point AP.

[0011] Preferably, the user association model includes:

[0012]

[0013]

[0014] and

[0015] in, Indicates time slot With the i-th mobile device MD i The associated access point AP set, represents a set of time slots; Indicates that the i-th mobile device MD in time slot t i and the jth access point AP j The relationship between Representing a collection The number of access points AP; Represents the set of all mobile device MDs, Represents the set of all access points AP.

[0016] Preferably, the downlink signal model includes:

[0017]

[0018]

[0019] g ij (t)=(d ij / d0) -α h ij (t)

[0020]

[0021] Among them, R sum (t) represents the network achievable rate of the non-cellular massive MIMO system in time slot t; R i (t) represents the i-th mobile device MD in time slot t i The received signal rate; Indicates the time slot t and the i-th mobile device MD i Associated access point AP set; represents the set of all mobile devices MD that have service requirements in time slot t; v ij (t) represents the i-th mobile device MD in time slot t i and the jth access point AP j The relationship between P ij (t) represents the jth access point AP in time slot t j Assigned to the i-th mobile device MD i The transmission power of represents the jth access point AP j Maximum transmission power; g ij (t) represents the jth access point AP j To the i-th mobile device MD i Downlink channel; represents the set of all APs in service at time slot t; Indicates that the jth access point AP in time slot t j To the i′th mobile device MD i′ The estimated channel gain of i′j (t) represents the jth access point AP in time slot t j Assigned to the i′th mobile device MD i′ The transmission power of i′j (t) represents the jth access point AP in time slot t j and the i′th mobile device MD i′ The relationship between i (t) represents the i-th mobile device MD in time slot t iReceived interference signal; d ij represents the jth access point AP j and the i-th mobile device MD i The actual distance between them; d0 represents the reference distance; α is the path attenuation factor; h ij (t) indicates that it follows a complex Gaussian distribution Small-scale decay.

[0022] Preferably, the system energy consumption model includes:

[0023]

[0024]

[0025]

[0026]

[0027]

[0028]

[0029] Where P(t) represents the total energy consumption of all serving APs in time slot t. represents the set of all APs in service at time slot t; P j (t) represents the jth access point AP in time slot t j Total energy consumption; Indicates that the jth access point AP in time slot t j Downlink content transmission power consumption; P ij (t) represents the jth access point AP in time slot t j Assigned to the i-th mobile device MD i The transmission power of Indicates that the jth access point AP in time slot t j The MD collection of the service; Indicates that the jth access point AP in time slot t j Energy consumption for updating or replacing service content; P FL The energy consumption required to transmit a unit of content on the fronthaul link; represents the jth access point AP j The set of service contents cached in time slot t, represents the jth access point AP j The maximum content cache capacity that can be cached; F represents the number of types of service content in the network; represents the jth access point AP j The set of service contents cached in time slot t-1; Indicates that the jth access point AP in time slot t j Energy consumption of content forwarding; pen represents the jth access point AP j Get the energy consumption factor of an uncached content; Indicates that in time slot t, the access point AP j A collection of all content requests of all associated mobile device MDs; Indicates that the i-th mobile device MD in time slot t i The content of the service requested, Represents a collection of service contents in the network.

[0030] Preferably, the target optimization problem model P1 includes:

[0031]

[0032]

[0033]

[0034]

[0035]

[0036]

[0037] in, is the content caching strategy of access point AP in time slot t, represents the jth access point AP in time slot t j The set of service contents cached in time slot t, N represents the number of access points AP; represents the association strategy between the access point AP and the mobile device MD in time slot t, Indicates time slot With the i-th mobile device MD i Associated access point AP set; is the power allocation strategy of the access point AP, For AP j The power allocation set, P ij (t) At time slot t, the jth access point AP j The transmission power allocated to the i-th mobile device MDi; T represents the number of time slots; represents mathematical expectation; R sum (t) represents the network achievable rate of the non-cellular massive MIMO system at time slot t; P(t) represents the total energy consumption of all serving APs at time slot t; v ij (t) represents the i-th mobile device MD in time slot t i and the jth access point APj The relationship between represents the jth access point AP j The maximum power value; Indicates AP j The MD collection of the service; represents the jth access point AP j The maximum content cache capacity that can be cached; Represents a collection of mobile devices; Indicates the access point AP set.

[0038] Preferably, constructing the target optimization problem model as a partially observable Markov decision process model comprises:

[0039] The target optimization problem model is converted into a Dec-POMDP model with N access points AP, where each AP represents an agent and is represented by a tuple Represents the global network environment status of the non-cellular massive MIMO system; represents the jth access point AP j The local observation space of represents the jth access point AP j action space, R is the reward function, γ∈[0,1) represents the discount factor;

[0040] At time slot t, the environmental state Defined as:

[0041]

[0042] in, represents the jth access point AP in time slot t j The content cache status of k fj(t) =1 indicates the jth access point AP j Content f has been cached at time slot t, otherwise k fj(t) =0; g ij (t) represents the jth access point AP in time slot t j With the i-th mobile device MD i The channel gain between i (t) represents the ith mobile device MD i location information;

[0043] At time slot t, the local observation Defined as:

[0044]

[0045] At time slot t, the action space Defined as:

[0046]

[0047] in, represents the jth access point AP j The set of service contents cached in time slot t; represents the jth access point AP j The set of mobile devices MD associated at time slot t; Indicates that the jth access point AP in time slot t i A power allocation set;

[0048] At time slot t, the reward function r(t)∈R is defined as:

[0049]

[0050] Among them, R sum (t) represents the network achievable rate of the non-cellular massive MIMO system at time slot t; P(t) represents the total energy consumption of all serving APs at time slot t.

[0051] Preferably, the method for solving the problem using a multi-agent reinforcement learning algorithm based on graph attention includes: a local action value network, a graph attention module and a hybrid module;

[0052] The local action value network configures a deep Q network composed of multi-layer perceptrons for each agent. At time slot t, the agent AP j Receive local observation o j (t), and choose action a j (t), o j (t) and a j (t) Input deep Q network output local action value Q j (o j (t), a j (t));

[0053] The graph attention module first inputs the environment state s(t) into the MLP encoder and encodes s(t) into local potential representation vectors h1(t), h2(t), ..., h N (t), where h j (t) represents the jth access point AP j The feature representation of the agent is then used to adaptively capture the correlation between agents to obtain the agent AP j The eigenvector of h′ j (t); then the agent AP j The eigenvector of h′ j (t) Input MLP as agent APj The local action value generates weight w j (t);

[0054] The mixing module is based on the local action value Q j (o j (t), a j (t)) and the local action value w j (t) Calculate the joint action value

[0055] The reinforcement learning model is trained by minimizing the loss function, namely:

[0056]

[0057] Among them, θ represents the parameters of the evaluation network, X represents the number of mini-batch samples randomly sampled from the experience replay pool, x represents the sample number, and y tot = r + γmax a′ Q tot (s′, a′; θ - ), r represents reward, a and a′ represent actions, s and s′ represent environmental states; θ - Represents the parameters of the target network;

[0058] The target optimization problem model is solved by the trained reinforcement learning model to obtain the association strategy between the access point AP and the mobile device MD, the content caching strategy of the access point AP, and the power allocation strategy of the access point AP.

[0059] The present invention has at least the following beneficial effects

[0060] Aiming at the dynamic time-varying network environment and incomplete network status observation, the present invention abstracts the above joint optimization problem into a distributed partially observable Markov decision process (Dec-POMDP), and designs autonomous decision-making content cache deployment, user association and transmission power control. Considering the diversified content cache requirements and wide-area differentiated network space characteristics in CF-mMIMO scenarios, the graph attention network is used to learn and capture network space characteristics to achieve adaptive interference control in the content delivery process and meet different business needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 A CF-mMIMO multi-user multi-content caching network scenario diagram of the present invention;

[0062] Figure 2 It is the GMJOC algorithm block diagram of the present invention;

[0063] Figure 3 It is a convergence performance curve diagram of the GMJOC algorithm of the present invention;

[0064] Figure 4 It is a curve diagram showing the change of the network rate of the GMJOC algorithm of the present invention as the number of training rounds increases;

[0065] Figure 5 This is a curve diagram showing how the power consumption of the GMJOC algorithm of the present invention changes as the number of training rounds increases. DETAILED DESCRIPTION

[0066] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0067] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0068] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0069] See also Figure 1 and Figure 2 The present invention provides a strategy optimization algorithm for a non-cellular large-scale MIMO system, the system comprising: N access points AP with cache resources and M mobile devices MD, including:

[0070] S1: constructing a user association model, wherein the user association model is used to model the association relationship between the mobile device MD and the access point AP in each time slot;

[0071] S2: construct a downlink signal model, where the downlink signal model is used to model the network achievable rate of the non-cellular massive MIMO system in each time slot;

[0072] S3: constructing a system energy consumption model, where the system energy consumption model is used to model the total energy consumption of all access points APs providing content services in each time slot;

[0073] S4: construct a target optimization problem model based on the constructed user association model, downlink signal model and system energy consumption model;

[0074] S5: The target optimization problem model is constructed as a partially observable Markov decision process model, and the multi-agent reinforcement learning algorithm based on graph attention is used to solve the association strategy between the access point AP and the mobile device MD, the content caching strategy of the access point AP, and the power allocation strategy of the access point AP.

[0075] In this embodiment, a typical CF-mMIMO network with multiple users and multiple content caches is considered, such as Figure 1 As shown, the network contains N APs and M mobile devices (MDs), and the AP and MD sets are defined as and Among them, different MDs have differentiated content requirements, that is, different MDs generate different content requests and are associated with different APs according to the current network status (distance between each AP, channel status, content deployment status); different APs cache corresponding service content according to content requirements within the service range, and are connected to the central processing unit (CPU) through optical fiber links. Assume that there are F types of service content of equal size in the network, define Represents the set of all service contents. The network runs in discrete time slots, and the time slot set is defined as

[0076] In this embodiment, a user association model is constructed. The user association model is used to model the association relationship between the mobile device MD and the access point AP in each time slot. The construction process is as follows:

[0077] In any time slot MD chooses to associate with different APs based on the distance, channel status, and content deployment status between each AP. Indicates that in time slot t and The associated AP set, Indicates MD iand The relationship between them is defined as:

[0078]

[0079] To ensure that all MDs can be served, In addition, not all APs are providing content services to MD. The set of all APs in service at time slot t is and

[0080] In this embodiment, a downlink signal model is constructed, and the downlink signal model is used to model the network achievable rate of the non-cellular massive MIMO system in each time slot. The construction process is as follows:

[0081] After the MD initiates a content request to its associated AP, if the AP has cached the content, the AP transmits the content to the corresponding MD via the downlink wireless channel. j To MD i The downlink channel can be expressed as:

[0082] g ij (t)=(d ij / d0) -α h ij (t)

[0083] Among them, g ij (t) represents the jth access point AP j To the i-th mobile device MD i Downlink channel; d ij Indicates AP j and MD i The actual distance between them, d0 represents the reference distance, usually d0 = 1m, α is the path attenuation factor, h ij (t) indicates that it follows a complex Gaussian distribution Small-scale decay.

[0084] Assume AP j Transfer to MD i The symbol is q i (t), where represents the mathematical expectation, then AP j The conjugate beamforming transmission signal with complete channel state information is x j (t), expressed as:

[0085]

[0086] in, Indicates that the jth access point AP in time slot t jAssigned to the i-th mobile device MD i The transmission power of Indicates AP j The maximum transmission power, For AP j and MD i The estimated channel gain at time slot t, For AP j The MD collection of the service.

[0087] Further, you can get MD i The received signal model r at time slot t i (t) is as follows:

[0088]

[0089] Among them, w i (t) is MD i The noise signal at time slot t, v i c (t)=v(t)\v i (t) represents the set of APs that do not provide services to MDi among all service APs. Denotes the set of all MDs with service requirements in time slot t.

[0090] According to Shannon's formula, we get MD i The received signal rate is:

[0091]

[0092] in, represents the estimated channel gain; w i Represents the noise signal received by MDi in time slot t.

[0093] We can further obtain the network achievable rate as

[0094]

[0095] in, represents the set of all MDs with service requirements in time slot t, R i (t) represents the i-th mobile device MD in time slot t i The received signal rate, R sum (t) represents the network achievable rate of the non-cellular massive MIMO system at time slot t.

[0096] In this embodiment, a system energy consumption model is constructed. The system energy consumption model is used to model the total energy consumption of all access points APs providing services in each time slot. The construction process is as follows:

[0097] APs are usually equipped with limited storage resources and can only store part of the service content, while CPUs have sufficient storage resources and can cache all network content. In CF-mMIMO network scenarios, different APs need to cache different service content according to the requirements of the associated MDs. Due to the limitation of AP storage capacity, a single AP cannot cache all content. represents the jth access point AP j The maximum content cache capacity that can be cached; represents the jth access point AP j The set of service contents cached in time slot t; where: and F represents the number of service contents in the network.

[0098] Assume that the total number of contents in the network is F equal to 100, numbered {1, 2, ..., 100}, and the content set is Indicates that in each time slot t, each AP caches 30 of them, using F j (t) indicates that each user MD generates a request, using f i (t) indicates. With access point AP j The collection of all content requests for all associated mobile device MDs is defined as Indicates AP j The set of MDs served. For each time slot t, if This means that the AP needs to obtain the data from the CPU. The energy consumption of missing content in the cache will be greater than that of directly obtaining it from the AP. The AP will continuously adjust the cache strategy to improve the hit rate and reduce energy consumption.

[0099] If the AP associated with the MD has cached the service content requested by the MD, the AP directly transmits the cached service content to the MD via the downlink wireless transmission link; if the AP has not cached the service content requested by the MD, the AP first downloads the corresponding service content from the CPU and then transmits it to the MD via the downlink wireless transmission link.

[0100] During the deployment and distribution of CF-mMIMO content cache, energy consumption mainly includes three parts:

[0101] 1) AP downlink content transmission power consumption:

[0102] The AP provides services to its associated MD. Different APs use different transmission powers for their associated MDs according to their customized power transmission strategies. In time slot t, AP j Downlink content transmission power consumption Defined as:

[0103]

[0104] in, represents the set of all MDs with service requirements in time slot t, P ij (t) represents the jth access point AP in time slot t j Assigned to the i-th mobile device MD i The transmission power of

[0105] 2) Energy consumption of AP service content update or replacement

[0106] To adapt to the dynamic changes of MD content requests, APs should cache, update or replace service content according to demand. j The set of service contents cached, updated or replaced in time slot t is Further AP j Energy consumption for updating or replacing service content in time slot t for:

[0107]

[0108] Among them, P FL The energy consumption required to transmit unit content on the forward link (FL); represents the jth access point AP j The set of service contents cached in time slot t; represents the jth access point AP j The set of service contents cached in time slot t-1.

[0109] 3) AP service content forwarding energy consumption

[0110] If the AP does not cache the service content requested by the MD, then the AP needs to obtain the corresponding service content from the CPU and forward it to the MD. In this case, the MD needs to receive the service content from the AP in time slot t. j The collection of content obtained from the CPU is Indicates that in time slot t, the access point AP j A collection of all content requests of all associated mobile device MDs; represents the jth access point AP j The set of service contents cached in time slot t can be used to define AP j The content forwarding energy consumption is:

[0111]

[0112] Among them, σ pen Indicates the energy consumption factor of obtaining an uncached content at the AP.

[0113] In summary, we can get AP j The energy consumption in time slot t is:

[0114]

[0115] In summary, the total energy consumption of all serving APs is:

[0116]

[0117] in, Indicates the set of APs that provide services to the MD.

[0118] In this embodiment, a target optimization problem model is constructed based on the constructed user association model, downlink signal model and system energy consumption model, and the process is as follows:

[0119] AP storage resources are limited, and content caching, user association, and power allocation issues are coupled to each other. This paper aims to achieve a higher network rate under a certain power consumption and models AP content caching, user association, and power allocation issues as follows:

[0120]

[0121]

[0122]

[0123]

[0124]

[0125]

[0126] in, is the content caching strategy of access point AP in time slot t, represents the jth access point AP in time slot t j The set of service contents cached in time slot t, N represents the number of access points AP; represents the association strategy between the access point AP and the mobile device MD in time slot t, Indicates time slot With the i-th mobile device MD i Associated access point AP set; is the power allocation strategy of the access point AP, For AP j The power allocation set, P ij (t) At time slot t, the jth access point AP j Assigned to the i-th mobile device MD iThe transmission power; T represents the number of time slots; represents mathematical expectation; R sum (t) represents the network achievable rate of the non-cellular massive MIMO system at time slot t; P(t) represents the total energy consumption of all serving APs at time slot t; v ij (t) represents the i-th mobile device MD in time slot t i and the jth access point AP j The relationship between represents the jth access point AP j The maximum power value; Indicates AP j The MD collection of the service; represents the jth access point AP j The maximum content cache capacity that can be cached; Represents a collection of mobile devices; Indicates the access point AP set.

[0127] In this embodiment, the target optimization problem model is constructed as a partially observable Markov decision process model, and is solved using a multi-agent reinforcement learning algorithm based on graph attention. The solution process is as follows:

[0128] The present invention designs a graph attention multi-agent deep reinforcement learningbasedJoint Optimization of content caching, user association and resource allocation for CF-mMIMO (GMJOC). AP is used as an agent to learn content caching strategy, user association strategy and power allocation strategy. Considering the spatial correlation between different APs, a multi-head attention mechanism is introduced when network parameters are aggregated to mine the correlation between agents.

[0129] The optimization problem P1 is transformed into a partially observable Markov decision process model (Dec-POMDP) ​​with N APs, where each AP represents an agent and is represented by a tuple <S,{O j} j∈N , {A j} j∈N , R,γ> to describe, Indicates the CF-mMIMO global network environment status, It is AP j The local observation space of It is APj action space, R is the reward function, and γ∈[0,1) represents the discount factor. At time slot t, each AP agent receives a local observation and select an action After that, the joint actions of all agents (such as content deployment, user association, resource allocation) can be obtained, using a(t)∈A=A1×...×A N After the agent performs the joint action, the environment returns a global reward r(t)=R(s(t), a(t)) and transfers the state to the next state s(t+1), where is the environment state of CF-mMIMO at time slot t. Next, we define the environment state, local observation, action and reward function.

[0130] 1) State Space

[0131] At time slot t, the environment state includes the content cache of all APs, channel state information, and user location state information. Therefore, the environment state Defined as

[0132]

[0133] in, represents the jth access point AP in time slot t j The content cache status of k fj(t) =1 indicates the jth access point AP j Content f has been cached at time slot t, otherwise k fj(t) =0; g ij (t) represents the jth access point AP in time slot t j With the i-th mobile device MD i The channel gain between i (t) represents the ith mobile device MD i location information.

[0134] 2) Local observation

[0135] In a partially observable CF-mMIMO environment, any AP can only observe the content cache status of its current time slot t and the user location information in the network. j The local observation at time slot t is expressed as:

[0136]

[0137] 3) Action Space

[0138] According to the optimization problem P1, AP iThe variables that can be optimized are content caching, user association, and power allocation; based on this, the intelligent AP is defined j Action a at time slot t j (t) is:

[0139]

[0140] in, represents the jth access point AP j The set of service contents cached in time slot t; Indicates the access point AP i The set of mobile devices MD associated at time slot t; Indicates that the jth access point AP in time slot t i A power allocation set;

[0141] 4) Reward Function

[0142] All agents perform a joint action a(t) = {a1(t), a2(t), ..., a N (t)}, the environment will return a global reward r(t) to evaluate the joint action. According to the optimization problem P1, the reward function is defined as:

[0143]

[0144] In partially observable environments, the agent AP j Receive the local observation value o j (t), and according to its local strategy π j Select action a j (t). Definition Represents the joint strategy of all agents. The ultimate goal of GMJOC is to learn a strategy for joint optimization of content caching, user association, and resource allocation to maximize the decaying cumulative global reward h represents the time slot set An element in , h = 0, 1, 2... Therefore, the joint action value function can be defined as:

[0145]

[0146] in, is the expected operation, the action value function Q π (s(t), a(t)) represents the expected decaying cumulative global reward with s(t) as the initial state, π as the initial joint strategy and a(t) as the initial action. The optimal joint strategy π * Is to make Q π (s(t), a(t)) is the largest joint strategy.

[0147] In this paper, the multi-agent environment is constructed as an undirected graph G(v, ε), where v is a set of nodes, each node represents an agent AP, ε is a set of edges representing the connectivity between nodes, and AP i There is an edge between it and the nearest AP. In the following, due to this one-to-one correspondence, AP, agent, and node all represent the same object. Then, this paper adopts a value decomposition network based on graph attention to decompose the joint action value function into a combination of local action value functions and attention-based relationships. The graph attention mechanism can mine the spatial correlation between APs and calculate the weights of the local action value function of each agent. This paper adopts the architecture of centralized training with decentralized execution (CTDE) for learning. In the centralized training stage, any agent learns and updates the network parameters of the agent in combination with the state information of other agents in the network environment. In the execution stage, each AP only needs to select its content caching, user association, and resource allocation actions based on its local observation information, without obtaining global state information. The GMJOC algorithm framework proposed in this paper is as follows Figure 2 As shown in Figure 3, it consists of three independent modules: 1) the agent’s local action-value network, 2) the graph attention module, and 3) the hybrid module.

[0148] 1) Local action value network: each agent is equipped with a deep Q-network (DQN) composed of a multilayer perceptron (MLP). Figure 2 As shown. At time slot t, the agent AP j Receive local observation o j (t), output a local action value function Q j (o j (t), a j (t)).

[0149] 2) Graph attention module, such as Figure 2 As shown in Figure 2, the environment state s(t) is first input into the MLP encoder, which encodes s(t) into local potential representation vectors h1(t), h2(t), ..., h N (t), where h j (t) is the feature representation of each node. Then, GAT is used to adaptively capture the correlation between agents.

[0150] In CF-mMIMO, any node (such as AP j ) has a set of neighbor nodes determined by the edge set ε If there is an edge between nodes, they are neighbors. In GAT, nodes APj Its neighboring nodes The attention coefficient between them can be expressed by e j,j′ = att(W·h j (t), W·h j′ (t)) is calculated, where att(.) is the self-attention mechanism and W is the corresponding learnable weight matrix. Attention coefficient e j,j′ Indicates neighbor node AP j′ The characteristics of the node AP j In this paper, we only consider the node The first-order neighbor nodes of (including node j). In order to facilitate comparison between different nodes, the attention coefficient needs to be normalized using the softmax function, as shown below:

[0151]

[0152] Among them, α j,j′ Represents the normalized attention coefficient;

[0153] In order to stabilize the learning process, this paper adopts a multi-head attention mechanism. After obtaining the normalized attention coefficient, the node AP with L independent attention mechanisms j The output feature representation vector h′ j (t) can be given by:

[0154]

[0155] Among them, σ is a nonlinear function, || represents the concatenation operation, and l represents the sequence number of the attention mechanism. Then, the MLP network is based on h′ j (t) as input and is the agent AP j The local action value function generates weights w j (t).

[0156] 3) Hybrid module. Based on the above analysis, the graph attention weight of the hybrid module can be obtained Then the joint action value function Q tot Can be decomposed into:

[0157]

[0158] The reinforcement learning model is trained by minimizing the loss function to learn the corresponding action selection strategy π, that is:

[0159]

[0160] Among them, θ represents the parameters of the evaluation network, X represents the number of mini-batch samples randomly sampled from the experience replay pool, x represents the sample number, and y tot= r + γmax a′ Q tot (s′, a′; θ - ), r represents reward, a and a′ represent actions, s and s′ represent environmental states; θ - Represents the parameters of the target network;

[0161] The target optimization problem model is solved by the trained reinforcement learning model to obtain the association strategy between the access point AP and the mobile device MD, the content caching strategy of the access point AP, and the power allocation strategy of the access point AP.

[0162] from Figure 3 It can be seen that with the increase of training rounds, the average reward of the GMJOC algorithm continues to increase, and finally stabilizes around the 400th round. At this time, the joint action strategy of the AP does not change much, and the reward value obtained is about 4, indicating that the agent is constantly optimizing its own cache strategy, user association strategy, and power allocation strategy. Figure 4 It can be seen that the system network rate is increasing and finally converges, Figure 5 The system power consumption in the algorithm decreases with the increase of training rounds, so the system energy efficiency gradually increases, which proves the effectiveness of the GMJOC algorithm.

[0163] In summary, the present invention abstracts the above joint optimization problem into a distributed partially observable Markov decision process (Decentralized partially observable Markov decision process, Dec-POMDP) ​​for dynamic time-varying network environment and incomplete network status observation, and designs autonomous decision-making content cache deployment, user association and transmission power control. Considering the diversified content caching requirements and wide-area differentiated network space characteristics in CF-mMIMO scenarios, a graph attention network is used to learn and capture network space characteristics to achieve adaptive interference control in the content delivery process and meet different business needs.

[0164] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

[0165] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A policy optimization algorithm for a non-cellular massive MIMO system, the system comprising: N access points AP with cache resources and M mobile devices MD, characterized in that: S1: constructing a user association model, wherein the user association model is used to model the association relationship between the mobile device MD and the access point AP in each time slot; S2: construct a downlink signal model, where the downlink signal model is used to model the network achievable rate of the non-cellular massive MIMO system in each time slot; S3: constructing a system energy consumption model, where the system energy consumption model is used to model the total energy consumption of all access points APs providing content services in each time slot; S4: construct a target optimization problem model based on the constructed user association model, downlink signal model and system energy consumption model; S5: The target optimization problem model is constructed as a partially observable Markov decision process model, and the multi-agent reinforcement learning algorithm based on graph attention is used to solve it to obtain the association strategy between the access point AP and the mobile device MD, the content caching strategy of the access point AP, and the power allocation strategy of the access point AP.

2. The strategy optimization algorithm for a non-cellular massive MIMO system according to claim 1, characterized in that: The user association model includes: in, Indicates time slot With the i-th mobile device MD i The associated access point AP set, represents a set of time slots; Indicates that the i-th mobile device MD in time slot t i and the jth access point AP j The relationship between Representing a collection The number of access points AP; Represents the set of all mobile device MDs, Represents the set of all access points AP.

3. The strategy optimization algorithm for a non-cellular massive MIMO system according to claim 1, characterized in that: The downlink signal model includes: g ij (t)=(d ij / d0) -α h ij (t) Among them, R sum (t) represents the network achievable rate of the non-cellular massive MIMO system in time slot t; R i 9t) represents the i-th mobile device MD in time slot t i The received signal rate; Indicates the time slot t and the i-th mobile device MD i Associated access point AP set; represents the set of all mobile devices MD that have service requirements in time slot t; v ij (t) represents the i-th mobile device MD in time slot t i and the jth access point AP j The relationship between P ij (t) represents the jth access point AP in time slot t j Assigned to the i-th mobile device MD i The transmission power of represents the jth access point AP j Maximum transmission power; g ij (t) represents the jth access point AP j To the i-th mobile device MD i Downlink channel; represents the set of all APs in service at time slot t; Indicates that the jth access point AP in time slot t j To the i′th mobile device MD i′ The estimated channel gain of i′j (t) represents the jth access point AP in time slot t j Assigned to the i′th mobile device MD i ′ transmission power; v i′j (t) represents the jth access point AP in time slot t j and the i′th mobile device MD i ′’s correlation relationship; i (t) represents the i-th mobile device MD in time slot t i Received interference signal; d ij represents the jth access point AP j and the i-th mobile device MD i The actual distance between them; d0 represents the reference distance; α is the path attenuation factor; h ij (t) indicates that it follows a complex Gaussian distribution Small-scale decay.

4. The strategy optimization algorithm for a non-cellular massive MIMO system according to claim 1, characterized in that: The system energy consumption model includes: Where P(t) represents the total energy consumption of all serving APs in time slot t. represents the set of all APs in service at time slot t; P j (t) represents the time slot t j Access Points j Total energy consumption; Indicates that in time slot t j Access Points j Downlink content transmission power consumption; P ij (t) represents the jth access point AP in time slot t j Assigned to the i-th mobile device MD i The transmission power of Indicates that the jth access point AP in time slot t j The MD collection of the service; Indicates that the jth access point AP in time slot t j Energy consumption for updating or replacing service content; P FL The energy consumption required to transmit a unit of content on the fronthaul link; represents the jth access point AP j The set of service contents cached in time slot t, represents the jth access point AP j The maximum content cache capacity that can be cached; F represents the number of types of service content in the network; represents the jth access point AP j The set of service contents cached in time slot t-1; Indicates that the jth access point AP in time slot t j Energy consumption of content forwarding; pen represents the jth access point AP j Get the energy consumption factor of an uncached content; Indicates that in time slot t, the access point AP j A collection of all content requests of all associated mobile device MDs; Indicates that the i-th mobile device MD in time slot t i The content of the service requested, Represents a collection of service contents in the network.

5. The strategy optimization algorithm for a non-cellular massive MIMO system according to claim 1, characterized in that: The target optimization problem model P1 includes: in, is the content caching strategy of access point AP in time slot t, represents the jth access point AP in time slot t j The set of service contents cached in time slot t, N represents the number of access points AP; represents the association strategy between the access point AP and the mobile device MD in time slot t, Indicates time slot With the i-th mobile device MD i Associated access point AP set; is the power allocation strategy of the access point AP, is the power allocation set of APj, P ij (t) At time slot t, the jth access point AP j Assigned to the i-th mobile device MD i The transmission power; T represents the number of time slots; represents mathematical expectation; R sum (t) represents the network achievable rate of the non-cellular massive MIMO system at time slot t; P(t) represents the total energy consumption of all serving APs at time slot t; v ij (t) represents the i-th mobile device MD in time slot t i and the jth access point AP j The relationship between represents the jth access point AP j The maximum power value; Indicates AP j The MD collection of the service; represents the jth access point AP j The maximum content cache capacity that can be cached; Represents a collection of mobile devices; Indicates the access point AP set.

6. The strategy optimization algorithm for a non-cellular massive MIMO system according to claim 1, characterized in that: The method of constructing the target optimization problem model as a partially observable Markov decision process model includes: The target optimization problem model is converted into a Dec-POMDP model with N access points AP, where each AP represents an agent and is represented by a tuple <S,{O j } j∈N ,{A j } j∈N ,R,γ>, Represents the global network environment status of the non-cellular massive MIMO system; represents the jth access point AP j The local observation space of represents the jth access point AP j action space, R is the reward function, γ∈[0,1) represents the discount factor; At time slot t, the environmental state Defined as: l1(t),l2(t),...,l M (t)} in, represents the jth access point AP in time slot t j The content cache status of k fj(t) =1 indicates the jth access point AP j Content f has been cached at time slot t, otherwise k fj(t) =0; g ij (t) represents the jth access point AP in time slot t j With the i-th mobile device MD i The channel gain between i (t) represents the ith mobile device MD i location information; At time slot t, the local observation Defined as: At time slot t, the action space Defined as: in, represents the jth access point AP j The set of service contents cached in time slot t; represents the jth access point AP j The set of mobile devices MD associated at time slot t; Indicates that in time slot t j Access Points j A power allocation set; At time slot t, the reward function r(t)∈R is defined as: Among them, R sum (t) represents the network achievable rate of the non-cellular massive MIMO system at time slot t; P(t) represents the total energy consumption of all serving APs at time slot t.

7. The strategy optimization algorithm for a non-cellular massive MIMO system according to claim 6, characterized in that: The multi-agent reinforcement learning algorithm based on graph attention is used to solve the problem, including: a local action value network, a graph attention module and a hybrid module; The local action value network configures a deep Q network composed of multi-layer perceptrons for each agent. At time slot t, the agent AP j Receive local observation o j (t), and choose action a j (t), o j (t) and a j (t) Input deep Q network output local action value Q j (o j (t),a j (t)); The graph attention module first inputs the environment state s(t) into the MLP encoder and encodes s(t) into local potential representation vectors h1(t),h2(t),…,h N (t), where h j (t) represents the jth access point AP j The feature representation of the agent is then used to adaptively capture the correlation between agents to obtain the agent AP j The eigenvector of h′ j (t); then the agent AP j The eigenvector of h′ j (t) Input MLP as agent AP j The local action value generates weight w j (t); The mixing module is based on the local action value Q j (o j (t),a j (t)) and the local action value w j (t) Calculate the joint action value The reinforcement learning model is trained by minimizing the loss function, namely: Among them, θ represents the parameters of the evaluation network, X represents the number of mini-batch samples randomly sampled from the experience replay pool, x represents the sample number, and y tot = r + γmax a′ Q tot (s′,a′;θ - ), r represents reward, a and a′ represent actions, s and s′ represent environmental states; θ - Represents the parameters of the target network; The target optimization problem model is solved by the trained reinforcement learning model to obtain the association strategy between the access point AP and the mobile device MD, the content caching strategy of the access point AP, and the power allocation strategy of the access point AP.

Citation Information

Patent Citations

  • Non-cellular large-scale MIMO power distribution method based on deep reinforcement learning

    CN114268348A

  • User-centered cellular-free large-scale MIMO power distribution method based on offline reinforcement learning

    CN115412134A

  • High-energy-efficiency transmission method in non-cellular large-scale URLLC system

    CN117062235A

  • Cellular-removing large-scale MIMO resource optimization method based on unmanned aerial vehicle-satellite cooperation

    CN117614520A

Cited By

  • Policy optimization algorithm for cell-free massive MIMO system

    WO2026040304A1