Dynamic power optimization method and system based on bidirectional multi-agent reinforcement learning

Abstract

The invention discloses a dynamic power optimization method and system based on bidirectional multi-agent reinforcement learning, and the method comprises the steps: collecting space-time joint channel data, an interference prediction result and neighbor base station feedback information, and constructing a base station multi-dimensional state space through a bidirectional real-time sensing mechanism; inputting the multi-dimensional state space of the base station into a bidirectional multi-agent reinforcement learning control module so as to carry out collaborative learning through a DEI-MADDPG algorithm, and constructing an output power adjustment strategy based on a collaborative Actor-Critic; generating a physical uplink shared channel control instruction based on a reinforcement learning strategy output result, and issuing the control instruction to realize dynamic allocation of base station transmitting power; and monitoring a network state and user feedback, executing strategy weight adjustment according to monitoring data, and returning to the step of constructing the base station multi-dimensional state space to realize base station transmission dynamic power optimization iteration closed-loop control. According to the invention, the transmitting power of the base station can be adaptively adjusted, and the whole SINR of the system can be improved while the same-frequency interference is suppressed.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent wireless communication, and in particular to a dynamic power optimization method and system based on bidirectional multi-agent reinforcement learning. BACKGROUND

[0002] In the process of 5G (5th Generation Mobile Communication Technology) communication system evolving to higher performance, the distributed network composed of multiple base stations (Base Station, BS) and multiple users (User Equipment, UE) faces three core challenges: first, the signal quality degradation problem caused by co-channel interference, as the network density improves, inter-cell interference (Inter-Cell Interference, ICI) has become a key factor limiting system performance; second, the channel time-varying characteristics caused by user mobility, traditional static optimization strategies are difficult to adapt to rapidly changing wireless environments; third, the dynamic balance problem of system capacity and quality of service (Quality of Service, QoS), which requires efficient allocation of resources under complex network topology. Traditional convex optimization and heuristic algorithms are difficult to adapt to the dynamic changes of network topology, and have exposed defects such as lack of real-time performance and low coordination efficiency in large-scale deployment scenarios, making it difficult to meet the needs of 5G and future communication systems.

[0003] With the Open Air Interface (OAI) architecture promoting the decentralization of the network, distributed intelligent control has become a key path to break through the above bottlenecks. In recent years, machine learning technology has achieved remarkable success in image processing, natural language processing and other fields, and its powerful adaptive ability and dynamic optimization characteristics provide a new direction for the functional improvement of wireless communication networks. In the 5G ultra-dense networking scenario, the high-density deployment of small base stations and macro base stations makes the network topology pseudo-random, resulting in significant differences in ICI characteristics from 4G (4th Generation Mobile Communication Technology) and previous networks. Research shows that nearly half of the signal receiving ends are faced with two or more interference sources, and traditional interference management techniques have been difficult to meet the demand. Although 5G systems have introduced advanced air interface technologies such as orthogonal frequency code division multiplexing and sparse code multiple access, more intelligent dynamic optimization mechanisms are still needed to cope with complex interference environments. SUMMARY

[0004] The present application aims to at least partially solve the technical problems in the related art. To this end, the first object of the present application is to provide a dynamic power optimization method based on bidirectional multi-agent reinforcement learning, which can adaptively adjust the base station transmit power, while suppressing co-channel interference and improving the overall system SINR.

[0005] The second object of the present application is to provide a dynamic power optimization system based on bidirectional multi-agent reinforcement learning.

[0006] A third object of the present application is to provide a computer-readable storage medium.

[0007] A fourth object of the present application is to provide an electronic device.

[0008] To achieve the above objects, the present application is implemented by the following technical solutions:

[0009] A dynamic power optimization method based on bidirectional multi-agent reinforcement learning, comprising:

[0010] Collecting spatio-temporal channel data, interference prediction results and neighbor base station feedback information, and constructing a base station multi-dimensional state space through a bidirectional real-time perception mechanism;

[0011] Inputting the base station multi-dimensional state space into a bidirectional multi-agent reinforcement learning control module, so that the bidirectional multi-agent reinforcement learning control module performs collaborative learning using a DEI-MADDPG algorithm, and outputs a power adjustment strategy based on a cooperative Actor-Critic architecture;

[0012] Generating a physical uplink shared channel control instruction based on the reinforcement learning strategy output result, and distributing the physical uplink shared channel control instruction to realize dynamic allocation of base station transmit power;

[0013] Monitoring network status and user feedback, adjusting strategy weights according to monitoring data, and returning to the step of constructing a base station multi-dimensional state space to realize iterative closed-loop control of base station transmit dynamic power optimization.

[0014] In one possible implementation, the interference prediction is implemented using a first-order autoregressive model, in which the current time interference value is generated by a weighted combination of historical interference data and Gaussian noise, and the weighting coefficient is a memory factor.

[0015] In one possible implementation, the cooperative Actor-Critic architecture uses a bidirectional reward function design, which includes a weighted combination of three parts: a local base station signal-to-interference-and-noise ratio indicator, a neighbor base station signal-to-interference-and-noise ratio weighted sum, and a local transmit power penalty term; wherein the weight of the neighbor base station is negatively correlated with its topological distance.

[0016] In one possible implementation, in the cooperative Actor-Critic architecture,

[0017] The Actor network outputs continuous power adjustment actions according to the local state, and updates the network parameters through a policy gradient method;

[0018] The Critic network inputs the joint state and action of all base stations, calculates the time difference target value using a double network structure, and synchronizes the target network parameters through a soft update mechanism.

[0019] In a possible implementation, the dynamic allocation of the base station transmit power is specifically: converting the power adjustment action into an actual transmit power value, and dynamically limiting the actual transmit power value to be between a preset maximum power and a zero power threshold.

[0020] In a possible implementation, the strategy weight adjustment includes:

[0021] An optimization objective function is constructed, and the optimization objective function is a weighted combination of three parts of a signal-to-interference-and-noise ratio positive gain, a transmit power negative cost, and an interference intensity negative cost;

[0022] The weight parameters of the optimization objective function are dynamically adjusted by a gradient descent method, and the step of constructing the base station multi-dimensional state space is returned, so as to realize iterative closed-loop control of base station transmit dynamic power optimization.

[0023] In a possible implementation, the method is applied to a 5G heterogeneous wireless network, and system model parameters of the 5G heterogeneous wireless network include a base station distribution model, a user movement model, a path loss calculation model, and a shadow fading calculation model.

[0024] To achieve the above object, the second aspect of the present application provides a dynamic power optimization system based on bidirectional multi-agent reinforcement learning, comprising:

[0025] A data perception module is configured to collect spatio-temporal joint channel data, interference prediction results, and neighbor base station feedback information, and construct a base station multi-dimensional state space through a bidirectional real-time perception mechanism;

[0026] A strategy learning and action decision module is configured to input the base station multi-dimensional state space into a bidirectional multi-agent reinforcement learning control module, so that the bidirectional multi-agent reinforcement learning control module performs collaborative learning by using a DEI-MADDPG algorithm, and outputs a power adjustment strategy based on a cooperative Actor-Critic architecture;

[0027] An action execution module is configured to generate a physical uplink shared channel control instruction based on a reinforcement learning strategy output result, and to implement dynamic allocation of base station transmit power by issuing the physical uplink shared channel control instruction.

[0028] A feedback collection module is configured to monitor network status and user feedback.

[0029] A closed-loop optimization module is configured to perform strategy weight adjustment according to the monitoring data, and to return to the step of constructing the base station multi-dimensional state space, so as to realize iterative closed-loop control of base station transmit dynamic power optimization.

[0030] To achieve the above object, the third aspect of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to realize the dynamic power optimization method based on bidirectional multi-agent reinforcement learning.

[0031] To achieve the above object, the fourth aspect of the present application provides an electronic device, which comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor executes the computer program to realize the dynamic power optimization method based on bidirectional multi-agent reinforcement learning.

[0032] The present application has at least the following technical effects:

[0033] The present application collects real-time spatio-temporal channel data including position, speed, path loss, interference prediction matrix and neighbor base station state feedback through a bidirectional perception mechanism, constructs a joint state space containing 20+ dimensional features, and then uses an improved multi-agent deep deterministic policy gradient DEI-MADDPG framework to realize adaptive adjustment and dynamic allocation of base station transmit power to suppress co-frequency interference through the introduction of an Actor-Critic network with a residual network structure, and to improve the overall system SINR while suppressing co-frequency interference. In addition, the present application realizes dynamic power optimization under the physical uplink shared channel and physical downlink shared channel cooperative mechanism. Specifically, the real-time perceived spatio-temporal channel data is downloaded to the user through the physical downlink control channel to guide the power allocation of the physical downlink shared channel, thereby realizing dynamic resource scheduling and interference coordination in multi-user and multi-target scenarios. This scheme breaks through the static optimization limitation of traditional power control, and its physical model-reinforcement learning fusion architecture provides a new paradigm for distributed wireless resource management under the OAI system, especially suitable for dynamic scenarios such as unmanned aerial base station cooperation and mobile edge computing in 6G (sixth generation mobile communication technology) networks, and has important significance for promoting the theoretical innovation and engineering practice of intelligent wireless communication networks.

[0034] Additional aspects and advantages of the present application will be given in part in the following description, will become apparent in part from the following description, or will be learned by practice of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0035] Figure 1 Flow chart of the dynamic power optimization method based on bidirectional multi-agent reinforcement learning of the embodiments of the present application.

[0036] Figure 2 Work flow chart of the dynamic power optimization method based on bidirectional multi-agent reinforcement learning of the embodiments of the present application.

[0037] Figure 3A base station and user position distribution map of an embodiment of the present application.

[0038] Figure 4 A SINR distribution histogram of an embodiment of the present application.

[0039] Figure 5 A user throughput distribution histogram of an embodiment of the present application.

[0040] Figure 6 A path loss distribution map of an embodiment of the present application.

[0041] Figure 7 A structure block diagram of a dynamic power optimization system based on bidirectional multi-agent reinforcement learning of an embodiment of the present application. DETAILED DESCRIPTION

[0042] The embodiments are described in detail below with reference to the accompanying drawings, in which the same or similar components have the same or similar designations and functions throughout various figures and / or portions of the drawings. The embodiments described below are exemplary and are intended to be illustrative of the present application rather than limiting.

[0043] The bidirectional multi-agent reinforcement learning-based dynamic power optimization method and system of the embodiments are described below with reference to the accompanying drawings.

[0044] Figure 1 A flowchart of a bidirectional multi-agent reinforcement learning-based dynamic power optimization method of an embodiment of the present application. As shown in Figure 1 , the method comprises:

[0045] Step S101: Collecting spatio-temporal joint channel data, interference prediction results, and neighbor base station feedback information, and constructing a base station multi-dimensional state space through a bidirectional real-time perception mechanism.

[0046] As shown in Figure 2 , a base station multi-dimensional state space can be constructed through a bidirectional real-time perception mechanism to fuse spatio-temporal joint channel data, interference prediction results, and neighbor base station feedback information.

[0047] In the embodiments, the local state of base station i at time t can be defined as .

[0048]

[0049] wherein, is the base station position, is the associated user position, is the associated user moving speed, and are the path loss and shadow fading, respectively. SINR, is the state message of neighbor base stations. In this embodiment, the interference prediction is implemented by using an AR(1) model of historical data, i.e., a first-order autoregressive model, in which the interference value at the current time is generated by a weighted combination of historical interference data and Gaussian noise, and the weighted coefficient is a memory factor. The first-order autoregressive model is specifically expressed as follows:

[0050]

[0051] wherein, are the interference data and interference prediction data of base station i at time t and t+1, respectively, and a is a memory factor, is the interference fluctuation variance of base station i, indicates a normal distribution.

[0052] Step S102: inputting the multi-dimensional state space of the base station into the bidirectional multi-agent reinforcement learning control module, so that the bidirectional multi-agent reinforcement learning control module performs collaborative learning by using a DEI-MADDPG algorithm, and outputs a power adjustment strategy based on a cooperative Actor-Critic architecture.

[0053] In a possible implementation, the cooperative Actor-Critic architecture adopts a bidirectional reward function design, and the bidirectional reward function includes a weighted combination of three parts, i.e., a local base station SINR index, a neighbor base station SINR weighted sum, and a local transmission power penalty term. The weight of the neighbor base station is negatively related to its topological distance.

[0054] Specifically, the strategy learning module in this embodiment adopts a bidirectional multi-agent deep deterministic policy gradient algorithm, and the core of the algorithm is to construct a bidirectional reward function:

[0055]

[0056] wherein, is the instantaneous reward of base station i, β1, β2, and β3 are the first to third weight coefficients, and j represents a neighbor base station, is a set of neighbor base stations, w i,j = 1-0.1|i-j| is a topological weight, is the SINR of the neighbor base station, is the transmission power of the local base station.

[0057] In one possible implementation, in the cooperative Actor-Critic architecture, the Actor (policy optimization neural network) network adjusts its actions based on the continuous power output of the local state and updates the network parameters through the policy gradient method; the Critic (value evaluation neural network) network takes into account the joint state and actions of all base stations, calculates the temporal difference target value using a dual-network structure, and synchronizes the target network parameters through a soft update mechanism.

[0058] Specifically, the Actor network policy function Output power adjustment action And update via policy gradient:

[0059]

[0060] in, The operational state of base station i All are gradient operators, and J is the agent-based objective function. To find the expected value function of the local state of base station i, ρ π The state distribution induced by strategy π, Q represents the gradient of the Actor network policy. ω (s,a) is the action value function of the Critic network. This represents the action gradient of the Critic network.

[0061] Critic network action value function Q ω (s,a) simplifies to Q ω Input joint state s = (s1,...,s N ) and the combined action a = (a1,...,a N ), where s N Let a be the local state of the Nth base station. N For the adjustment action of the Nth base station, where N is the total number of base stations, dual-Q (two independent Q networks) learning optimization is then used:

[0062]

[0063] in, Let be the target Q-value for performing the corresponding action in state t, γ be the discount factor used to balance the importance of current and future rewards, min be the minimum function, and s be the minimum value function. t+1 a t+1 Let Q be the joint state and joint action at time t+1. ω′ (s t+1 ,a t+1 Let be the Critic network action value function at time t+1. Where, a t+1Generated by the target Actor network, the target network parameters are maintained through soft update:

[0064]

[0065] Wherein, θ i is the Actor network weight parameter, and τ is a smoothing coefficient.

[0066] Step S103: A physical uplink shared channel control instruction is generated based on the output result of the reinforcement learning strategy, and the physical uplink shared channel control instruction is issued to realize dynamic allocation of the base station transmit power.

[0067] In the embodiment, the dynamic allocation of the base station transmit power is specifically: the power adjustment action is converted into an actual transmit power value, and is dynamically limited to between the preset maximum power and the zero power threshold.

[0068] Specifically, the action execution module converts the strategy output into a power control instruction, and then issues the instruction to realize dynamic adjustment of the base station transmit power:

[0069]

[0070] Wherein, is the base station transmit power at t+1, clip is a numerical clipping function, ΔP is a power adjustment amount, P max is the preset maximum power.

[0071] Step S104: The network state and user feedback are monitored, the strategy weight is adjusted according to the monitoring data, and the step of constructing the base station multi-dimensional state space is returned to realize the base station transmit dynamic power optimization iterative closed loop control.

[0072] In an embodiment of the application, the strategy weight adjustment includes: constructing an optimization objective function, the optimization objective function is a weighted combination of three parts of the signal-to-interference ratio positive gain, the transmit power negative cost, and the interference intensity negative cost; the weight parameters of each term of the optimization objective function are dynamically adjusted by the gradient descent method, and the step of constructing the base station multi-dimensional state space is returned to realize the base station transmit dynamic power optimization iterative closed loop control.

[0073] Specifically, the system performance monitoring collects network state feedback, and optimizes the learning process through strategy weight adjustment, and the optimization objective function is defined as follows:

[0074]

[0075] Wherein, To optimize the objective function, the global performance of resource allocation in a dynamic communication environment is quantified. Network state feedback includes multi-dimensional communication environment data, such as SINR, channel gain, and other channel quality data, as well as user location, movement data, base station transmit power, and interference data. This embodiment adjusts the weight parameter λ = [λ1, λ2, λ3] through gradient descent to form a closed-loop optimization link of "perception-decision-execution-feedback", and ultimately achieve Pareto optimality of system resource allocation. This research route enhances the collaboration ability of agents through a two-way information interaction mechanism, and combines space-time channel modeling and deep reinforcement learning to provide a systematic solution for interference coordination and power optimization in a dynamic communication environment.

[0076] In a possible implementation, the method is applied to a 5G heterogeneous wireless network, and system model parameters of the 5G heterogeneous wireless network include a base station distribution model, a user movement model, a path loss calculation model, and a shadow fading calculation model.

[0077] Optionally, the system framework architecture and core parameter definitions are as follows:

[0078] The system model is oriented to a 5G heterogeneous wireless network scenario, and a distributed communication network composed of N = 5 base stations (BS) and M = 10 user equipment (UE) is constructed, with T = 2000 sampling steps in the time dimension and a time interval Δt = 1 second. The core parameters of the system include: the maximum transmit power of the base station Pmax = 46 dBm, the maximum user movement speed vmax = 20 m / s, the operating frequency band f = 3.5 GHz, and the spatial range is limited within a square region of 1000 m x 1000 m.

[0079] The spatial distribution model of the base station and the user is as follows: the base station adopts a clustered spatial distribution, and the differentiated positions are generated by cluster center offset:

[0080]

[0081] where BS i is the actual position vector of the i-th base station, C i is the i-th cluster center, represents a Gaussian distribution, σ b = 30 m is the position offset standard deviation, and I2 is a two-dimensional identity matrix.

[0082] The initial position of each user relative to the base station to which it belongs satisfies:

[0083]

[0084] where UE i,j is the initial position vector of the j-th user under the i-th base station, D i is the user distribution radius of the i-th base station, is a two-dimensional uniform distribution random variable, representing the random offset of user's location relative to the base station, 12 is a two-dimensional all-one vector, ensuring the difference of user distribution range of different base stations.

[0085] The user mobility dynamic model is as follows: the user speed adopts a "speed- acceleration" second-order dynamic model, and the base station-specific noise is introduced:

[0086]

[0087] where v i,j (t) and v i,j (t+1) are the user moving speed at t and t+1 respectively, is a normal distribution random variable with mean 0 and variance , and σ a (i) = 0.5 + 0.1i is the standard deviation of the acceleration noise of the ith base station, and Δt is the time interval. The speed amplitude limit is:

[0088]

[0089] where v max is the maximum user moving speed.

[0090] The user position is updated linearly over time as follows:

[0091] p i,j (t+1) = p i,j (t) + v i,j (t+1) Δt (13)

[0092] where p i,j (t) and p i,j (t+1) are the user position at t and t+1 respectively.

[0093] The position boundary constraint is:

[0094] p i,j (t+1) = clip (p i,j (t+1), 0, 1000) (14)

[0095] The base station and user position distribution diagram is shown as follows Figure 3 .

[0096] The following are the SINR and user throughput collected in the communication process according to different timestamps, respectively, the SINR distribution histogram and the user throughput distribution histogram, as follows Figure 4 , Figure 5 .

[0097] In the communication scenario of 3.5 GHz frequency band, the path loss model is as follows:

[0098] L(d) = 128.1 + 37.6log 10 (d / 1000) dB (15)

[0099] Wherein, L(d) represents the path loss between the base station and the user, d represents the distance between the base station and the user, and is calculated in meters. When the distance changes, combined with the attenuation characteristics of the electric wave in the space propagation, the loss value will change according to the logarithmic law. The path loss distribution diagram is shown in Figure 6 Considering the shielding effect of obstacles on the signal in the actual communication environment, a shadow fading model based on the first-order autoregressive (AR (1)) process is introduced, and the discrete time expression is as follows:

[0100]

[0101] Wherein, S(t) and S(t+1) are the shadow fading values at t and t+1 respectively, and a = 0.9 is a memory factor for quantifying the time correlation of shadow fading; is a high-pass Gaussian white noise with zero mean and variance , and represents the standard deviation of the shadow fading in the coverage area of the i-th base station. The formula simulates the random attenuation characteristics of the signal caused by environmental factors such as topography, buildings, etc.

[0102] The channel gain, as the core parameter for evaluating the quality of the wireless channel, is determined by the path loss and the shadow fading, and its expression is:

[0103] G(d,t) = -L(d) + S(t) dB (17)

[0104] Wherein, G(d,t) is the channel gain, and dB is the unit.

[0105] The negative path loss term represents the basic gain of the signal, and after superimposing the random shadow fading, it can more accurately reflect the change of the signal strength in the actual channel. In terms of signal interference and quality evaluation, first define the signal-to-noise ratio (SNR) of the received signal as:

[0106] SNR = P t + G(d,t) - N0 dB (18)

[0107] Wherein, Pt is the base station transmit power (unit dBm), and N0 = -100 dBm is the system noise floor. For the same frequency interference problem, the interference power from other base stations is calculated by the following formula:

[0108]

[0109] where I j is the total interference power received by the jth user from other base stations, w i,k = 1 - 0.1|i - k| is the interference weight, representing the distance-dependent interference attenuation, P k is the transmit power of other base stations, G k,j (d k,j , t) is the channel gain from the kth base station to the jth user, d k,j is the distance between the kth base station and the jth user.

[0110] The interference power is converted to dBm (decibel-milliwatt) value as follows:

[0111] INR = 10log 10 (I j + 10 -9 ) dBm (20)

[0112] where INR is the value of the interference power converted to dBm.

[0113] The signal-to-interference-and-noise ratio (SINR) evaluation, and the final SINR is:

[0114]

[0115] To enable those skilled in the art to clearly understand the DEI-MADDPG (Multi-Agent Deep Deterministic Policy Gradient, an improved algorithm of MADDPG) algorithm of the embodiment, the algorithm is described in detail as follows.

[0116] Consider a distributed communication system consisting of N base stations and M users, each base station i can be regarded as an agent, and the core goal is to optimize the communication quality in the coverage area by adjusting the base station transmit power P i . The system state space S includes base station positions (x i , y i ), user positions (x j , y j ), user speeds (v jx , v jy ), channel parameters (path loss, shadow fading, channel gain), and received signal quality indicators (SNR, interference power, SINR), etc. The action space A is defined as the adjustment amount of the base station transmit power, and the reward function R is designed based on the normalized value of SINR to quantify the communication quality improvement effect. In the multi-agent reinforcement learning system, each agent i interacts with the environment, the state space is S = S1 x S2 x … x S N , the action space is A = A1 x A2 x … x A N , S NA is the local state of the Nth base station N is the policy adjustment action of the Nth base station. The policy of agent i is defined as π i i → A i , and the joint policy is π = (π 1 , π 2 ,..., π N ). π N is the policy of the Nth base station. The cumulative discounted return J i (π) is:

[0117]

[0118] wherein, is the reward of agent i at time t, γ is the discount factor, and ε is the environment dynamics, is the state and action sequence obtained according to the joint policy π and the environment dynamics. The DEI-MADDPG algorithm is a multi-agent reinforcement learning method based on the Actor-Critic architecture. The joint state is the state space S = (S1, S2,..., S N ); and the joint action is the action space A = (A1, A2,..., A N ). The Actor network updates the Actor network of each base station i maps the local state S i to the deterministic action A i , and the optimization goal is to maximize the long-term cumulative reward. The policy optimization is realized by the following formula:

[0119]

[0120] wherein, Q ω (S i , A i ) is the action value function estimated by the Critic network, and ρ π is the state distribution under the policy π. The action value function Q ω (S i , A i ) estimated by the Critic network inputs the joint state S and the joint action A, and outputs the action value. The parameters are updated using the temporal difference learning:

[0121] y j = R j + γQ′ ω′ (S j+1 , A′ j+1 ) (24)

[0122] ​

[0123] where y j is the target Q-value for performing the corresponding action in state S j is the immediate reward of base station j, Q′ ω′ (S j+1 ,A′ j+1 ) is the target Critic network, S j+1 is the next state performed by the environment, A′ j+1 is generated by the target Actor network, L is the loss function with mean square error, Q ω (S j ,A j ) is the output of the current Critic network, representing the Q-value estimate for performing action A j in state S j .

[0124] Further, the transmit power adjustment amount ΔP i of base station i has a relationship with the signal-to-interference-and-noise ratio (SINR) as follows:

[0125]

[0126] where SINR i is the signal-to-interference-and-noise ratio of base station i, G ii is the channel gain from base station i to its served users, G ij is the channel gain from base station i to user j, and N0 is the background noise power. The signal-to-interference-and-noise ratio of base station i is finally expressed as follows:

[0127]

[0128] where is the updated signal-to-interference-and-noise ratio of base station i after adjusting the transmit power, is the signal-to-interference-and-noise ratio of base station i before adjustment, and A j is the action value of the jth base station.

[0129] To guide the base stations to optimize the transmit power, a normalized reward function based on SINR is designed as follows:

[0130]

[0131] where R i is the immediate reward of base station i, μ(SINR) and σ(SINR) are the mean and standard deviation of SINR, respectively, and ∈ is a minimum value to avoid division by zero error. Each base station agent considers the joint state and action of all base stations when training, and captures the interference coupling relationship between base stations through the Critic network:

[0132] Q i (S,A)=f i(S1, S2,..., S N ,A1,A2,...,A N ) (28)

[0133] where Q i (S,A) is the action value function of the i-th base station agent.

[0134] This centralized training-decentralized execution mechanism makes the base station only need to make decisions based on the local state when actually deployed, which adapts to the deployment requirements of distributed communication systems. To improve the stability of training, the soft update method is used to update the target network parameters: θ' i ←τθ i +(1-τ)θ′ i , where the smoothing coefficient τ = 0.001:

[0135] ω′ i ←τω i +(1-τ)ω′ i (29)

[0136] where θ' i is the updated Actor network weight parameter, θ i is the Actor network weight parameter, ω i is the current Critic network weight parameter of the i-th agent (base station), and ω' i is the target Critic network weight parameter of the i-th agent.

[0137] The experience replay buffer stores the transition samples (s, a, r, s', d), and the sample correlation is reduced by batch sampling:

[0138]

[0139] where d t is the termination flag.

[0140] To enhance the exploration ability, the Ornstein-Uhlenbeck noise is used:

[0141] dx t = h(μ-x t )dt + σdW t (31)

[0142] where x t is the noise vector, μ is the regression mean, h is the regression strength, σ is the noise strength, and dW t is a random differential term.

[0143] where the noise strength gradually decays as the training progresses:

[0144] σ t =max(σ min ,σ t-1 ·decay) (32)

[0145] Where, σ t Let σ be the noise intensity at time t. min σ is the lower limit of noise intensity. t-1 Let be the noise intensity at time t-1, and decay be the attenuation coefficient.

[0146] In this embodiment, by storing historical samples D and randomly sampling, the training oscillation problem caused by data correlation can be solved, improving convergence robustness. Additionally, the noise intensity σ... t Dynamic decay can balance the breadth of early exploration with the accuracy of later strategies.

[0147] This invention verifies the effectiveness of the DEI-MADDPG algorithm in transmit power optimization through simulation experiments on a multi-base station communication system. Experimental results show that the trained base station agent can adaptively adjust transmit power, improving the overall SINR of the system while suppressing co-channel interference. Closed-loop simulations show that after five adjustments, the average SINR of each base station increases by 15%-20%, and the transmit power distribution becomes more reasonable, avoiding the power waste or interference overload problems of traditional fixed power configurations.

[0148] This invention applies the DEI-MADDPG algorithm to the transmit power optimization of a multi-base station, multi-user distributed communication system, and solves the dynamic interference coordination problem that is difficult to handle by traditional optimization methods through reinforcement learning.

[0149] Furthermore, the present invention also provides a dynamic power optimization system based on bidirectional multi-agent reinforcement learning. Figure 7 This is a block diagram of a dynamic power optimization system based on bidirectional multi-agent reinforcement learning, according to an embodiment of the present invention. Figure 7 As shown, the dynamic power optimization system 100 based on bidirectional multi-agent reinforcement learning includes a data perception module 10, a policy learning and action decision module 20, an action execution module 30, a feedback acquisition module 40, and a closed-loop optimization module 50 connected in sequence.

[0150] The data perception module 10 is used to collect spatio-temporal joint channel data, interference prediction results and neighbor base station feedback information, and construct a base station multi-dimensional state space through a bidirectional real-time perception mechanism; the strategy learning and action decision module 20 is used to input the base station multi-dimensional state space into the bidirectional multi-agent reinforcement learning control module, so that the bidirectional multi-agent reinforcement learning control module performs collaborative learning by using a DEI-MADDPG algorithm, and outputs a power adjustment strategy based on a cooperative Actor-Critic architecture; the action execution module 30 is used to generate a physical uplink shared channel control instruction based on the reinforcement learning strategy output result, and the physical uplink shared channel control instruction is used to realize dynamic allocation of base station transmission power; the feedback collection module 40 is used to monitor network state and user feedback; and the closed-loop optimization module 50 is used to perform strategy weight adjustment according to the monitoring data, and return to the step of constructing the base station multi-dimensional state space, so as to realize iterative closed-loop control of base station transmission dynamic power optimization.

[0151] It should be noted that the specific implementation of the dynamic power optimization system based on bidirectional multi-agent reinforcement learning in the embodiment can refer to the specific implementation of the dynamic power optimization method based on bidirectional multi-agent reinforcement learning described above, and details are not repeated here to avoid redundancy.

[0152] In summary, the application realizes real-time collection of spatio-temporal joint channel data including position, speed, path loss, interference prediction matrix and neighbor base station state feedback through a bidirectional perception mechanism, constructs a joint state space containing 20+ dimensional features, and then uses an improved multi-agent deep deterministic policy gradient DEI-MADDPG framework to realize adaptive adjustment and dynamic allocation of base station transmission power to suppress co-frequency interference, and also improve the overall system SINR while suppressing co-frequency interference. In addition, the scheme realizes dynamic power optimization under the physical uplink shared channel and physical downlink shared channel cooperative mechanism. Specifically, the spatio-temporal joint channel data perceived in real time is sent to the user through the physical downlink control channel to guide the power allocation of the physical downlink shared channel, thereby realizing dynamic resource scheduling and interference coordination in a multi-user, multi-target scenario. The scheme breaks through the static optimization limitation of traditional power control, and its physical model-reinforcement learning fusion architecture provides a new paradigm for distributed wireless resource management under the OAI system, especially suitable for dynamic scenarios such as unmanned aerial base station cooperation and mobile edge computing in 6G networks, and has important significance for promoting the theoretical innovation and engineering practice of intelligent wireless communication networks.

[0153] Further, the application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to realize the dynamic power optimization method based on bidirectional multi-agent reinforcement learning.

[0154] Further, the present application also provides an electronic device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, when the processor executes the computer program, the dynamic power optimization method based on bidirectional multi-agent reinforcement learning is realized.

[0155] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiment scheme according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0156] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be realized by means of software and the necessary general hardware platform, and of course can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of software products, and the computer software product can be stored in a computer readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the method described in each embodiment or some parts of the embodiment.

[0157] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A dynamic power optimization method based on bidirectional multi-agent reinforcement learning, characterized in that, include: Collect spatiotemporal joint channel data, interference prediction results, and feedback information from neighboring base stations, and construct a multi-dimensional state space for the base station through a two-way real-time sensing mechanism; The base station's multi-dimensional state space is input into the bidirectional multi-agent reinforcement learning control module, so that the bidirectional multi-agent reinforcement learning control module can perform collaborative learning using the DEI-MADDPG algorithm and output a power adjustment strategy based on the cooperative Actor-Critic architecture. Based on the output of the reinforcement learning strategy, a physical uplink shared channel control command is generated, and the physical uplink shared channel control command is issued to realize the dynamic allocation of base station transmit power; The system monitors network status and user feedback, performs policy weight adjustments based on monitoring data, and returns the steps for constructing the multi-dimensional state space of the base station to achieve iterative closed-loop control of dynamic power optimization for base station transmission.

2. The dynamic power optimization method based on bidirectional multi-agent reinforcement learning as described in claim 1, characterized in that, Interference prediction is achieved using a first-order autoregressive model, where the current interference value is generated by a weighted combination of historical interference data and Gaussian noise, with the weighting coefficient being a memory factor.

3. The dynamic power optimization method based on bidirectional multi-agent reinforcement learning as described in claim 1, characterized in that, The cooperative Actor-Critic architecture adopts a bidirectional reward function design, which includes a weighted combination of three parts: the local base station signal-to-interference-plus-noise ratio (SIR) index, the weighted sum of the SIR of neighboring base stations, and the local transmit power penalty term; wherein, the weight of the neighboring base station is negatively correlated with its topological distance.

4. The dynamic power optimization method based on bidirectional multi-agent reinforcement learning as described in claim 1, characterized in that, In the collaborative Actor-Critic architecture The Actor network adjusts its actions by outputting continuous power based on the local state and updates the network parameters using the policy gradient method. The Critic network takes into account the joint state and actions of all base stations, uses a dual-network structure to calculate the time-difference target value, and synchronizes the target network parameters through a soft update mechanism.

5. The dynamic power optimization method based on bidirectional multi-agent reinforcement learning as described in claim 1, characterized in that, The dynamic allocation of base station transmit power is specifically as follows: the power adjustment action is converted into the actual transmit power value, and the power is dynamically limited to between the preset maximum power and zero power threshold.

6. The dynamic power optimization method based on bidirectional multi-agent reinforcement learning as described in claim 1, characterized in that, Strategy weight adjustments include: An optimization objective function is constructed, which is a weighted combination of three parts: positive gain of signal-to-interference-plus-noise ratio, negative cost of transmit power, and negative cost of interference intensity. The gradient descent method is used to dynamically adjust the weight parameters of the objective function and return to the steps of constructing the multidimensional state space of the base station, so as to realize the iterative closed-loop control of dynamic power optimization of the base station.

7. The dynamic power optimization method based on bidirectional multi-agent reinforcement learning as described in any one of claims 1-6, characterized in that, The method is applied to 5G heterogeneous wireless networks, and the system model parameters of the 5G heterogeneous wireless network include base station distribution model, user mobility model, path loss calculation model and shadow fading calculation model.

8. A dynamic power optimization system based on bidirectional multi-agent reinforcement learning, characterized in that, include: The data sensing module is used to collect spatiotemporal joint channel data, interference prediction results and feedback information from neighboring base stations, and to construct a multi-dimensional state space of the base station through a two-way real-time sensing mechanism. The strategy learning and action decision module is used to input the base station's multi-dimensional state space into the bidirectional multi-agent reinforcement learning control module, so that the bidirectional multi-agent reinforcement learning control module can perform collaborative learning using the DEI-MADDPG algorithm and output a power adjustment strategy based on a cooperative Actor-Critic architecture. The action execution module is used to generate physical uplink shared channel control commands based on the output results of reinforcement learning strategies, and to issue physical uplink shared channel control commands to realize the dynamic allocation of base station transmit power; The feedback collection module is used to monitor network status and user feedback; The closed-loop optimization module is used to perform policy weight adjustments based on monitoring data and return the steps for constructing the multi-dimensional state space of the base station, thereby realizing iterative closed-loop control for dynamic power optimization of the base station's transmission.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-7.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1-7.

Images