A heterogeneous network high energy efficiency power control method based on intelligent cloud edge collaboration

Abstract

The present application belongs to the technical field of wireless communication and deep reinforcement learning, and particularly relates to a high energy efficiency power control method based on intelligent cloud edge cooperation for a heterogeneous network. The present application sets a target to optimize the transmission power of base stations in a typical heterogeneous network to maximize the global energy efficiency of the network. By establishing a target optimization model, the present application proposes a lightweight cloud-edge cooperative framework, in which the cloud can only collect data rate and energy loss information from the edge base station and then feed back the calculated global reward to the edge base station. In addition, the present application develops a multi-agent independent actor critic algorithm, so that each base station can independently optimize the transmission power strategy in real time. By using the algorithm proposed in the present application, the edge base station can configure the transmission power based on only local information, and optimize the strategy with the help of the global reward feedback from the cloud, and finally can achieve a global energy efficiency performance similar to that of the traditional optimization algorithm and greatly reduce the time complexity.

Description

Technical Field

[0001] This invention belongs to the fields of wireless communication and deep reinforcement learning technology, specifically relating to a high-efficiency power control method for heterogeneous networks based on intelligent cloud-edge collaboration. Background Technology

[0002] The surge in the number of smart devices has led to a significant increase in data throughput demands, placing a heavy burden on traditional cellular network systems. To address this issue, a new type of network called heterogeneous networks has emerged as an effective solution for enhancing network capacity and coverage by deploying different types of base stations within the service area. However, the ultra-dense deployment of heterogeneous base stations further increases network energy consumption, potentially reducing energy efficiency. Energy efficiency is one of the key metrics in next-generation wireless communication systems, defined as the ratio of total throughput to total power consumption, measured in bits per joule. Maximizing energy consumption in wireless networks is crucial for reducing carbon emissions and creating sustainable communication systems.

[0003] Joint power control for maximizing energy efficiency has been mathematically proven to be a challenging nondeterministic polynomial problem, making it difficult to obtain the optimal solution. To address this issue, various energy-saving algorithms have been proposed. Existing energy-efficient power control methods can be broadly categorized into three types: iterative optimization-based methods, deep learning-based methods, and deep reinforcement learning-based methods. Iterative optimization-based methods include the following: For example, a Sequential Fractional Programming (SFP) algorithm for joint power optimization has been proposed by combining fractional programming and sequential optimization techniques. This algorithm solves the global energy efficiency maximization problem and achieves near-optimal performance. Another algorithm based on branch and bound can be used to solve common high-energy-efficiency power control problems and obtains a globally optimal solution. Furthermore, there is an optimization framework for solving fractional polynomial problems that can be applied to the energy efficiency maximization problem in cellular networks. However, these traditional power control optimization algorithms require collecting instantaneous channel state information from all base stations, which is challenging in real-world scenarios. Moreover, they typically operate iteratively, which means high computational complexity and may render the optimization results outdated.

[0004] In recent years, deep learning and deep reinforcement learning-based methods have gradually demonstrated significant advantages in the field of wireless communication. Some studies have successfully employed these methods to solve the energy efficiency maximization problem. Specifically, this includes using unsupervised methods to train deep neural networks for power optimization, achieving higher energy efficiency fairness performance compared to traditional optimization techniques. Centralized, distributed, and transfer learning-based solutions based on deep reinforcement learning have been proposed to investigate the energy efficiency maximization problem in 5G cognitive heterogeneous networks. Among these solutions, the multi-agent distributed solution achieves optimal energy efficiency performance due to effective coordination among agents.

[0005] However, deep learning-based algorithms are susceptible to inconsistencies between offline training and online test datasets in typical heterogeneous networks, especially when the heterogeneous network environment changes rapidly. Deep reinforcement learning-based optimization algorithms can achieve high energy efficiency, but require extensive information exchange between the core network and different base stations. Summary of the Invention

[0006] To address the aforementioned issues, this invention proposes an intelligent power control method that enables each base station to independently control its transmission power using only local information, thereby improving overall energy efficiency.

[0007] Problem Description

[0008] This invention considers a typical heterogeneous network, which includes one macro base station and N micro base stations, such as... Figure 1 As shown. Each base station serves a corresponding user, with all base stations and users using a single antenna. In this invention, the user served by base station n is represented as user n, where n ∈ {0, 1, 2, ..., N}, and index 0 corresponds to the macro base station and its served user. Note that macro base stations and micro base stations share the spectrum band used for synchronizing downlink transmission.

[0009] This invention models the wireless channel between the base station and the user as consisting of two parts: large-scale fading (path loss and shadowing) and small-scale blocky Rayleigh fading. Path loss is determined by the distance between the base station and the user, while shadowing fading is caused by physical congestion of the base station-user link. This invention represents large-scale fading as the sum of path loss and shadowing, denoted as χ. n→k =L n→k +φ n→k L n→k and φ n→k These represent path loss attenuation and shadowing attenuation from base station n to user k, respectively. Small-scale block Rayleigh fading is caused by multipath transmission in the base station-user link and is a rapidly changing random variable. Specifically, the block Rayleigh fading from base station n to user k is represented as h. n→kFurthermore, it follows a cyclic symmetric complex Gaussian distribution with zero mean and unit variance, i.e.

[0010] Then, the channel gain can be expressed as g n→k =φ n→k |h n→k | 2 The signal-to-interference-plus-noise ratio measured at user n can be expressed as:

[0011]

[0012] Where, p n (t) represents the transmit power of base station n at time slot t, σ 2 This is the noise power spectral density present at the user's location. Therefore, the downlink rate from base station n to user n at time slot t can be modeled as...

[0013] r n (t)=Blog2(1+γ n (t)), (2)

[0014] Where B is the spectral bandwidth. Considering that the total power consumption consists of the base station's transmit power and the power consumed by the hardware circuitry within the base station, the global energy efficiency of the heterogeneous network at time slot t can be expressed as:

[0015]

[0016] Where ψ n p is the reciprocal of the power amplifier efficiency of base station n. c It is the total power of the circuit.

[0017] As mentioned above, the problem of maximizing global energy efficiency can be formulated as a joint power optimization problem, i.e.,

[0018]

[0019]

[0020] Where, p n,max This represents the maximum transmit power constraint for base station n. Considering that typical heterogeneous networks deploy different base stations to serve users in different areas, these base stations may have different limits on their maximum transmit power.

[0021] B. Framework of High-Efficiency Power Control Methods

[0022] As can be seen from (4), the global energy efficiency optimization problem in heterogeneous networks is highly related to the network's global instantaneous information, including downlink data rates and energy loss information of all base stations. However, constructing this global instantaneous information requires a large amount of information exchange between different base stations, which will lead to significant information exchange overhead. Therefore, this invention designs a lightweight collaboration between the core network (cloud) and different base stations (edge). Specifically, edge base stations do not need to exchange local instantaneous information with each other, while the cloud only needs to collect historical local energy efficiency-related data from each edge base station and calculate historical global rewards, and then use global reward feedback to improve the power control strategy of local base stations.

[0023] Based on the above discussion, this invention proposes a lightweight cloud-edge collaboration framework, such as... Figure 2 As shown. In this framework, each edge base station is connected to the cloud via a two-way wired or wireless link with latency.

[0024] At the edge, this invention establishes an independent actor-commentator structure and two experience buffers for each base station: a local experience replay buffer and a local-global experience replay buffer. Each edge base station will utilize its local actor neural network to determine an appropriate downlink transmit power and store the corresponding local experience and timestamp in its local experience replay buffer. Then, it only needs to upload the local energy efficiency-related data to the cloud.

[0025] In the cloud, this invention establishes a set of queues (receive queue and send queue) for exchanging historical data with each edge base station. Furthermore, this invention also designs a global calculation module that schedules the received data in the receive queue and calculates the historical global reward. The calculated historical global reward, along with its corresponding timestamp, is placed in each send queue, which then sends it to each edge base station in a first-in, first-out manner.

[0026] After receiving data from the cloud, each edge base station retrieves local experiences with the same timestamp from its local experience replay cache and combines them with global reward feedback to form a local-global experience. This local-global experience is then stored in the local-global experience replay cache. In this way, each edge base station can sample mini-batch local-global experiences in each update cycle and then periodically train the actor deep neural network and the critic deep neural network until each deep neural network converges. At this point, each actor deep neural network can learn a better local power control strategy to improve global energy efficiency performance.

[0027] C-type multi-agent independent actor-judge algorithm design

[0028] 1) Edge Network:

[0029] Edge state design: The state of the actor deep neural network n consists of historical local information from the previous time slot, local instantaneous information from the current time slot, and local auxiliary information, that is:

[0030]

[0031] The historical local information includes channel gain g. n→n (t-1), the transmit power p of edge base station n n (t-1), the received interference The obtained signal-to-interference-plus-noise ratio γ n (t-1), and downlink achievable rate r n (t-1). Then, the local instantaneous information includes the channel gain g. n→n (t), interference received at the beginning of the current time slot before configuring the new transmit power. In addition, the local auxiliary information is the reciprocal ψ of the power amplifier efficiency of the edge base station n. n To obtain local instantaneous information at time slot t, base station n transmits a power of p to user n at the beginning of time slot t. n The orthogonal pilot sequence of (t-1) is used to obtain the channel gain g locally. n→n (t) and interference

[0032] Edge Action Design: At time slot t, the action design of the actor deep neural network n is the downlink transmit power, i.e., a. n =p n ;

[0033] Edge experience: At the end of time slot t, edge base station n can acquire local experience and construct it into a set, including state-action pairs from the previous time slot and the state in the current time slot, i.e.

[0034]

[0035] Upon receiving historical global rewards from the cloud, each edge base station can construct a local-global experience by combining its local experience with the historical global rewards using the corresponding timestamps. If the global reward is represented as R and the transmission delay between the edge and the cloud is represented as T... l Then the local-global experience of edge base station n at time slot t can be expressed as:

[0036]

[0037] Edge Actor and Judge Deep Neural Network Design: The network structure design of the actor deep neural network n is as follows Figure 3As shown. Specifically, it is a fully connected deep neural network, with the input layer having a state corresponding to the design state s. n The number of ports is eight, and the output is mapped to 0 and p. n,max The feasible transmit power between. Figure 4 The paper designs a deep neural network structure for critics, n, which includes a state module, an action module, and a value evaluation module. The state module receives local states s. n As input, the action module receives the local action a. n As input, the value assessment module takes the outputs of the state and action modules as input. The value assessment module then outputs a long-term Q-value, which is used to evaluate the performance of global energy efficiency.

[0038] 2) Cloud network:

[0039] As discussed in the framework, the cloud is designed to calculate historical global rewards by collecting local energy efficiency-related data from each base station. To achieve this, the present invention establishes N+1 queues and a global computing module in the cloud. Each queue (one receive queue and one transmit queue) is associated with each edge base station and operates the data in a first-in, first-out manner.

[0040] Global Reward Design: At a given time slot, the global reward is designed to evaluate the global energy efficiency performance achievable by all base stations. To ensure effectiveness and simplicity, this invention designs the global reward at time slot t as follows:

[0041] R(t) = GEE(tT) l (8)

[0042] Then, according to (4), at time slot t, the local energy efficiency related data of edge base station n is designed as data rate r n (t), and energy consumption information, including transmit power p n (t), the reciprocal of the power amplifier efficiency ψ n And local circuit power. In this way, each edge base station can upload the acquired local energy efficiency data and timestamps to the cloud at the end of each time slot.

[0043] D Deep Neural Network Training Process

[0044] This invention introduces a multi-agent independent actor-judge power control algorithm to train edge deep neural networks by extending the deep deterministic policy gradient algorithm. Based on the edge design, this invention designs deep neural networks for each actor and judge as follows: and in and These are the weights of the actor deep neural network and the judge deep neural network, respectively. To ensure the stability of the training process of the judge deep neural network and the actor deep neural network, this invention creates a target judge deep neural network for each judge deep neural network, which is... The representation, and the target actor deep neural network n built for each actor deep neural network n, are derived from... It should be noted that the weights of the target commentator deep neural network and the target actor deep neural network are initialized using the corresponding weights of the commentator deep neural network and the actor deep neural network.

[0045] Initially, all edge base stations determine a random transmit power for downlink transmission at the start of each time slot. Then, by continuously exchanging local energy efficiency data and historical global rewards with the cloud, edge base stations are able to build and store local-global experiences in their local-global experience replay caches. Once each local-global experience replay cache has stored at least D experiences, the edge base station randomly samples mini-batch local-global experiences to train the commentator deep neural network and the actor deep neural network. Below, the invention provides the training processes for the commentator deep neural network, the actor deep neural network, and the target deep neural network, respectively.

[0046] 1) Training the judge's deep neural network:

[0047] For an edge base station n, its local-global experience can be represented as E = {s} n ,a n ,R,s′ n Then, the target value of the deep neural network n is evaluated. It can be represented as

[0048]

[0049] Where η is the discount factor, a′ n It is a target actor neural network The output action. Then, the loss function of each commenter's deep neural network can be expressed as the mean squared error between the predicted Q-value and the target's long-term global energy efficiency, i.e.,

[0050]

[0051] Therefore, each commentator's deep neural network can backpropagate its loss and update its weights using a gradient descent-based method. By repeating the above process, each commentator's deep neural network can gradually optimize its weights in the direction of maximizing global energy efficiency performance.

[0052] 2) Training the actor deep neural network:

[0053] The training of the actor deep neural network will utilize gradients obtained from the critic deep neural network. Specifically, the weight updates of the actor deep neural network n require maximizing the long-term Q-value of the critic deep neural network n, i.e.

[0054]

[0055] Where α (a) It is the learning rate of the actor deep neural network n. Is it the expected Q-value pair? The partial derivative is expressed as:

[0056]

[0057] To balance the utilization of learned policies and the exploration of better policies during the training phase of the actor-based deep neural network, this invention adds Gaussian noise to the output of the actor policy, i.e.,

[0058]

[0059] in, It is zero-mean action noise, and its variance ξ determines the trade-off between policy exploitation and exploration. Intuitively, higher variance can accelerate the learning of deep neural networks in the early training phase, while lower variance can help stabilize performance in the later training phase. Here, the present invention designs the noise variance ξ with an initial value ξ. ini and minimum value ξ min It is initialized to ξ ini It decays exponentially at a fixed rate λ as the time slot increases, and when it is below ξ... min The time will remain as ξ min .

[0060] 3) Training the target deep neural network:

[0061] This invention employs a soft update method to update the target deep neural network to stabilize the training process, that is,

[0062]

[0063]

[0064] Where τ (c) and τ (a) These are the soft update rates of the target evaluator deep neural network and the target actor deep neural network, respectively.

[0065] The beneficial effects of this invention are as follows: Compared with traditional power control optimization algorithms, the proposed framework allows each edge base station to determine the appropriate transmit power using only local information. This ensures real-time power control and reduces the computation caused by centralized optimization. Secondly, the proposed framework enables lightweight collaboration between the cloud and different edge base stations, thereby minimizing the information exchange overhead between the cloud and edge base stations. Finally, this invention significantly reduces time complexity while maintaining global energy efficiency performance similar to traditional iterative optimization algorithms. Attached Figure Description

[0066] Figure 1 This is a schematic diagram of a downlink heterogeneous network structure.

[0067] Figure 2 This is a schematic diagram of the lightweight cloud-edge collaboration framework of the present invention.

[0068] Figure 3 This is a schematic diagram of the deep neural network structure for actors.

[0069] Figure 4 A schematic diagram of a deep neural network structure for evaluation.

[0070] Figure 5 This is a schematic diagram comparing the global energy efficiency performance during the training phase.

[0071] Figure 6 This is a schematic diagram comparing the global energy efficiency performance during the testing phase. Detailed Implementation

[0072] The practicality of this invention is illustrated below with simulation results and accompanying figures. First, the system model settings and hyperparameters used in the simulation are provided. Then, simulation results are presented to evaluate the performance of the proposed algorithm. The simulation results compare the proposed algorithm with the SFP algorithm, stochastic power algorithm, and maximum power algorithm in terms of global energy efficiency and time complexity.

[0073] Table 1. Channel parameters and algorithm hyperparameters used in the simulation.

[0074]

[0075]

[0076] This invention considers a two-layer heterogeneous network scenario with one macro base station and four micro base stations. The macro base station is deployed in the first layer, located at coordinates (0, 0). Micro base stations 1 to 4 are distributed in the second layer, located at coordinates (500, 0), (0, 500), (-500, 0), and (0, -500), respectively. Each base station is responsible for signal coverage of a circular area. The maximum radius of the macro base station is 1000m, and the maximum radius of the micro base stations is 200m. A minimum coverage radius of 10m is set for all base stations. Notably, the macro base station is subject to a maximum power constraint of 30dBm, and the micro base stations are subject to a maximum power constraint of 23dBm.

[0077] Specifically, this invention divides the simulation into a training phase and a testing phase. During the training phase, the proposed algorithm trains a deep neural network for 30,000 time slots. During the testing phase, the proposed algorithm tests for 2,000 time slots, directly configuring the base station's transmission power using the well-trained actor deep neural network without any further parameter updates. To ensure the robustness of the simulation results, this invention conducted 10 independent trials using different seed settings. During each trial, the user's location was randomly generated within their coverage area.

[0078] Figure 5 The results of the algorithm during the training phase are provided. It can be seen that after approximately 1000 time slots, the proposed algorithm rapidly outperforms both the random power and maximum power algorithms. After this point, the algorithm converges rapidly and achieves global energy efficiency performance close to that of the SFP algorithm at the end of the training phase. Figure 6 The performance results during the testing phase are presented. It can be seen that the global energy efficiency of the proposed algorithm is much higher than that of the random algorithm and the maximum power algorithm, while achieving approximately 99.08% of the performance of the SFP algorithm.

[0079] It is important to note that while the SFP algorithm utilizes global instantaneous channel state information for centralized power optimization at each time slot, the proposed algorithm enables each base station to determine the appropriate transmit power using only local information. These results highlight the advantages of the proposed algorithm in optimizing global energy efficiency performance in heterogeneous networks.

[0080] Table 2. Training parameters for deep neural networks.

[0081]

[0082] The average time complexity of the proposed algorithm is compared with that of the SFP algorithm. Note that the (target) actor and (target) commentator deep neural networks only require local information as input, which reduces the complexity of the input data. Therefore, these deep neural networks can be trained efficiently with relatively fewer training parameters. Furthermore, Table 2 shows that the training time for each deep neural network is approximately 2 ms, and the training time for each target deep neural network is less than 3 ms. During the testing phase, the proposed algorithm requires approximately 1.020 ms per actor deep neural network to calculate the transmit power for global energy efficiency, while the SFP algorithm requires approximately 53.042 ms to optimize the transmit power. This result demonstrates the significant advantage of the proposed algorithm in terms of time complexity. This is expected because each edge base station can efficiently utilize the local actor deep neural network to directly calculate the transmit power, while traditional methods rely on iterative calculations, which are very time-consuming.

Claims

1. A high-efficiency power control method for heterogeneous networks based on intelligent cloud-edge collaboration, comprising a macro base station and Each micro base station serves one corresponding user, and all base stations and users use a single antenna, thus creating an edge base station. The user of the service is represented as a user. ,in and index Corresponding to macro base stations and their served users Macro base stations and micro base stations share the spectrum band used for synchronous downlink transmission; its characteristic is that, The power control method is as follows: The objective is to jointly optimize the transmit power of each macro base station and micro base station, and maximize the global energy efficiency of the entire heterogeneous network. The objective model is as follows: ， in, It is a time slot Time edge base station The transmission power, It is an edge base station Maximum transmit power constraint, It is a time slot Global energy efficiency of time-varying heterogeneous networks It is the spectrum bandwidth. At the edge base station The signal-to-interference-plus-noise ratio measured at the service user's location. It is an edge base station The reciprocal of the power amplifier efficiency, It is the total power of the circuit; Based on the target model, deep reinforcement learning is used to enable edge base stations to jointly optimize transmission power, and a lightweight cloud-edge collaboration framework is designed, specifically as follows: This eliminates the need for edge base stations to exchange local instantaneous information with each other. Meanwhile, the cloud only collects historical local energy efficiency data from each edge base station and calculates historical global rewards, thereby using global reward feedback to improve the power control strategy of local base stations. At the edge, an independent actor-commentator structure and two experience buffers are established for each base station, namely the local experience replay buffer and the local-global experience replay buffer. Each edge base station will use its local actor neural network to determine the downlink transmit power and store the corresponding local experience and timestamp in its local experience replay buffer, and then upload the local energy efficiency related data to the cloud. In the cloud, a set of queues is established to exchange historical data with each edge base station. A global computing module is also used to schedule the received data in the receiving queue and calculate the historical global reward. The calculated historical global reward, along with the corresponding timestamp, is placed in each sending queue, and then the sending queue sends it to each edge base station in a first-in-first-out manner. After receiving data from the cloud, each edge base station retrieves local experiences with the same timestamp from its local experience replay cache and combines them with global reward feedback to form a local-global experience. Finally, this local-global experience is stored in the local-global experience replay cache. In this way, each edge base station samples a small batch of local-global experiences in each update cycle and then periodically trains the actor deep neural network and the critic deep neural network until each deep neural network converges. Thus, each actor deep neural network learns a better local power control strategy to improve global energy efficiency performance. The specific design of the edge base station and the cloud-based core network is as follows: 1) Edge Network: Edge state design: Deployed at edge base stations The state of the actor-based deep neural network consists of historical local information from the previous time slot, local instantaneous information from the current time slot, and local auxiliary information, namely: ， Historical local information includes channel gain. Edge base station Transmission power The received interference The obtained signal-to-interference-plus-noise ratio and downlink achievable speed Then, the local instantaneous information includes channel gain. Interference received at the beginning of the current time slot before configuring the new transmit power. Local auxiliary information is for edge base stations. The reciprocal of the power amplifier efficiency In order to obtain time slots Local instantaneous information at any given moment, edge base station In the time slot The beginning of users Transmission power is The orthogonal pilot sequence is used to obtain the channel gain locally. and interference ; Edge motion design: in time slots At any time, deployed at edge base stations Action design of deep neural networks for actors Downlink transmit power ,Right now ; Edge Experience: In Time Slots At the end of the time, the edge base station Acquire local experience and construct it into a set, including state-action pairs from the previous time slot and the state in the current time slot, i.e.: ， When receiving historical global rewards from the cloud, each edge base station constructs a local-global experience by combining its local experience with the historical global rewards using the corresponding timestamp. If the global reward is represented as... And the transmission delay between the edge and the cloud is expressed as Then edge base station In the time slot The local-global experience at any given moment is represented as: ； 2) Cloud network: The cloud is designed to calculate historical global rewards by collecting local energy efficiency-related data from each base station. Queues and a global computing module are established in the cloud, where each queue is associated with each edge base station. Each queue includes a receive queue and a send queue, and the data is processed in a first-in-first-out manner. Global Reward Design: At a given time slot, the global reward is designed to evaluate the global energy efficiency performance achieved by all base stations, and the time slot is designed accordingly. The global reward at time step is: ， In the time slot At any time, edge base stations Local energy efficiency data is designed to be data rate and energy consumption information, including transmission power. The reciprocal of the power amplifier efficiency In this way, each edge base station uploads the local energy efficiency-related data and timestamps it obtains to the cloud at the end of each time slot, along with local circuit power.

Images