Beam forming method for active RIS-assisted wireless energy supply air calculation based on deep reinforcement learning

Abstract

The invention relates to the technical field of wireless communication, and discloses a beam forming method for active reconfigurable intelligent surface (RIS)-assisted wireless energy supply air computing based on deep reinforcement learning, which comprises the following steps: constructing an active RIS-assisted wireless energy supply air computing system; according to the method, multiple constraint conditions such as energy collection and active RIS are comprehensively considered, and an optimization model for joint optimization of a hybrid access node emission covariance matrix, a receiving beam vector, an RIS reflection matrix and Internet of Things equipment emission power is established by taking minimization of an air calculation mean square error (MSE) as a target; modeling a beam forming behavior into a Markov decision process, solving an optimization problem by adopting a reinforcement learning algorithm based on depth deterministic policy gradient (DDPG), and improving training convergence by introducing an exponential weighted moving average mechanism; the effectiveness of the algorithm is verified through simulation, and the influence of parameters such as different transmitting powers, the number of RIS units and the number of devices on data aggregation is analyzed. Compared with the prior art, small MSE can be obtained through efficient downlink energy beam forming and uplink transmission beam forming, and the data aggregation precision of the Internet of Things is improved.

Description

Technical Field

[0001] The present invention relates to the field of wireless communication technologies, and in particular to a beamforming method for active RIS-assisted wireless powered air computing based on deep reinforcement learning to minimize data aggregation errors. Background Art

[0002] With the rapid development of Internet of Things (IoT) technology and the exponential growth of IoT device deployment, future IoT applications face two key challenges. First, the scarcity of radio spectrum resources makes it increasingly challenging to meet the needs of fast wireless data aggregation (WDA) for numerous distributed IoT devices. Second, the lifespan of IoT devices is limited by the capacity of their energy-constrained batteries. Therefore, how to provide efficient and sustainable WDA for IoT devices has become a challenging problem in IoT networks.

[0003] To address these challenges, over-the-air computation (AirComp) and wireless power transfer (WPT) have emerged as two promising technologies. Specifically, AirComp leverages the signal superposition properties of wireless multiple-access channels to compute various functions over the air, thereby reducing communication and computational latency and enabling efficient data aggregation. Meanwhile, WPT offers an effective approach to overcome the energy limitations of IoT devices, enabling them to harvest energy from ambient RF radiation.

[0004] However, unfavorable channel conditions can significantly degrade AirComp's performance, and the harvested energy may be insufficient to support WDA. Reconfigurable intelligent surfaces (RIS) can reconfigure the channel environment, creating a favorable propagation environment to improve energy and spectral efficiency. RIS can enhance the energy harvesting and data transmission capabilities of IoT users by optimizing the RIS reflection matrix. Compared to traditional RIS, active RIS can overcome the multiplicative fading caused by passive RIS, resulting in higher channel gain. In summary, the introduction of active RIS offers the potential for improving the performance of wirelessly powered airborne computing systems.

[0005] However, in practical deployments, improving data aggregation accuracy through the joint design of parameters such as transmit power, RIS reflection matrix, and receive beam vectors, under the conditions of time-varying channels and the surge in IoT device data, remains a technical challenge. Traditional optimization algorithms demonstrate good performance when the number of base stations and RIS reflectors is small. However, as the number of base station antennas and RIS reflectors increases, they face challenges such as high computational complexity and slow response to environmental changes, making them difficult to adapt to the real-time optimization needs of dynamic, large-scale networks. Furthermore, they require a precise mathematical model of the environment in advance, making it difficult to quickly deliver optimization results in real-world scenarios. Summary of the Invention

[0006] The present invention provides a beamforming method for a wireless powered airborne computing network assisted by active RIS based on deep reinforcement learning. This method overcomes the problem that when traditional optimization algorithms are applied to resource allocation in airborne computing networks, they require an accurate mathematical model of the environment to be known in advance. In the face of a constantly changing channel environment, it is difficult to implement efficient beamforming in actual scenarios to obtain accurate data aggregation information.

[0007] To achieve the above objectives, the present invention provides a beamforming method based on deep reinforcement learning. This method overcomes the requirement for a precise mathematical model of the environment to be known in advance when traditional optimization algorithms are applied to beamforming design in an active RIS-assisted wireless power-supply airborne computing network. Specifically, a beamforming method based on deep reinforcement learning for active RIS-assisted wireless power-supply airborne computing is proposed, comprising the following steps: Construct an active RIS-assisted wireless power supply airborne computing system, wherein the airborne computing system framework includes a M hybrid access point with 1 antenna, 1 with N Active RIS of a reflector unit and K A single-antenna IoT device that needs to transmit data; A joint optimization problem is established to minimize the over-the-air mean square error (RMSE). Under the constraints of energy harvesting and active RIS, the transmit covariance matrix of the hybrid access point, the receive beam vector, the active RIS uplink and downlink reflection matrix, and the transmit power coefficient of the IoT device are jointly optimized to minimize the MSE. The beamforming behavior is modeled as a Markov decision process, solved using a deep reinforcement learning algorithm based on DDPG, and an exponentially weighted moving average mechanism is introduced to improve training convergence. The performance of the beamforming behavior is verified based on the simulation platform. By adjusting parameters such as transmit power, number of RIS units, and number of devices, the data aggregation optimization effect of the beamforming behavior under different parameters is determined.

[0008] Furthermore, when constructing an active RIS-assisted wireless power supply aerial computing system, it includes: Configure a M A hybrid access point with three antennas for transmitting energy to multiple IoT devices simultaneously. From hybrid access points to IoT devices k The beamforming vector of . represents the hybrid access point transmit covariance matrix, and it obeys the maximum transmit power constraint at the hybrid access point .

[0009] Configure a N The active reconfigurable smart surface consists of a number of reflector units. Each reflector unit is equipped with the function of independently adjusting the phase and amplitude to regulate the uplink and downlink channels respectively, and dynamically adjust the amplitude and phase of the reflector unit according to the wireless channel environment. The diagonal beamforming matrix of the downlink active RIS is expressed as ,in is the active RIS reflection coefficient vector in the downlink phase.

[0010] The diagonal beamforming matrix of the uplink active RIS is expressed as ,in, is the active RIS reflection coefficient vector in the uplink phase.

[0011] Configuration K A single-antenna IoT device equipped with a wireless energy receiving module. The IoT device can use energy harvesting technology to collect energy from downlink wireless energy transmission, and then use the collected energy for air computing to achieve fast data transmission.

[0012] Next, we build a downlink energy beamforming transmission model and an uplink data aggregation beamforming transmission model. In a time slot, the entire system transmission can be divided into two parts: the downlink transmission phase and the uplink transmission phase. In the downlink transmission phase, IoT devices use energy harvesting technology to obtain energy. k The energy collected during wireless energy transmission can be expressed as in Represents IoT devices The energy conversion efficiency, , They represent the channel parameters from the hybrid access point to the active RIS and from the active RIS to the IoT device, Indicates the downlink wireless energy transmission time.

[0013] In the uplink transmission phase, based on the superposition of wireless multiple access channels, the hybrid access point can directly calculate the column function of the signal sent by the IoT device while reducing the communication overhead. The specific form is ,function 、 Represent the pre-processing function and post-processing function respectively.

[0014] The collinear function that needs to be calculated in the present invention is specifically: IoT devices The collected data is represented as , is represented as the signal sent by IoT device k, where Represents a preprocessing function.

[0015] Furthermore, when establishing a joint optimization problem with the goal of minimizing the mean square error of airborne calculations, the following steps are involved: Construct the objective function and determine the mean square error between the objective function of data aggregation and the estimation function as the optimization objective function: Clarify optimization parameters and jointly optimize the transmit covariance matrix of hybrid access points With the aggregated beam vector , Active RIS downlink and uplink reflection matrix 、 , and the transmission power coefficient of IoT devices . Configure constraints, including: Energy collection constraints on IoT devices , Hybrid access point maximum transmit power constraint Amplitude constraints of the downlink and uplink active RIS , With phase constraint 、 ; Furthermore, the specific optimization problem is in represents the downlink active RIS reflection phase shift matrix; represents the uplink active RIS phase shift matrix; represents the hybrid access point transmit covariance matrix; Indicates the device transmission power coefficient; represents the base station receiving beam vector; and Represent the downlink and uplink active RIS phase shift matrix amplitude constraints, and Represent the phase constraints of the downlink and uplink active RIS phase shift matrices respectively. is the maximum transmit power constraint of the hybrid access point, Energy harvesting constraints for IoT device k.

[0016] Furthermore, when the optimization problem P1 is converted into a Markov decision process, it includes: Construct the state space of the system, and express the channel state information between the hybrid access point and each IoT device, the configuration parameters of the active RIS reflection unit, the current transmission power and other information as a state vector to characterize the current environment state. At time t, the state space is expressed as in, Indicates the channel between the hybrid access point and the active RIS. Represents the channel between the active RIS and the IoT device. represents the hybrid access point transmit covariance matrix, represents the uplink hybrid access point receive beam vector, Indicates the transmit power vector of the IoT device. 、 denote the active RIS phase shift matrices for downlink and uplink respectively, Indicates the Time slot actions; Construct an action space, take the transmit covariance matrix and receive beam vector of the hybrid access point, the uplink and downlink reflection matrix of the active reconfigurable smart surface, and the transmit power coefficient parameters of each IoT device as continuous action variables, and learn the optimal action selection in each state based on the actor-critic network. Since the neural network requires real number input, it is decomposed into real and imaginary parts. The action at time t is defined as Which contains the downlink transmit covariance matrix , IoT device transmission power vector , uplink hybrid access point receive beam vector , downlink and uplink active RIS phase shift matrix 、 ; Design a reward function related to air computing performance, with minimizing the mean square error of air computing as the main goal, and set a comprehensive reward function by combining power constraints and MSE minimization indicators. The reward function at time t is in Control reward decay rate, exponential function Ensure that the reward increases when the MSE decreases. The second term penalizes the transmission power of IoT devices. Exceeding the maximum allowable power , applying a negative reward if the constraint is violated. is the discount factor; Furthermore, we build a deep reinforcement learning structure based on DDPG, including the actor network , critics network Target actor network , target critic network and experience replay buffer , where the actor network is configured to generate the optimal action of the hybrid access point transmit covariance matrix and receive beam, the active reconfigurable smart surface reflection matrix and the device transmit power coefficient, and the critic network is configured to evaluate the state action value Q of the current strategy under a specific state, and the error loss between the predicted Q value and the target value is used to update the critic network, which is defined as in, represents the number of training data samples, and Respectively represent The state and action of a sample at time t, Indicates the The target state-action Q value of a sample is defined as The specific network parameter update of DDPG starts from the parameters of the critic network, which is expressed as in, and denote the parameters of the critic network at time t+1 and t, respectively. represents the learning rate of the critic network, represents the gradient of the loss function with respect to the parameters of the critic network.

[0017] Subsequently, the actor network is updated as follows in, and denote the parameters of the actor network at time t + 1 and t, respectively. Represents the learning rate of the actor network. represents the gradient of the target critic network with respect to the action, represents the gradient of the actor network with respect to its own parameters.

[0018] Then the target actor and critic networks are updated based on the soft target instead of directly copying the weights of the main network, which are represented as , in Indicates the soft update factor.

[0019] In the deep reinforcement learning training process, an exponentially weighted moving average mechanism is used, which is expressed as in To adjust the smoothing coefficient to balance the impact of old rewards on learning, thereby improving the stability and training efficiency of the DDPG algorithm.

[0020] Furthermore, the performance verification of beamforming behavior based on the simulation platform includes: Build a system simulation model that includes hybrid access points, active reconfigurable smart surfaces, and multiple IoT devices. This model supports joint modeling of uplink and downlink communication links, and can simulate electromagnetic wave reflection paths, energy harvesting, and data aggregation and transmission processes. A simulation platform parameter configuration module is set up, and the transmit power level, number of RIS reflector units, and number of IoT devices in the system are adjusted according to the simulation platform parameter configuration module. Multiple sets of beamforming simulation environments with different parameters are established. The data aggregation performance results under the influence of different parameters are compared horizontally, and the stability and convergence of the proposed deep reinforcement learning beamforming method under different transmit powers are verified.

[0021] Compared with the prior art, the present invention has the following beneficial effects: 1. A beamforming method for active RIS-assisted wireless powered airborne computing based on deep reinforcement learning was developed. An active RIS-assisted wireless powered airborne computing network architecture was constructed. This can achieve efficient data aggregation of IoT devices in dynamic wireless environments by optimizing downlink energy beamforming and uplink information transmission beamforming.

[0022] 2. This method transforms the optimization problem into a Markov decision process and uses a deep reinforcement learning algorithm based on DDPG to train and update the system strategy. This algorithm has strong adaptability and the ability to process continuous action spaces. Especially when the number of users and antennas is large, this method achieves better wireless data aggregation than traditional computational methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Various other advantages and benefits will become apparent to those skilled in the art upon reading the detailed description of the preferred embodiment below. The accompanying drawings are for illustration purposes only and are not to be considered as limiting the present invention. The same reference symbols are used throughout the drawings to represent the same components. In the drawings: Figure 1A flowchart illustrating a specific implementation of beamforming in a wireless powered airborne computing system assisted by an active RIS according to an embodiment of the present invention; Figure 2 A schematic diagram of a model provided by an embodiment of the present invention; Figure 3 This is a structural diagram of the DDPG algorithm proposed in an embodiment of the present invention; Figure 4 A schematic diagram of the convergence of the deep deterministic policy gradient algorithm provided by an embodiment of the present invention; Figure 5 A schematic diagram showing the effect of the number of RIS basic units on MSE provided in an embodiment of the present invention; Figure 6 Schematic diagram of the effect of different learning rates on MSE of the algorithm provided in an embodiment of the present invention; Figure 7 This is a schematic diagram of the impact of different transmit powers on MSE provided by an embodiment of the present invention.

[0024] Figure 8 This is a schematic diagram of the impact of the number of IoT devices on MSE provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0025] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art. It should be noted that, unless there is a conflict, the embodiments of the present disclosure and the features in the embodiments can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.

[0026] like Figure 1 As shown, in some embodiments of the present application, this embodiment provides a beamforming method for active RIS-assisted wireless powered air computing based on deep reinforcement learning, including: Step S100: constructing an active RIS-assisted wireless power supply airborne computing system, wherein the wireless power supply airborne computing system includes a hybrid access point, an active RIS, and multiple IoT devices.

[0027] Specifically, when building an active RIS-assisted wireless power supply airborne computing system, it includes: configuring multiple IoT devices, where the IoT devices first collect energy through downlink wireless energy transmission, then use the energy to collect data, pre-process the data, and then use airborne computing technology to aggregate and transmit the data; configuring a hybrid access point, where the hybrid access point is configured to transmit energy to multiple IoT devices at the same time, and perform fusion calculations on the data transmitted by the IoT devices; configuring an active RIS, where the active RIS is configured with multiple reflection units, and each reflection unit is configured with the function of independently adjusting the phase and amplitude; the reflection unit is connected to the controller, and the reflection unit can reduce the wireless path fading between the hybrid access point and the IoT device.

[0028] The transmission process includes two stages: downlink wireless power transmission (WPT) and uplink air computing (AirComp). Figure 2 The wireless communication system model consists of a hybrid access point, an active RIS, and K IoT devices. Specifically, the hybrid access point is equipped with N antennas and the number of active RIS reflection units is M. A quasi-static channel model is considered, where the uplink and downlink channel states remain unchanged during each energy transmission and over-the-air calculation process.

[0029] Specifically, wireless powered air computing runs in two phases within a fixed time slot T, i.e., users harvest energy in the first WPT phase and then use the harvested energy for wireless data aggregation in the second AirComp phase. We denote the duration of the WPT phase and the AirComp phase as and ,satisfy In the downlink WPT phase, the hybrid access point acts as a power transmitter and uses wireless power transmission to power IoT devices with the assistance of active RIS. , , represents the channel from the hybrid access point to the active RIS, and from the active RIS to the IoT device. The diagonal beamforming matrix of the active RIS can be expressed as There is a reflection coefficient vector ,in , Represent the amplitude and phase of each active RIS unit respectively.

[0030] In addition, IoT devices Has a certain efficiency in the energy conversion process The energy beam generated by the hybrid access point is usually highly concentrated, and the energy collected from the side lobes of the unintended beam is negligible, so the energy collected by the IoT device k during the WPT process can be expressed as ,in represents the beamforming vector from the hybrid access point to IoT device k. Assume represents the hybrid access point transmit covariance matrix, and it obeys the maximum transmit power constraint at the hybrid access point .

[0031] In the uplink air computing process, the hybrid access point acts as an access point and a fusion center (FC) at the current stage. Assuming that all the collected energy is used for air computing, in a specific time slot, the IoT device The collected data is represented as ,use is represented as the signal sent by IoT device k, where Represents the preprocessing function. And they are independent of each other and satisfy ,in Based on the superposition of wireless multiple access channels, FC can directly calculate the column function of the signal sent by the IoT device while reducing the communication overhead. The specific form is ,function 、 Represent the pre-processing function and post-processing function respectively.

[0032] In order to achieve fast functional processing of data, the column line function that needs to be calculated in the present invention is specifically: IoT devices The collected data is represented as , is represented as the signal sent by IoT device k, where Represents a preprocessing function.

[0033] The energy that IoT device k can use to upload data is ,make represents the scalar sent by IoT device k, and represents the transmit power vector. Since the transmit power cannot exceed the collected energy, the transmit power of IoT device k should be subject to the following constraints: In addition, IoT devices The transmitted signal at can be expressed as The diagonal beamforming matrix of active RIS is expressed as , with a reflection coefficient vector , where the parameters , Represent the amplitude and phase of each active RIS unit respectively.

[0034] The cascaded channel response from IoT device k to the hybrid access point is expressed as , the received signal at FC can be expressed as: ,in represents the thermal noise introduced by the active RIS, Represents the additive white Gaussian noise (AWGN) on the hybrid access point side.

[0035] By receiving the beam vector and multiply it by the averaging factor It can realize the reception of signals The reception, that is In this paper, the mean square error between the estimation function and the objective function is used as the metric for air calculation, which is expressed as , in, .

[0036] Step S200: Establish a joint optimization problem with the goal of minimizing the mean square error of the over-the-air calculation. Under conditions such as energy collection constraints and active RIS constraints, jointly optimize the hybrid access point transmit covariance matrix, receive beam vector, active RIS uplink and downlink reflection matrix, and device transmit power coefficient.

[0037] Specifically, establishing a joint optimization problem with the goal of minimizing the mean squared error of over-the-air calculations includes: constructing an objective function, determining the mean squared error between the network estimation function and the objective function as the optimization objective function; configuring constraints, where the constraints include: energy collection constraints on the IoT device side, maximum transmit power constraints on the hybrid access point, and reflection amplitude and phase constraints on the active RIS; under the above constraints, constructing a joint optimization problem with the goal of minimizing the mean squared error of over-the-air calculations.

[0038] Specifically, based on the above, a joint optimization problem is established to minimize the mean square error of the system air calculation. Specifically, the hybrid access point transmission covariance matrix is ​​jointly optimized under the constraints of energy collection constraints and active RIS constraints. , the transmission power of IoT devices , the active RIS downlink phase shift matrix , , and the receive beam vector of the hybrid access point To minimize the mean square error. The optimization goal is specifically: in represents the downlink active RIS reflection phase shift matrix; represents the uplink active RIS phase shift matrix; represents the hybrid access point transmit covariance matrix; Indicates the device transmission power coefficient; represents the base station receiving beam vector; and Represent the downlink and uplink active RIS phase shift matrix amplitude constraints, and Represent the phase constraints of the downlink and uplink active RIS phase shift matrices respectively. is the maximum transmit power constraint of the hybrid access point, is the energy harvesting constraint.

[0039] It should be noted that, in general, solving optimal goals in this invention typically involves transforming the problem into an iterative optimization problem and applying relevant algorithms. However, in wireless communication system models, the environment is time-varying and highly complex, making traditional model-based optimization algorithms incapable of handling such optimization problems. To address this technical issue, this invention transforms the optimization problem into a Markov process model based on the downlink wireless energy transmission and uplink over-the-air computation processes of active RIS-assisted transmission. In deep reinforcement learning, an agent is an entity capable of perceiving the environment, making decisions, and taking actions. It continuously learns through interaction with the environment, gradually mastering the optimal behavioral strategy to achieve a specific goal.

[0040] Step S300: Model the beamforming behavior as a Markov decision process, solve it based on a deep deterministic policy gradient reinforcement learning algorithm, and introduce an exponentially weighted moving average mechanism to improve training convergence.

[0041] Specifically, when transforming the wave optimization problem P1 into a Markov decision process, the system's state space is constructed, integrating information such as the channel state between the hybrid access point and IoT devices, the reflector configuration parameters of the active reconfigurable smart surface, and the current transmit power allocation into a state vector. Simultaneously, an action space is constructed, using the transmit covariance matrix of the hybrid access point, the uplink and downlink reflectance matrices of the active reconfigurable smart surface, the transmit power coefficients of each device, and the receive beam vector parameters as continuous action variables. An actor network is used to learn the optimal action for each state. Furthermore, a reward function is designed that is correlated with over-the-air computation performance, with the primary objective being to minimize the mean squared error of over-the-air computation. A comprehensive reward function is also formulated in conjunction with power constraints. A deep reinforcement learning architecture consisting of an actor network and a critic network is constructed using a DDPG-based deep reinforcement learning algorithm. The actor network generates the optimal action for the hybrid access point transmit covariance matrix, the active reconfigurable smart surface reflectance matrix, and the device transmit power parameters, while the critic network evaluates the expected reward of the current strategy in a specific state. Using an experience replay mechanism, we store the states, actions, rewards, and next-state information generated during the interaction process and randomly sample them during training to improve sample independence and learning stability. Furthermore, using an exponentially weighted moving average mechanism, we smooth the policy output and cumulative reward curves to improve training convergence.

[0042] Specifically, the optimization problem in the active RIS-assisted wireless power supply air computing scenario is converted into a Markov decision process and solved using a deep reinforcement learning algorithm based on DDPG. The main elements of the deep reinforcement learning model based on the DDPG algorithm are defined as follows: (1) State: At time t, the state space is represented as in, Indicates the channel between the hybrid access point and the active RIS. Represents the channel between the active RIS and the IoT device. represents the hybrid access point transmit covariance matrix, represents the uplink hybrid access point receive beam vector, Indicates the transmit power vector of the IoT device. 、 denote the active RIS phase shift matrices for downlink and uplink respectively, Indicates the Time slot action.

[0043] (2) Action: At the tth training step, the action space includes the downlink transmit covariance matrix , IoT device transmission power vector , uplink hybrid access point receive beam vector , downlink and uplink active RIS phase shift matrix 、 Since neural networks require real-valued input, we decompose it into real and imaginary components, and then the action at time step t is defined as (3) Reward: The optimization goal is to minimize the MSE to guide the DDPG algorithm to learn the optimal strategy, and MSE is used as the main optimization metric. In addition, a monotonically decreasing function is used to map the MSE to a specific reward value range. Specifically, the reward formula at time step t is in Control reward decay rate, exponential function Ensure that the reward increases when the MSE decreases. The second term penalizes the user's transmission power Exceeding the maximum allowable power , applying a negative reward if the constraint is violated. is the discount factor.

[0044] It should be noted that in this application, DDPG is an algorithm widely used in the field of deep reinforcement learning, proposed by Lillicrap et al. in 2015. DDPG aims to address some of the challenges of policy gradient methods in continuous action spaces, such as sample efficiency and ease of implementation. Its core concept is to combine the advantages of policy gradient methods and Q-learning by using two neural networks (one for the policy and the other for the value function) to learn the optimal policy. Furthermore, the algorithm introduces an exponentially weighted moving average (EMA) to smooth the model weights and calculate the historical average weight after each update, thereby improving the model's generalization ability. DDPG uses a soft update method to update the target network weights, which can reduce fluctuations during training and improve learning stability. Therefore, the algorithm demonstrates good performance and stability when dealing with problems in continuous action spaces.

[0045] The optimization problem is solved based on the deep reinforcement learning algorithm DDPG. Specifically, DDPG consists of a main network and a target network with the same structure but different parameters. The specific network structure of DDPG is as follows: Figure 2 As shown in the figure. The main components of DDPG are as follows: Actor Network , critics network Target actor network , target critic network and experience replay buffer The main goal of DDPG is to train the actor to learn the optimal policy by maximizing the critic’s estimated state-action value Q. During training, the loss is calculated as the error between its predicted Q value and the target value. The loss is used to update the critic network and is defined as in, represents the number of training data samples, and Respectively represent The state and action of a sample at time t, Indicates the The target state-action Q value of a sample is defined as The specific network parameter update of DDPG starts from the parameters of the critic network, which is expressed as in, and denote the parameters of the critic network at time t+1 and t, respectively. represents the learning rate of the critic network, represents the gradient of the loss function with respect to the parameters of the critic network.

[0046] Subsequently, the actor network is updated as follows in, and denote the parameters of the actor network at time t + 1 and t, respectively. Represents the learning rate of the actor network. represents the gradient of the target critic network with respect to the action, represents the gradient of the actor network with respect to its own parameters.

[0047] Then the target actor and critic networks are updated based on the soft target instead of directly copying the weights of the main network, which are represented as , in Indicates the soft update factor.

[0048] The specific algorithm steps are as follows: (1) Input: channel matrix , (2) Output: hybrid access point transmit energy beamforming matrix and receive beam vector, downlink / uplink active RIS reflection phase shift matrix, IoT device transmit power vector.

[0049] (3) Initialization: Initialize the experience replay pool , the size is . Initialize training actor network parameters , critics network Target actor network and target critic network Initialize the hybrid access point transmit covariance matrix and the receive beam vector , downlink, uplink active RIS reflection phase shift matrix 、 , IoT device transmission power vector ; (4) In each iteration of the loop phase, obtain the channel estimation operation number m and the state iteration number n. First, determine whether the channel estimation operation number m is 0: If m=0, the deep deterministic policy gradient algorithm ends; If m≠0, continue to determine whether the state iteration number is 0: If n=0, set m=m-1, and then re-determine whether m is 0; If n≠0, initialize the state The actor network then predicts the state at time j Next action , and perform the action to the environment, and the environment feedback reward Then, transfer to the next state , and then store the experience in the playback buffer, that is, store the quadruple ; Then update the current state for , and then continue to sample new interaction data until the experience pool storage reaches the preset capacity C. When the experience pool collects a sufficient number of samples, it uses random sampling of small batches of data Extract samples from the experience pool, calculate the target Q-value using the target network, calculate the critic network loss and update the critic network, calculate the policy gradient and update the actor network, and then soft-update the target network. Update the iteration count: If the number of state iterations n≠0, set n=n-1 and re-enter the agent interaction and data storage step; if n=0, decrement m and decide whether to continue training.

[0050] (5) Iterate continuously until the loop ends. It should be noted that in order to improve the stability and training efficiency of the DDPG algorithm, the exponentially weighted moving average is used to calculate the average reward, which is specifically expressed as .in To adjust the smoothing coefficient to balance the impact of old rewards on learning.

[0051] Step S400: verify the performance of the beamforming behavior based on the simulation platform, and determine the optimization effect of data aggregation by adjusting the parameters such as the transmit power, the number of RIS units, and the number of devices.

[0052] Specifically, in order to verify the effectiveness of the beamforming method of the active RIS-assisted wireless power supply air computing system proposed in the present invention, some parameters are simulated to verify the effectiveness of the method of the present invention. In the system, the maximum transmission power of the hybrid access point is set to , the initial energy of IoT devices is set to 0 joules, and the energy conversion efficiency . K IoT devices are 0 m in height and are evenly distributed on a circle with a radius of 10 m and a center of (150, 20, 0). The active RIS and hybrid access points are located at (90, 0, 20) and (0, 30, 5), respectively, and the line-of-sight link between the IoT devices and the hybrid access points is blocked by obstacles. This embodiment uses three comparison methods for comparison: Passive RIS-AO, Active RIS-AO, and Passive RIS-DDPG, where Passive RIS represents passive RIS and AO represents alternating optimization algorithm.

[0053] Figure 4 This demonstrates the convergence of the DDPG algorithm at different transmit powers. Initially, all curves rise rapidly, indicating that the agent is continuously learning and optimizing its policy. As the step size increases, the reward values ​​gradually converge, and the policy stabilizes. Furthermore, the fluctuations in the instantaneous reward curves decrease as training progresses, further verifying the convergence of the policy.

[0054] like Figure 5 As shown in the figure, the MSE of all schemes decreases as the number of RIS reflectors increases. This is because more reflectors allow the RIS to reflect more signals, enabling more precise signal phase adjustment, resulting in lower MSE and better quality of service. Compared to active RIS, passive RIS lacks signal amplification and can only passively reflect signals, resulting in a larger MSE. Furthermore, compared to traditional alternating optimization algorithms, the DDPG algorithm achieves a smaller MSE, demonstrating that the proposed DDPG algorithm can significantly improve transmission quality in complex environments.

[0055] In deep reinforcement learning (DRL), the choice of network parameters is crucial to the performance of the algorithm, especially the learning rate, which has a significant impact on the convergence and stability of the neural network. In order to deeply analyze the impact of the learning rate on the DDPG algorithm, we conducted experiments to study the relationship between the average reward and the time step under different learning rates. The results are as follows: Figure 6 As shown in the figure. In the experiment, the learning rate is set to 0.005, 0.001, and 0.0001 respectively. The experimental results show that when the learning rate is set to 0.0001, the DDPG algorithm can achieve the best performance.

[0056] Figure 7 The impact of the maximum transmission power at the base station on the MSE achieved by the algorithm was evaluated. As can be seen from the figure, at the base station, the MSE decreases with increasing maximum transmission power. This is because IoT devices collect more energy for effective transmit beamforming, thereby reducing the MSE. Furthermore, the MSE of the active RIS approach is significantly reduced compared to the passive RIS solution. This further demonstrates the importance of optimizing the RIS phase shift and amplitude in reducing computational error. Furthermore, the deep reinforcement learning-based approach achieves an 8% reduction in MSE compared to the alternating optimization algorithm, demonstrating that the proposed deep reinforcement learning algorithm can further reduce the MSE of over-the-air computations.

[0057] In general, the present invention proposes a DDPG-based deep reinforcement learning to achieve efficient joint beamforming for an active RIS-assisted wireless power supply aerial computing system to improve the accuracy of IoT data aggregation. The method of the present invention studies how to establish an optimization model suitable for a dynamic environment, which involves considering factors such as dynamic changes in the environment, interference and noise, and incorporating them into the model. The present invention will use deep reinforcement learning as the basis of the optimization algorithm and study how to design a deep reinforcement learning framework suitable for joint beamforming of wireless power supply aerial computing systems, including state representation, action selection, reward design and value function estimation. The present invention designs a reasonable environmental interaction mechanism to enable the optimization parameters to interact with the dynamic environment. Through interaction with the environment, iterative updates of the strategy will be achieved to gradually optimize the accuracy of IoT data aggregation.

[0058] Those skilled in the art will appreciate that the embodiments of the present application may be expressed as methods, systems, or computer program products, and may be implemented entirely in hardware, entirely in software, or as a combination of hardware and software. Furthermore, the present application may also be implemented as a computer program product embodied on one or more computer-readable storage media containing computer-usable program code (e.g., disk storage, CD-ROM, optical storage, etc.). The description of this application refers to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present application. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations thereof, may be implemented using computer program instructions. These instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine that, by executing the instructions, implements the functions specified in the flowcharts or block diagrams.

[0059] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the memory generate a device that implements the specified functions in the flowchart or block diagram. It should be pointed out that the above embodiments are only used to illustrate the technical solution of the present invention and are not intended to limit it. Although the present invention has been described in detail through the above embodiments, those skilled in the art should understand that the specific embodiments of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the present invention. Any such modification or equivalent replacement should be included in the scope of protection of the claims of the present invention.

Claims

1. A beamforming method for active RIS-assisted wireless powered air computing based on deep reinforcement learning, characterized in that: include: S1. Construct a downlink energy beamforming transmission model and an uplink data aggregation beamforming transmission model for an active RIS-assisted wireless power supply airborne computing system. S2, with the goal of minimizing the mean square error of air calculation, considering the energy collection constraints and active RIS constraints, establish a joint optimization hybrid access point transmission covariance matrix , receive beam vector , Active RIS downlink reflective matrix 、 and IoT device transmission power coefficient Joint optimization problem of S3. Convert the optimization problem into a Markov decision process (MDP). Design a deep reinforcement learning algorithm based on DDPG to solve the optimization problem, and introduce an exponentially weighted moving average mechanism to improve training convergence. S4. Verify the performance of beamforming based on the simulation platform, and determine the accuracy of over-the-air data aggregation under different parameters such as transmit power, number of RIS units, and number of IoT devices.

2. The beamforming method for active RIS-assisted wireless powered air computing based on deep reinforcement learning as claimed in claim 1, characterized in that: In step S1, a downlink energy beamforming transmission model and an uplink data aggregation beamforming transmission model of an active RIS-assisted wireless power supply airborne computing system are constructed, including the following steps: First, an active RIS-assisted wireless power-supply aerial computing system is constructed. Configure a M A hybrid access point with three antennas for transmitting energy to multiple IoT devices simultaneously. From hybrid access points to IoT devices k The beamforming vector of . represents the hybrid access point transmit covariance matrix, and it obeys the maximum transmit power constraint at the hybrid access point . Configure a N The active reconfigurable smart surface consists of a number of reflector units. Each reflector unit is equipped with the function of independently adjusting the phase and amplitude to regulate the uplink and downlink channels respectively, and dynamically adjust the amplitude and phase of the reflector unit according to the wireless channel environment. The diagonal beamforming matrix of the downlink active RIS is expressed as ,in is the active RIS reflection coefficient vector in the downlink phase. The diagonal beamforming matrix of the uplink active RIS is expressed as ,in, is the active RIS reflection coefficient vector in the uplink phase. Configuration K A single-antenna IoT device equipped with a wireless energy receiving module. The IoT device can use energy harvesting technology to collect energy from downlink wireless energy transmission, and then use the collected energy for air computing to achieve data aggregation. Furthermore, a downlink energy beamforming transmission model and an uplink data aggregation beamforming transmission model are constructed. In a time slot, the entire system transmission can be divided into two parts: the downlink transmission phase and the uplink transmission phase. In the downlink transmission phase, IoT devices use energy harvesting technology to obtain energy. k The energy collected during wireless energy transmission can be expressed as in Represents IoT devices The energy conversion efficiency, , They represent the channel parameters from the hybrid access point to the active RIS and from the active RIS to the IoT device, Indicates the downlink wireless energy transmission time. In the uplink transmission phase, based on the superposition of wireless multiple access channels, the hybrid access point can directly calculate the column function of the signal sent by the IoT device while reducing the communication overhead. The specific form is ,function 、 Represent the pre-processing function and post-processing function respectively; The collinear function that needs to be calculated in the present invention is specifically: IoT devices The collected data is represented as , is represented as the signal sent by IoT device k, where Represents a preprocessing function.

3. The beamforming method for active RIS-assisted wireless powered air computing based on deep reinforcement learning as claimed in claim 1, characterized in that: In step S2, establishing a joint optimization problem with the goal of minimizing the mean square error of the airborne calculation includes: Construct the objective function and determine the mean square error between the objective function of data aggregation and the estimation function as the optimization objective function: Clarify optimization parameters and jointly optimize the transmit covariance matrix of hybrid access points and the receive beam vector , Active RIS downlink and uplink reflection matrix 、 , and the transmission power coefficient of IoT devices . Configure constraints, including: Energy collection constraints on IoT devices , Hybrid access point maximum transmit power constraint Amplitude constraints of the downlink and uplink active RIS , With phase constraint 、 ; The specific optimization problem is in represents the downlink active RIS reflection phase shift matrix; represents the uplink active RIS phase shift matrix; represents the hybrid access point transmit covariance matrix; Indicates the device transmission power coefficient; represents the hybrid access point receiving beam vector; and Represent the downlink and uplink active RIS phase shift matrix amplitude constraints, and Represent the phase constraints of the downlink and uplink active RIS phase shift matrices respectively. is the maximum transmit power constraint of the hybrid access point, Energy harvesting constraints for IoT device k.

4. The beamforming method for active RIS-assisted wireless powered air computing based on deep reinforcement learning according to claim 1, characterized in that: In step S3, the optimization problem P1 is converted into a Markov decision process, including: Construct the state space of the system, and express the channel state information between the hybrid access point and each IoT device, the configuration parameters of the active RIS reflection unit, the current transmission power and other information as a state vector to characterize the current environment state. At time t, the state space is expressed as in, Indicates the channel between the hybrid access point and the active RIS. Represents the channel between the active RIS and the IoT device. represents the hybrid access point transmit covariance matrix, represents the uplink hybrid access point receive beam vector, Indicates the transmit power vector of the IoT device. and represents the active RIS phase shift matrix for downlink and uplink, Indicates the Time slot action. Construct an action space, take the transmit covariance matrix and receive beam vector of the hybrid access point, the uplink and downlink reflection matrix of the active reconfigurable smart surface, and the transmit power coefficient parameters of each IoT device as continuous action variables, and learn the optimal action selection in each state based on the actor-critic network. Since the neural network requires real number input, it is decomposed into real and imaginary parts. The action at time t is defined as Which contains the downlink transmit covariance matrix , IoT device transmission power vector , uplink hybrid access point receive beam vector , downlink and uplink active RIS phase shift matrix 、 . Design a reward function related to air computing performance, with minimizing the mean square error of air computing as the main goal, and set a comprehensive reward function by combining power constraints and MSE minimization indicators. The reward function at time t is in Control reward decay rate, exponential function Ensure that the reward increases when the MSE decreases. The second term penalizes the transmission power of IoT devices. Exceeding the maximum allowable power , applying a negative reward if the constraint is violated. is the discount factor.

5. The beamforming method for active RIS-assisted wireless powered air computing based on deep reinforcement learning as claimed in claim 1, characterized in that: In step S3, the solution is solved using the DDPG-based reinforcement learning algorithm and the exponentially weighted moving average mechanism is introduced to improve training convergence, including: Build a deep reinforcement learning structure based on DDPG, including actor network , critics network Target actor network , target critic network and experience replay buffer , where the actor network is configured to generate the optimal action of the hybrid access point transmit covariance matrix and receive beam, the active reconfigurable smart surface reflection matrix and the device transmit power coefficient, and the critic network is configured to evaluate the state action value Q of the current strategy under a specific state, and the error loss between the predicted Q value and the target value is used to update the critic network, which is defined as in, represents the number of training data samples, and Respectively represent The state and action of a sample at time t, Indicates the The target state-action Q value of a sample is defined as The specific network parameter update of DDPG starts from the parameters of the critic network, which is expressed as in, and denote the parameters of the critic network at time t+1 and t, respectively. represents the learning rate of the critic network, represents the gradient of the loss function with respect to the parameters of the critic network. Subsequently, the actor network is updated as follows in, and denote the parameters of the actor network at time t + 1 and t, respectively. Represents the learning rate of the actor network. represents the gradient of the target critic network with respect to the action, represents the gradient of the actor network with respect to its own parameters. Then the target actor and critic networks are updated based on the soft target instead of directly copying the weights of the main network, which are represented as ， in Indicates the soft update factor. In the deep reinforcement learning training process, an exponentially weighted moving average mechanism is used, which is expressed as in To adjust the smoothing coefficient to balance the impact of old rewards on learning, thereby improving the stability and training efficiency of the DDPG algorithm.

6. The beamforming method for active RIS-assisted wireless powered air computing based on deep reinforcement learning as claimed in claim 1, characterized in that: In step S4, the performance verification of the beamforming behavior based on the simulation platform includes: Build a system simulation model that includes hybrid access points, active reconfigurable smart surfaces, and multiple IoT devices. This model supports joint modeling of uplink and downlink communication links, and can simulate electromagnetic wave reflection paths, energy harvesting, and data aggregation and transmission processes. A simulation platform parameter configuration module is set up, and the transmit power level, number of RIS reflector units, and number of IoT devices in the system are adjusted according to the simulation platform parameter configuration module. Multiple sets of beamforming simulation environments with different parameters are established. The data aggregation performance results under the influence of different parameters are compared horizontally, and the stability and convergence of the proposed deep reinforcement learning beamforming method under different transmit powers are verified.

Images