Aerial-RIS-assisted wireless power supply communication network energy efficiency optimization method and system

Abstract

The invention belongs to but is not limited to the technical field of wireless communication, and discloses an Aerial-RIS-assisted wireless power supply communication network energy efficiency optimization method based on deep reinforcement learning, and the method comprises the steps: deploying an RIS on an unmanned aerial vehicle as a relay, and constructing a system architecture of an Aerial-RIS-assisted wireless power supply communication network; according to an Aerial-RIS-assisted WPCN system based on two modes of TDMA and DTDMA, an RIS phase shift matrix, a TDMA / DTDMA time slot allocation ratio tau and an unmanned aerial vehicle trajectory are jointly optimized; in order to improve the energy efficiency of the system, on the basis of a deep reinforcement learning algorithm, the energy efficiency is gradually maximized by observing rewards and correspondingly iteratively adjusting parameters of a DRL algorithm, so that an optimal strategy is obtained. According to the method, the problem of low efficiency of wireless power transmission due to transmission distance, obstacles and the like in a complex environment is solved, and the average throughput and the energy efficiency are improved.

Description

Technical Field

[0001] The present invention belongs to but is not limited to the field of wireless communication technology, and in particular relates to an Aerial-RIS-assisted wireless power supply communication network energy efficiency optimization method based on deep reinforcement learning. Background Art

[0002] As communication demands increase, the energy consumption of wireless devices is also increasing, and energy-constrained devices are gradually becoming a bottleneck restricting the quality of network services. How to charge these energy-constrained devices in a green and low-carbon way is an urgent issue. To solve this problem, wireless power communication networks have emerged, in which hybrid access points transmit energy to energy-constrained devices on the downlink, and wireless devices use the collected energy to transmit information in the uplink. This new technology provides a new way to achieve efficient and durable communication. In addition, considering the efficiency bottleneck of WPCN under the influence of limited transmission distance and obstacles, in order to seek a more efficient and economical solution, reconfigurable smart surfaces, as an innovative low-power, high-energy-efficiency emerging technology, are gradually entering people's field of vision and are widely favored.

[0003] Therefore, it is particularly important to propose an Aerial-RIS assisted wireless power communication network to improve the energy efficiency of the system. Summary of the invention

[0004] In view of the problems existing in the prior art, the present invention provides an Aerial-RIS-assisted wireless power supply communication network energy efficiency optimization method and system based on deep reinforcement learning.

[0005] The present invention is implemented as follows: a method for optimizing energy efficiency of wireless power supply communication network assisted by Aerial-RIS based on deep reinforcement learning, comprising the following steps:

[0006] Step 1: Deploy RIS on the drone as a relay to build the system architecture of the Aerial-RIS assisted wireless power supply communication network;

[0007] Step 2: Based on the Aerial-RIS assisted WPCN system of TDMA and DTDMA, the RIS phase shift matrix, TDMA / DTDMA time slot allocation ratio τ and UAV trajectory are jointly optimized;

[0008] Step three, based on the deep reinforcement learning algorithm, by observing the rewards and iteratively adjusting the parameters of the DRL algorithm accordingly, the energy efficiency is gradually maximized to obtain the optimal strategy.

[0009] Furthermore, in step 1, the UAV is used as a mobile carrier to carry the RIS device, and the communication link is optimized by flexibly moving in the air and adjusting the phase of the reflection element; the HAP is responsible for sending energy signals to the Aerial-RIS and receiving information signals from the Aerial-RIS; the ground node receives energy from the UAV and RIS, and uses the collected energy to transmit information.

[0010] Furthermore, in the step 2, an Aerial-RIS assisted WPCN system model based on TDMA and DTDMA is established, including a signal transmission model, an energy collection model, a communication model, etc.

[0011] Further, the system model based on TDMA:

[0012] The ground node receiving power at time slot n is:

[0013] E k [n] = η(τ k [n]P B )|g UK [n] H Θ[n]g BU [n]| 2 (1)

[0014] Where η is the energy collection efficiency, τ k [n] is the energy harvesting time slot allocation ratio, P B is the transmission power of HAP, g UK [n] is the channel gain between Aerial-RIS and the node, g BU [n] is the channel gain from HAP to each reflective element of RIS, and Θ[n] is the phase shift matrix of RIS.

[0015] The throughput of node k in time slot n can be expressed as:

[0016]

[0017] Where P T [n] is the transmission power of the kth node, where σ 2 is a noise signal that follows a Gaussian distribution

[0018] Further, the system model based on DTDMA:

[0019] The ground node receiving power at time slot n is:

[0020]

[0021] Where η is the energy collection efficiency.

[0022] The throughput of node k in time slot n can be expressed as:

[0023]

[0024] Further, the specific steps of step three are:

[0025] Input: system initial state, discount factor γ, learning rate, experience replay size N D , mini-batch size N B

[0026] Output: Best strategy

[0027] Initialization: actor network π(s|θ π ) and the critic network Q(s,a|θ Q ), target actor network π'(s|θ π′ ) and the target critic network Q'(s,a|θ Q' ), θ π '=θ π ,θ Q '=θ Q , experience playback buffer D, phase shift matrix Φ

[0028] for episode=1,N do:

[0029] Accept the current environment and get the initial state s(0);

[0030] Initialize a random noise

[0031] forn=1:Ndo

[0032] With status s n As input, get action a from the policy network n ;

[0033] Perform actions and get rewards and status n+1 ;

[0034] Will Stored in experience replay pool D;

[0035] Randomly draw a batch of samples from the experience replay pool D;

[0036] Calculate the target Q value;

[0037] Update the value network;

[0038] Use SGD to update the gradient of the policy network;

[0039] Update the policy network so that the actions of the policy network can better approximate the real actions;

[0040] Soft update of target network parameters of the policy network and value network;

[0041] Update Status n =s n+1

[0042] end for

[0043] end for

[0044] Another object of the present invention is to provide an Aerial-RIS-assisted wireless power supply communication network energy efficiency optimization method based on deep reinforcement learning, and an Aerial-RIS-assisted wireless power supply communication network energy efficiency optimization system based on deep reinforcement learning, comprising:

[0045] System architecture building module, deploying RIS on drones to act as a relay and building the system architecture of the Aerial-RIS assisted wireless power supply communication network;

[0046] Joint optimization module, based on the Aerial-RIS assisted WPCN system of TDMA and DTDMA, jointly optimizes the RIS phase shift matrix, TDMA / DTDMA time slot allocation ratio τ and UAV trajectory;

[0047] The parameter adjustment module, based on the deep reinforcement learning algorithm, gradually maximizes the energy efficiency by observing the rewards and iteratively adjusting the parameters of the DRL algorithm accordingly, thereby obtaining the optimal strategy.

[0048] Another object of the present invention is to provide a computer device, the computer device comprising a memory and a processor, the memory storing a computer program, and when the computer program is executed by the processor, the processor executes the steps of the Aerial-RIS-assisted wireless power supply communication network energy efficiency optimization method based on deep reinforcement learning.

[0049] Another object of the present invention is to provide a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor executes the steps of the Aerial-RIS-assisted wireless power supply communication network energy efficiency optimization method based on deep reinforcement learning.

[0050] Another object of the present invention is to provide an information data processing terminal, which includes the Aerial-RIS-assisted wireless power supply communication network energy efficiency optimization system based on deep reinforcement learning.

[0051] In combination with the above technical solutions and the technical problems solved, the advantages and positive effects of the technical solutions to be protected by the present invention are as follows:

[0052] First, in order to improve the communication energy efficiency of the entire system, the present invention provides an Aerial-RIS assisted wireless power supply communication network, and considers two time division multiple access schemes to achieve efficient parallel transmission of information and energy. At the same time, the present invention constructs a Markov decision model and transforms the optimization problem into a DRL model to overcome the problems of large amount of calculation and complex calculation of traditional methods such as SCA, so as to make the performance of the entire system better.

[0053] Aiming at the efficiency bottleneck of WPCN under the influence of limited transmission distance and obstacles, the present invention proposes an Aerial-RIS-assisted wireless power transmission and wireless information transmission architecture. In order to achieve fine division of the time dimension, time division multiple access technology is used to achieve efficient parallel transmission of information and energy through fine division of the time dimension. On the basis of this architecture, in order to fully tap the potential of time resources, the present invention also adopts dynamic TDMA technology, which adopts a double-layer time division structure to achieve refined management and utilization of resources.

[0054] Second, the technical solution of the present invention solves several key problems in the prior art in industrial applications and has achieved significant technical progress, which is specifically reflected in the following aspects:

[0055] 1. Improve energy efficiency and communication efficiency:

[0056] The existing wireless power supply communication network has major bottlenecks in energy efficiency and communication efficiency, especially in the scenarios of long-distance power supply and multi-node communication, with high energy loss and poor communication link stability. This invention introduces Aerial-RIS (Programmable Reflection Array) technology and uses drones to flexibly deploy RIS devices, thereby improving energy transmission and communication link efficiency while optimizing wireless channels. Through the deep reinforcement learning algorithm, the drone trajectory, time slot allocation and RIS phase shift matrix are further optimized, thereby greatly improving energy efficiency and communication performance.

[0057] 2. Solve the problem of insufficient coverage of traditional wireless power supply communication networks:

[0058] Traditional wireless power supply networks usually rely on fixed base stations for energy and signal transmission, which is limited by the limited coverage of fixed base stations, easily causing signal blind spots or insufficient energy supply. The present invention deploys RIS equipment on drones, and with the help of the flexible mobility of drones, adaptively adjusts the flight trajectory and signal reflection path according to the node position in a dynamic environment, breaking through the coverage limitations of fixed base stations and significantly expanding the coverage of wireless power supply and communication networks.

[0059] 3. Enhance communication stability in complex scenarios:

[0060] In complex environments, such as urban buildings, forests or mountainous areas, obstacles can affect signal propagation, leading to unstable communication or inefficient energy transmission. Through the present invention, the Aerial-RIS device can adjust the phase shift of the reflection element to optimize the channel state, bypass obstacles, and achieve seamless reflection of the signal, thereby improving the quality and stability of the communication link in complex environments.

[0061] 4. Dynamic optimization mechanism based on deep reinforcement learning:

[0062] Existing wireless power supply network optimization methods usually rely on static or preset algorithms, which are difficult to cope with the ever-changing environment and node requirements. Through deep reinforcement learning (DRL) technology, the present invention can monitor the network status in real time, dynamically adjust the drone trajectory, time slot allocation ratio and RIS phase shift matrix, and gradually achieve the optimal solution for energy efficiency. This dynamic adjustment capability enables the network to be adaptively optimized under different load and environmental conditions, thereby significantly improving the overall performance of the system.

[0063] 5. Accurate management of large-scale multi-node power supply communication:

[0064] In traditional wireless power communication networks, the allocation of energy and communication resources for multiple nodes is relatively complex, and resource waste or imbalance is prone to occur. The present invention combines TDMA (time division multiple access) and DTDMA (dynamic time division multiple access) models to achieve accurate energy management and communication resource allocation for multiple nodes by optimizing time slot allocation and power control. This method can not only avoid resource conflicts, but also significantly improve the power supply efficiency and communication throughput of multiple nodes.

[0065] 6. Significantly reduce network energy consumption and costs:

[0066] The present invention replaces traditional fixed base stations and relay equipment by carrying RIS equipment on drones, reducing the construction and maintenance costs of infrastructure. At the same time, the deep reinforcement learning algorithm maximizes energy efficiency and reduces the overall energy consumption of the network in the joint optimization of energy transmission and communication links. For large-scale deployed wireless power supply communication networks, the solution of the present invention can effectively reduce energy consumption and improve the economic benefits of the system.

[0067] The present invention has achieved remarkable technical progress in energy efficiency improvement, network coverage, communication stability, dynamic optimization, resource management and cost control, solved multiple bottleneck problems in the prior art, and has broad industrial application prospects.

[0068] Second, the technical solution of the present invention mainly solves the technical problems of low energy efficiency, limited communication coverage, inflexible resource allocation and low energy collection efficiency in the existing wireless power supply communication network in industrial applications, and achieves significant technological progress by introducing Aerial-RIS (reflective intelligent surface) and deep reinforcement learning algorithm. The following are the technical problems solved by the present invention in industrial applications and the technological progress achieved:

[0069] 1) Improve energy efficiency and reduce system energy consumption: Traditional wireless power communication networks have low energy efficiency in the process of energy collection and transmission, resulting in low overall energy utilization of the system. This invention jointly optimizes the RIS phase shift matrix, TDMA / DTDMA time slot allocation and drone flight trajectory, so that each ground node can efficiently collect energy and minimize the total energy consumption of the system while meeting the communication quality. The deep reinforcement learning algorithm is used for dynamic optimization to further improve the system energy efficiency.

[0070] 2) Expand signal coverage and enhance communication performance: In existing wireless power communication networks, signal coverage is limited due to fixed base station deployment, especially in complex or remote geographical environments, where communication performance is difficult to guarantee. This invention deploys Aerial-RIS on drones, enabling them to move flexibly and dynamically adjust signal coverage areas, greatly enhancing the signal coverage capability of the system, and optimizing signal transmission quality through RIS phase shift, significantly improving the communication performance of the network.

[0071] 3) Intelligent resource allocation to improve system flexibility: In traditional wireless power communication networks, resource allocation is usually static and difficult to flexibly respond to dynamically changing communication needs. Based on a deep reinforcement learning algorithm, the present invention can dynamically adjust resource allocation strategies, such as time slot allocation, energy collection and communication phase switching, according to environmental changes and communication needs. By interacting with the environment in real time and automatically adjusting parameters, the system can be ensured to operate efficiently in complex environments, thereby improving the intelligence and flexibility of the network.

[0072] 4) Solve the balance problem between drone dynamics and energy collection: In existing drone-assisted wireless communication networks, the flight path and energy collection of drones are often difficult to coordinate effectively, affecting the overall performance of the system. This invention optimizes the flight trajectory of the drone to ensure that it can fly on a reasonable path to meet the energy and communication needs of the ground nodes. At the same time, it takes into account the dynamic limitations of the drone, achieves a balance between energy collection and flight path, and significantly improves the operating efficiency of the system.

[0073] 5) Innovative application of deep reinforcement learning algorithm: Traditional wireless power network optimization methods are mostly based on static models and preset parameters, which are difficult to deal with complex problems in dynamic environments. The present invention innovatively introduces a deep reinforcement learning algorithm, which can optimize system parameters through interactive iterations with the environment, and gradually improve the energy efficiency and performance of the system. The introduction of the DRL algorithm not only improves the intelligence level of the system, but also can autonomously learn and obtain the optimal strategy without fully understanding the environment, thus solving the limitations of traditional optimization methods.

[0074] The present invention successfully solves the problems of low energy efficiency, poor signal coverage, inflexible resource allocation, etc. in the existing wireless power supply communication network through the combination of Aerial-RIS and deep reinforcement learning, significantly improves the overall performance and operation efficiency of the network, and has broad industrial application prospects, especially suitable for application scenarios that require high-efficiency communication such as remote power supply, Internet of Things, and smart cities. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] Figure 1 It is a flow chart of the Aerial-RIS-assisted wireless power supply communication network energy efficiency optimization method based on deep reinforcement learning provided by an embodiment of the present invention;

[0076] Figure 2 It is a structural diagram of the WPCN framework based on Aerial-RIS assistance provided by an embodiment of the present invention;

[0077] Figure 3 is a comparison chart of cumulative reward changes provided by an embodiment of the present invention;

[0078] Figure 4 is a graph showing the relationship between the average throughput of the system and the number of ground nodes provided by an embodiment of the present invention;

[0079] Figure 5 is a graph showing the relationship between the system average throughput and the number of ground nodes provided by an embodiment of the present invention;

[0080] Figure 6 This is a structural diagram of an Aerial-RIS-assisted wireless power supply communication network energy efficiency optimization system based on deep reinforcement learning provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0081] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0082] Example 1: Wireless power supply and communication network optimization for IoT devices

[0083] In smart city or industrial IoT applications, IoT devices are usually distributed in a wide geographical area, and traditional power supply methods are difficult to cover devices in remote and remote areas. At the same time, these devices have high requirements for energy efficiency and communication quality. The Aerial-RIS-assisted wireless power supply communication network optimization method based on the present invention can provide wireless energy transmission and communication services for a wide range of IoT devices by deploying drones carrying reflective smart surfaces (RIS).

[0084] Drones can flexibly adjust flight trajectories based on the distribution of IoT devices, while using RIS to enhance signal coverage. The optimization algorithm based on deep reinforcement learning can adjust the phase shift matrix, time slot allocation, and flight path of RIS in real time to ensure that each IoT node can efficiently receive energy and communicate data. This approach significantly improves the energy efficiency of the IoT system, solves the problems of limited coverage and insufficient energy supply in traditional wireless power supply networks, and ensures the quality of data transmission between devices.

[0085] Example 2: Communication network and power supply guarantee in rural or remote areas

[0086] In rural, remote or disaster areas, the communication infrastructure is usually imperfect, power supply equipment is difficult to cover, and traditional communication networks are difficult to meet the needs of local residents or emergency rescue. The Aerial-RIS auxiliary wireless power supply communication network provided by the present invention can play a key role in these areas.

[0087] By deploying drones carrying Aerial-RIS, base stations can perform long-distance energy transmission to remote areas and enhance signal coverage and transmission efficiency through RIS. In this case, drones can move flexibly in the air, adjust flight trajectories according to geographical environment and communication needs, and ensure that all ground nodes receive enough energy for communication. The application of deep reinforcement learning algorithms can automatically optimize system parameters according to real-time environment and needs, ensure efficient power supply and communication, improve the coverage and stability of communication networks, and greatly improve communication conditions and power supply capabilities in rural and remote areas.

[0088] like Figure 1 As shown, an Aerial-RIS-assisted wireless power supply communication network energy efficiency optimization method based on deep reinforcement learning provided by an embodiment of the present invention includes the following steps:

[0089] Step 1: Deploy RIS on the drone as a relay to build the system architecture of the Aerial-RIS assisted wireless power supply communication network ( Figure 2 );

[0090] Step 2: Based on the Aerial-RIS assisted WPCN system of TDMA and DTDMA, the RIS phase shift matrix, TDMA / DTDMA time slot allocation ratio τ and UAV trajectory are jointly optimized;

[0091] Step three, based on the deep reinforcement learning algorithm, by observing the rewards and iteratively adjusting the parameters of the DRL algorithm accordingly, the energy efficiency is gradually maximized to obtain the optimal strategy.

[0092] In step 1, the base station is responsible for providing initial wireless energy transmission and communication services, while Aerial-RIS moves flexibly in the air as a relay to enhance signal coverage and transmission efficiency. The ground node uses the energy received from the base station and the drone relay to transmit information.

[0093] In step 2, an Aerial-RIS-assisted WPCN system model based on TDMA and DTDMA is established, including a signal transmission model, an energy collection model, a communication model, etc. The system energy efficiency optimization problem is transformed into a joint optimization problem, including optimizing the RIS phase shift matrix, the time slot allocation ratio τ in the energy collection phase, and the flight trajectory of the UAV. The goal is to maximize the energy efficiency of the entire system, that is, to minimize the total energy efficiency of the system while meeting the energy requirements and communication quality of all terminals. . This optimization problem aims to minimize the total energy consumption while ensuring the quality of information transmission, and it is necessary to consider the RIS physical constraints, time slot allocation protocol, UAV dynamics limitations, and the balance between energy collection and transmission. An intelligent optimization algorithm is introduced to automatically adjust parameters through interaction with the environment to maximize the system energy efficiency.

[0094] The present invention is based on the Aerial-RIS assisted wireless power supply communication network energy efficiency optimization method based on deep reinforcement learning. Its working principle can be divided into multiple key links, covering the construction of system architecture, signal and energy transmission model, joint optimization process and application of deep reinforcement learning.

[0095] First, in step one, Aerial-RIS (Reflective Intelligent Surface) is deployed on the drone as a relay node to build an Aerial-RIS-assisted wireless power communication network system. The base station is responsible for providing the initial wireless energy transmission, and the ground node transmits information through the energy received by the RIS relay. The Aerial-RIS carried by the drone moves flexibly in the air, adjusting its position and reflection phase shift matrix to enhance the signal coverage and transmission efficiency. By combining RIS with drones, the system can dynamically adapt to environmental changes and improve communication performance and energy transmission efficiency.

[0096] Secondly, in step 2, an Aerial-RIS assisted wireless power supply communication network model based on time division multiple access (TDMA) and dynamic time division multiple access (DTDMA) was established. The model includes a signal transmission model, an energy collection model, and a communication model. At this stage, the system controls the allocation of energy collection and communication time through the time slot allocation mechanism of TDMA / DTDMA. At the same time, the optimization of the RIS phase shift matrix is ​​used to maximize the signal gain, and the flight trajectory of the drone is planned according to the distribution and energy requirements of the ground nodes to ensure that each node can effectively receive energy and transmit information.

[0097] Third, the optimization problem is defined as a joint optimization problem, which aims to maximize the energy efficiency of the system by jointly adjusting the RIS phase shift matrix, TDMA / DTDMA time slot allocation ratio, and UAV trajectory. This process needs to consider multiple constraints, including the physical limitations of RIS, the design of the time slot allocation protocol, the dynamic limitations of the UAV, and the balance between energy collection and information transmission. The optimization goal is to minimize the total energy consumption of the system while meeting the energy requirements and communication quality of all terminals, thereby improving the energy efficiency of the entire system.

[0098] Fourth, the present invention introduces the deep reinforcement learning (DRL) algorithm as an intelligent optimization tool to solve the above joint optimization problem. During the optimization process, the DRL algorithm observes the state and reward of the system through interaction with the environment, and dynamically adjusts the algorithm parameters according to the reward feedback, gradually improving the optimization strategy. Through multiple iterations, the DRL algorithm can gradually learn the optimal RIS phase shift matrix configuration, time slot allocation and drone trajectory planning, thereby maximizing the system energy efficiency.

[0099] Fifth, during the DRL training process, the system's energy efficiency is fed back to the algorithm as a reward signal, and the decision is continuously optimized through interaction with the environment. Each adjustment of the strategy will affect the RIS phase shift matrix, time slot allocation, and the flight trajectory of the drone, thereby affecting the energy efficiency of the entire system. Through this continuous iterative process, the DRL algorithm gradually converges to an optimal solution, enabling the system to effectively respond to the energy requirements and signal transmission quality requirements of different nodes in a complex communication environment.

[0100] Finally, the final optimization strategy of the entire system is obtained through multiple rounds of learning iterations, ensuring that the total energy consumption of the system can be minimized while meeting the terminal communication needs. By combining deep reinforcement learning with Aerial-RIS, the system can dynamically adapt to the changing environment and make intelligent decisions, improving the overall energy efficiency of the system, and is widely applicable to wireless power communication scenarios that require efficient energy transmission.

[0101] TDMA-based system model:

[0102] The ground node receiving power at time slot n is:

[0103] E k [n] = η(τ k [n]P B )|g UK [n] H Θ[n]g BU [n]| 2 (1)

[0104] Where η is the energy collection efficiency, τ k [n] is the energy harvesting time slot allocation ratio, P B is the transmission power of HAP, g UK [n] is the channel gain between Aerial-RIS and the node, g BU [n] is the channel gain from HAP to each reflective element of RIS, and Θ[n] is the phase shift matrix of RIS.

[0105] The throughput of node k in time slot n can be expressed as:

[0106]

[0107] Where P T [n] is the transmission power of the kth node, where σ 2 is a noise signal that follows a Gaussian distribution

[0108] System model based on DTDMA:

[0109] The ground node receiving power at time slot n is:

[0110]

[0111] Where η is the energy collection efficiency.

[0112] The throughput of node k in time slot n can be expressed as:

[0113]

[0114] In addition, the optimization objective function of the system energy efficiency is defined, including key indicators such as total energy consumption and information transmission rate.

[0115] The goal of this invention is to maximize the energy efficiency of the system by jointly optimizing the RIS phase shift reflection coefficient, the position of the UAV and the time slot allocation ratio τ of the EH, and to achieve the coordinated management of communication quality and transmission energy consumption. Based on the above analysis, the optimization problem can be expressed as:

[0116]

[0117] In the formula, Q total is the total energy consumption of the system.

[0118] In step 3, when constructing the Aerial-RIS-assisted wireless power communication network, a deep reinforcement learning algorithm is used to optimize the system energy efficiency. By continuously observing the system's reward feedback for specific behaviors, the internal parameters are iteratively adjusted to gradually approach the optimal strategy for maximizing energy efficiency. Through continuous iterative learning and optimization, the DRL algorithm can adaptively respond to environmental changes and provide an efficient, reliable and flexible energy efficiency optimization solution for WPCN.

[0119] The specific steps are:

[0120] Input: system initial state, discount factor γ, learning rate, experience replay size N D , mini-batch size N B

[0121] Output: Best strategy

[0122] Initialization: actor network π(s|θ π ) and the critic network Q(s,a|θ Q ), target actor network π'(s|θ π′ ) and the target critic network Q'(s,a|θ Q' ), θ π '=θ π ,θ Q '=θ Q , experience playback buffer D, phase shift matrix Φ

[0123] for episode=1,N do:

[0124] Accept the current environment and get the initial state s(0);

[0125] Initialize a random noise

[0126] for n=1:N do

[0127] With status s n As input, get action a from the policy network n ;

[0128] Perform actions and get rewards and status n+1 ;

[0129] Will Stored in experience replay pool D;

[0130] Randomly draw a batch of samples from the experience replay pool D;

[0131] Calculate the target Q value;

[0132] Update the value network;

[0133] Use SGD to update the gradient of the policy network;

[0134] Update the policy network so that the actions of the policy network can better approximate the real actions;

[0135] Soft update of target network parameters of the policy network and value network;

[0136] Update Status n =s n+1

[0137] end for

[0138] end for.

[0139] like Figure 6 As shown, an Aerial-RIS-assisted wireless power supply communication network energy efficiency optimization method based on deep reinforcement learning and an Aerial-RIS-assisted wireless power supply communication network energy efficiency optimization system based on deep reinforcement learning are provided in an embodiment of the present invention, including:

[0140] System architecture building module, deploying RIS on drones to act as a relay and building the system architecture of the Aerial-RIS assisted wireless power supply communication network;

[0141] Joint optimization module, based on the Aerial-RIS assisted WPCN system of TDMA and DTDMA, jointly optimizes the RIS phase shift matrix, TDMA / DTDMA time slot allocation ratio τ and UAV trajectory;

[0142] The parameter adjustment module, based on the deep reinforcement learning algorithm, gradually maximizes the energy efficiency by observing the rewards and iteratively adjusting the parameters of the DRL algorithm accordingly, thereby obtaining the optimal strategy.

[0143] Application Example 1: Smart City Wireless Power Supply and Communication Network Optimization

[0144] In the construction of smart cities, Internet of Things (IoT) devices are widely distributed in every corner of the city, and these devices usually require low-power wireless power supply network support. However, traditional wireless power supply communication networks have problems such as low energy efficiency and difficulty in link optimization. The present invention provides an Aerial-RIS assisted wireless power supply communication network optimization method based on deep reinforcement learning, which is flexibly deployed in the air through a programmable reflect array (RIS) carried by a drone as a relay device to optimize the wireless communication link.

[0145] Implementation steps:

[0146] 1. System architecture deployment: In a smart city environment, drones carry RIS devices and fly in different areas to optimize the communication link with IoT devices, while optimizing the reflected signal through the RIS phase shift matrix. The ground node receives energy from the drone and RIS for information transmission of the device.

[0147] 2. Optimization goal: Through the deep reinforcement learning algorithm, according to the distribution of ground nodes in the actual scene, the flight trajectory of the drone, the RIS phase shift matrix and the time slot allocation strategy are iteratively adjusted to maximize the energy transmission and communication efficiency.

[0148] 3. Specific achievements: In smart cities, the optimization algorithm of the present invention greatly improves the power supply efficiency and communication reliability of IoT devices, reduces energy consumption and communication delays, and significantly improves the overall intelligence level of the city.

[0149] Application Example 2: Agricultural Monitoring and Wireless Power Supply System

[0150] In modern smart agriculture, a large number of sensors are deployed in the fields to monitor the crop growth environment. These devices usually require wireless power supply and remote data transmission. However, due to the large area of ​​agricultural areas, traditional power supply and communication networks are easily affected by obstacles, resulting in low power supply efficiency and unstable communication signals. The present invention optimizes the energy efficiency of agricultural sensor networks by using drones to carry RIS devices to optimize wireless power supply and communication networks.

[0151] Implementation steps:

[0152] 1. System architecture deployment: UAVs are flexibly deployed over agricultural areas, carrying RIS devices to provide wireless power supply support for sensor nodes on the ground. By optimizing the flight path of the drone, the RIS device is ensured to reflect the signal at the best position, improving energy transmission and communication quality.

[0153] 2. Optimization goal: Based on the deep reinforcement learning algorithm, adjust the RIS phase shift matrix, TDMA / DTDMA time slot allocation ratio τ and the trajectory of the drone to optimize the wireless energy transmission efficiency and the communication reliability of sensor data return.

[0154] 3. Specific results: By optimizing the wireless power supply and communication network, sensor equipment in agricultural areas can efficiently receive wireless energy, and sensor data transmission is stable, which improves the accuracy and real-time performance of agricultural environmental monitoring, while reducing maintenance costs and energy consumption.

[0155] These two specific application embodiments demonstrate the wide applicability of the present invention in different fields, especially the significant technological progress in large-scale wireless sensor networks such as smart cities and smart agriculture.

[0156] 2. Relevant evidence of the technical effects obtained by the embodiments of the present invention.

[0157] It should be noted that the embodiments of the present invention can be implemented by hardware, software, or a combination of software and hardware. The hardware part can be implemented using dedicated logic; the software part can be stored in a memory and executed by an appropriate instruction execution system, such as a microprocessor or dedicated design hardware. It can be understood by a person of ordinary skill in the art that the above-mentioned devices and methods can be implemented using computer executable instructions and / or contained in a processor control code, such as a carrier medium such as a disk, CD or DVD-ROM, a programmable memory such as a read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. Such code is provided on the carrier medium. The device and its modules of the present invention can be implemented by hardware circuits such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, etc., or programmable hardware devices such as field programmable gate arrays, programmable logic devices, etc., can also be implemented by software executed by various types of processors, and can also be implemented by a combination of the above-mentioned hardware circuits and software, such as firmware.

[0158] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with the technical field within the technical scope disclosed by the present invention and within the spirit and principle of the present invention should be covered by the protection scope of the present invention.

Claims

1. A method for optimizing energy efficiency of wireless power supply communication network assisted by Aerial-RIS based on deep reinforcement learning, characterized in that: The following steps are involved: Step 1: Deploy RIS on the drone as a relay to build the system architecture of the Aerial-RIS assisted wireless power supply communication network; Step 2: Based on the Aerial-RIS assisted WPCN system of TDMA and DTDMA, the RIS phase shift matrix, TDMA / DTDMA time slot allocation ratio τ and UAV trajectory are jointly optimized; Step three, based on the deep reinforcement learning algorithm, by observing the rewards and iteratively adjusting the parameters of the DRL algorithm accordingly, the energy efficiency is gradually maximized to obtain the optimal strategy.

2. The Aerial-RIS-assisted wireless power supply communication network energy efficiency optimization method based on deep reinforcement learning as claimed in claim 1, characterized in that: In the step 1, the UAV is used as a mobile carrier to carry the RIS device, and the communication link is optimized by flexibly moving in the air and adjusting the phase of the reflection element; the HAP is used to send energy signals to the Aerial-RIS and receive information signals from the Aerial-RIS; the ground node receives energy from the UAV and RIS, and uses the collected energy to transmit information.

3. The Aerial-RIS-assisted wireless power supply communication network energy efficiency optimization method based on deep reinforcement learning as claimed in claim 1, characterized in that: In the step 2, an Aerial-RIS assisted WPCN system model based on TDMA and DTDMA is established, including a signal transmission model, an energy collection model, and a communication model.

4. The Aerial-RIS-assisted wireless power supply communication network energy efficiency optimization method based on deep reinforcement learning as claimed in claim 3, characterized in that: TDMA-based system model: The ground node receiving power at time slot n is: E k [n]=η(τ k [n]P B )|g UK [n] H I[n]g BU [n]| 2 (1) Where η is the energy collection efficiency, τ k [n] is the energy harvesting time slot allocation ratio, P B is the transmission power of HAP, g UK [n] is the channel gain between Aerial-RIS and the node, g BU [n] is the channel gain from HAP to each reflective element of RIS, Θ[n] is the phase shift matrix of RIS; The throughput of node k in time slot n is expressed as: Where P T [n] is the transmission power of the kth node, where σ 2 is a noise signal that follows a Gaussian distribution 5. The Aerial-RIS-assisted wireless power supply communication network energy efficiency optimization method based on deep reinforcement learning as claimed in claim 3, characterized in that: System model based on DTDMA: The ground node receiving power at time slot n is: Where η is the energy collection efficiency; The throughput of node k in time slot n is expressed as:

6. The Aerial-RIS-assisted wireless power supply communication network energy efficiency optimization method based on deep reinforcement learning as claimed in claim 1, characterized in that: The specific steps of step three are: Input: system initial state, discount factor γ, learning rate, experience replay size N D , mini-batch size N B Output: Best strategy Initialization: actor network π(s|θ π ) and the critic network Q(s,a|θ Q ), targetactor networkπ'(s|θ π′ ) and targetcritic network Q'(s,a|θ Q' ), θ π' =θ π ,θ Q' =θ Q , experience playback buffer D, phase shift matrix Φ for episode = 1, N do: Accept the current environment and get the initial state s(0); Initialize a random noise for n=1:N do With status s n As input, get action a from the policy network n ; Perform actions and get rewards and status n+1 ; Will Stored in experience replay pool D; Randomly draw a batch of samples from the experience replay pool D; Calculate the target Q value; Update the value network; Use SGD to update the gradient of the policy network; Update the policy network so that the actions of the policy network can better approximate the real actions; Soft update of target network parameters of the policy network and value network; Update Status end for end for.

7. A wireless power supply communication network energy efficiency optimization system based on Aerial-RIS assisted by deep reinforcement learning according to any one of claims 1 to 6, characterized in that: include: System architecture building module, deploying RIS on drones to act as a relay and building the system architecture of the Aerial-RIS assisted wireless power supply communication network; Joint optimization module, based on the Aerial-RIS assisted WPCN system of TDMA and DTDMA, jointly optimizes the RIS phase shift matrix, TDMA / DTDMA time slot allocation ratio τ and UAV trajectory; The parameter adjustment module, based on the deep reinforcement learning algorithm, gradually maximizes the energy efficiency by observing the rewards and iteratively adjusting the parameters of the DRL algorithm accordingly, thereby obtaining the optimal strategy.

8. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the Aerial-RIS-assisted wireless power supply communication network energy efficiency optimization method based on deep reinforcement learning as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor executes the steps of the Aerial-RIS-assisted wireless power supply communication network energy efficiency optimization method based on deep reinforcement learning as described in any one of claims 1 to 6.

10. An information data processing terminal, characterized in that: The information data processing terminal includes the Aerial-RIS-assisted wireless power supply communication network energy efficiency optimization system based on deep reinforcement learning as described in claim 7.

Images